class10_Bash Shell scripting.pdf
-
Upload
pavan-behara -
Category
Documents
-
view
227 -
download
0
Transcript of class10_Bash Shell scripting.pdf
-
7/29/2019 class10_Bash Shell scripting.pdf
1/46
Bash Survival Guide for HPC
M. D. Jones, Ph.D.
Center for Computational ResearchUniversity at Buffalo
State University of New York
High Performance Computing I, 2012
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 1 / 52
-
7/29/2019 class10_Bash Shell scripting.pdf
2/46
Introduction A Little History
History of bash
The concept of the shell environment has been around for a long time
(long = lifetime of UNIX operating system):
bash is a twist on one of the earliest UNIX shells (the so-called
Bourne shell, sh on most systems).bash is a superset of the (limited) Bourne shell syntax
There is quite an extended family of shells, a good link for whichcan be found here:
http://en.wikipedia.org/wiki/Comparison_of_command_shells
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 3 / 52
http://en.wikipedia.org/wiki/Comparison_of_command_shellshttp://en.wikipedia.org/wiki/Comparison_of_command_shells -
7/29/2019 class10_Bash Shell scripting.pdf
3/46
Introduction Feature Subset
bash Features
Subset of important features of bash:
Basically a command line interface (CLI) to the operating system
Command line completion (via TAB key)
Command line editing, history
Output redirection
Control constructs
...
Basically all of the shells give you the ability to enter text-based
commands/queries much faster than using a GUI (if your particular OSeven supports a GUI).
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 4 / 52
-
7/29/2019 class10_Bash Shell scripting.pdf
4/46
Introduction Startup Scripts
bash Invocation/Startup Scripts
bash will read from various startup scripts depending on how it was
invoked:
Interactive login shell: uses /etc/profile, then
~/.bash_profile, ~/.bash_login, ~/.profile in thatorder (if they are present at all).
Interactive non-login shell: ~/.bashrc (suppressed by --norcoption, or --rcfile file alternative)
Non-interactive: uses $BASH_ENV (subject to substitution, if resultis a file, executes the file)
as sh: attempts to duplicate Bourne shell behavior, reads andexecutes /etc/profile and ~/.profile, in order
via rshd: will read and execute ~/.bashrc (if it is able todetermine that is was launched via rsh)
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 5 / 52
-
7/29/2019 class10_Bash Shell scripting.pdf
5/46
Introduction Navigation
CLI Navigation
bash supports a rich set of line editing features making it easier for
you to move the cursor around the command line, recall pastcommands and edit them, etc. The default syntax follows that of theemacs text editor (you can customize the syntax, of course, see manbash). Very briefly, the arrow keys can typically be used to navigatethe current command line (left and right arrow keys, up and down to
scroll though past commands), plus some common key bindings (thereare many more than this small sample):C-a go to the beginning of the line
C-e go to the end of the line
C-f,C-b go forward,back one character
M-f,M-b go forward,back one word
where C- denotes the Control key on the keyboard, and M- the METAkey - not many keyboards have META keys these days, in which case
ESC is used instead.
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 6 / 52
I d i N i i
-
7/29/2019 class10_Bash Shell scripting.pdf
6/46
Introduction Navigation
CLI History
Unsurprisingly, the shell records a history of what commands youexecute. The history command will list the commands in your shells
history with an identification number that you can use as shorthand forrecalling them:
[ u2 : ~ ] $ ! ! # r e p e a t v e r y l a s t commandl s l[ u 2 : ~ ] $ h i s t o r y # s p i t s o u t l a s t HISTSIZE c ommands[ u 2 : ~ ] $ ! 15 74 # p i c k one b y number , p r ef a ce w i t h !who am ijo ne sm p t s /2 3 20120525 1 2 :5 3 ( c as h . c c r . b u f f a l o . edu )[ u 2 : ~ ] $ ! 1 56 2 : s/ x/a / # p i ck one , use a r ege x t o m od if y on t he f l ypbsnodes a | l e s s
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 7 / 52
I t d ti E i t V i bl
-
7/29/2019 class10_Bash Shell scripting.pdf
7/46
Introduction Environment Variables
Environment Variables
Environment variables are heavily used by all shells to set the user
environment and control its behavior, and are also frequently used byapplications. For now, here are just a few of the simplest of the
defaults:
$HOME , your home directory location.
$PATH , search path for commands$LD_LIBRARY_PATH , search path for dynamic libraries
$MANPATH , search path for man pages
$PS1 , command prompt syntax
Note that the modules environment package used in CCR is designedto allow you to more easily manipulate the environment (especially$PATH, $LD_LIBRARY_PATH, and $MANPATH) for various softwarepackages.
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 8 / 52
Introduction Environment Variables
-
7/29/2019 class10_Bash Shell scripting.pdf
8/46
Introduction Environment Variables
You can set environment variables very simply by assignment or usingthe export command, the difference being whether the value is
passed on to sub-shells (i.e. if you run another script or executable) -you can also run a program with specific settings using the envcommand:
# t hes e v a r ia b le s ar e s e t o nl y i n t he c u rr e nt s h e l lNP=` cat $PBS_NODEFILE | wc l ` # counts l in es in $PBS_NODEFILENODES= ` cat $PBS_NODEFILE | un iq ` # counts unique l in es in $PBS_NODEFILE
# t h i s a pp li es t o a l l c h i ld re n o f t h i s s h e l l# ( I n t e l MKL core a f f i n i t y s e t t i n g )export KMP_AFFINITY=no war nin gs , compact# t h i s r uns a s h e l l s c r i p t us in g a l o c a l s e t t i n g f o r $HOME# ( MATLAB c o m pi le d b i n a r i e s l i k e t o abus e $HOME d i r e c t o r i e s . . . )env HOME=/tmp . / ru n _ e xt ra c t . sh
You can add common settings to your $HOME/.bashrc file if you wantthem to apply to all of your logins/shells. Using export without anyarguments will print the current settings for your shell (as will the envcommand).
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 9 / 52
Basic Syntax Format
-
7/29/2019 class10_Bash Shell scripting.pdf
9/46
Basic Syntax Format
The First Line
Shell scripts can be no more than an ordered list of commands to run.
By convention, the first line declares the shell:
# ! / bin / bash## pound sy mbol g e n e r a l l y s t a r t s a comment , e xc ep t f o r t h a t f i r s t l i n eecho ' H e l l o , w o r l d ! '
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 11 / 52
Basic Syntax Format
-
7/29/2019 class10_Bash Shell scripting.pdf
10/46
Basic Syntax Format
Variables
Variables in bash are easily declared:# some v a r i a b l e d e c l a r a t i o n s :st r1 =" Goodbye , wo rl d ! " # n o te q uo te s t o p r o t e c t t h e s pac es t r 2 = ` d at e # b a c k t i c k s t o e x e cu t e c ommands ,
# c ap t ur e th e ou tp utecho n '$USER = ' # si n g l e q u o tes do n o t e xa nd $USER
# and n d oes n ot b re ak l i n eecho "$USER" # d o u bl e q u o tes w i l l e xp an d $USER
# p r ec e di n g t w o l i n e s p r i n t o u t $USER = jo hn do e f o r u s e r i d j oh nd oe# t h e r e a re a bunc h o f p r e d e f in e d v a r i a b l e s , $USER i s oneecho " $ { s t r 1 } A ar gh ! " # b r ac es a re handy t o p r o t e c t v a r i a b l e ss t r 3 = " "i f [ n " $ s t r3 " ] ; then # n i n d i c a t es nonempty argument
echo ' $ s t r 3 i s n o t empty ! 'f i
That last bit is a good example of why variables are often enclosed inquotes (they do not have to be), in case they evaluate as the emptystring.
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 12 / 52
Basic Syntax Command Substitution
-
7/29/2019 class10_Bash Shell scripting.pdf
11/46
Basic Syntax Command Substitution
Command Substitution
Command substitution is an invaluable way to access the output of acommand from the environment. You can do this either with backticks,
command, or subshells, $(commands):
f i l e s = " $ ( l s ) " # l i s t i n g o f cwdh t ml _ f i l e s =` l s / va r /www/ h tml # l i s t i n g o f / v a r / www / h tm li n d e x = ` expr 4 * 2 + 1 ` # u se e x pr t o e v a l u at e math
Note that $() is very easy to nest, which gives it an edge overbackticks. Newlines are not allowed, so they are stripped out in both
methods.
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 13 / 52
Basic Syntax Arithmetic Expansion
-
7/29/2019 class10_Bash Shell scripting.pdf
12/46
Basic Syntax Arithmetic Expansion
Arithmetic
bash has no notion of floating-point math, although you can resort toexternal tools like bc to work around that lack. Integer arithmetic isdone using backticks, double parentheses, or let:
z = ` expr $ z + 1 ` # expr command does the workz=$ ( ( $ z+1 ))z =$ ( ( z + 1 ) ) # p a ra m et e r der e f e r e n c i ng o p t i o n a l i n s i d e d ou bl e p ar en th es es( ( z += 1 ) ) # s h o r tc u tl e t z =z +1 # l e t commandl e t " z += 1 " # q uo te s l e t you use s pa ce s
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 14 / 52
Basic Syntax I/O Redirection
-
7/29/2019 class10_Bash Shell scripting.pdf
13/46
y
I/O Redirection
Very handy topic - how to redirect input/output in bash. First, though,there are three default files:
stdin (keyboard), descriptor 0
stdout (screen), descriptor 1stderr (error messages to the screen), descriptor 2
Redirection just means capturing output from a file, command, or script
(or portion of a script) and sending it elsewhere as input. Note that
descriptors 3-9 are also available for use.
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 15 / 52
Basic Syntax I/O Redirection
-
7/29/2019 class10_Bash Shell scripting.pdf
14/46
y
Generally you can redirect input with (file), but there are quite a few more variations best illustrated byexample:
echo ' b la h ' > f i le n am e # o v e r w r i t es f i le n am eecho ' b la h ' >> f i l e n am e # a pp en ds f i l e n a m e. / somecode 1> f i l enam e # f i l e n a m e c a p t u r e s t d o u t f r om somecode
. / somecode 2> f i l enam e # f i l e na m e c a p tu r e s s t d e r r f ro m somecode. / somecode &> f i l enam e # f i le n am e c a pt u re s b ot h s t d o ut and s t d e r r
. / somecode > f i l enam e 2>&1 # same as abov e , o l d ( B our ne ) s t y l e
. / o t he r co d e < f i l e na m e # o th er co de g et s i n p u t f ro m f i le n am e
. / code < inname &> outname # c o m bi n at i on s a re okcommand1 | command2 | command3 # p ip es a re s i m i l a r t o > b ut more g en er al
# f o r c h a i n i n g commands t o g e t h e rcat * . t x t | s or t | u n i q > r e s u l t # see ?
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 16 / 52
Control Constructs Conditionals
-
7/29/2019 class10_Bash Shell scripting.pdf
15/46
if--fi
bash has a conditional construct, of course:
i f c o n d i t i o n A
thensta te me n t0sta te me n t1. . .
e l i f c o n d i t i o n Bthen
morestatements1morestatements2. . .
elseyetmore0yetmore1. . .
f i
We will go into the sytnax for the conditions next, but note that the
conditional branches are often written more concisely using the ";"separator:
i f c o n d i t i o n A ; thenl i s t o f s t a t e m e n t s
. . .f i
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 18 / 52
Control Constructs Conditionals
-
7/29/2019 class10_Bash Shell scripting.pdf
16/46
Is There in Truth No Beauty?
The test command (and its shorthand version) gets used quite often in
conditionals (and in general):
# f i r s t f o rmt e s t o pe ran d1 o p e ra to r o p eran d2# ex am pl e :
i f t es t
n "$PBS_ENVIRONMENT" ; thenecho " Hey , I am i n a PBS e n v ir o n me n t ! "f i# secon d fo rm[ o pe ra nd 1 o p e r a t o r o pe ra nd 2 ]# e xa mp le :i f [ n "$PBS_ENVIRONMENT" ] ; then
echo " Hey , I am s t i l l i n a PBS e n v ir o nm e n t ! "f i
# I c heat ed a b i t ,
n i s a u na ry o p er a to r . . .
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 19 / 52
Control Constructs Conditionals
-
7/29/2019 class10_Bash Shell scripting.pdf
17/46
Operators
I am not going to show the full set of test operators (see man bash),
but here is a quick subset:
Operator Operands-n 1 nonzero length-z 1 zero length-f 1 file given by operand exists
-d 1 directory given by operand exists== 2 string equality!= 2 2 string lexical-eq,-ne 2 integer (in)equality-lt,-gt 2 integer comparisons-le,-ge 2 integer comparisons
|| 2 logical OR&& 2 logical AND
Note the absence of floating-point operations (serious shortcoming).
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 20 / 52
Control Constructs Loop Constructs
-
7/29/2019 class10_Bash Shell scripting.pdf
18/46
for Loops
First of the main loop types in bash, for loops repeat over a simple
index of space separated items:f o r name i n " $ LIST "do
echo " working on $name"b u n ch o fsta te me n ts1
done
So the loop counter, name is set to each item in the list as the loop isexecuted. Note that the familiar form from C for integer indices is also
valid:
f o r ( ( e xp r1 ; e xp r2 ; e xp r3 ) ) ; do l i s t o f s ta t e m e nt s ; done# f o r ex am pl e :f o r ( ( i = 1 ; i < 10 ; i + + ) )
doecho " i = $ i "
done
Note that bash is pretty slow (most interpreted languages are slow),
so the C-like loop syntax is not encouraged.
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 21 / 52
Control Constructs Loop Constructs
-
7/29/2019 class10_Bash Shell scripting.pdf
19/46
Globbing
You can make handy use of substitution by globbing, or using * tomatch all filenames.
# s i m p l e g l o b b i n g e xa mp lef o r fna me i n * . c # grabs a l l . c f i l e s i n c ur re nt d i re c to r ydo
g re p ' d o ub l e ' $ fn ame # p u l l a l l o u t a l l th e d oublesdone
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 22 / 52
Control Constructs Loop Constructs
L
-
7/29/2019 class10_Bash Shell scripting.pdf
20/46
while Loops
While loops are just about what you would expect:
while l i s t ; do l i s t o f s t a t e m e n t s ; done# si mp l e e xa mp lewhile [ " $ c o u n te r " l t "$UPPER_LIMIT" ] ; do
echo " c o u n t e r = $ c o u n te r "c o u n t e r = `expr $ c o u n t e r + 1 ` # c o u nt e r =$ ( ( $ c ou n te r + 1 ) ) s h ou ld a l s o w or k
# l e t " c o u n t e r += 1 " sho uld al s o workdone
There is also an until variant of while that has the same syntax
(just negates the test).
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 23 / 52
Miscellaneous Advanced Features Conditional Execution
C diti l E ti
-
7/29/2019 class10_Bash Shell scripting.pdf
21/46
Conditional Execution
A very typical bash-ism is conditional execution, namely use the return
code from a command (you know that all commands have returncodes to indicate success or failure, right?):
command1 && command2 # command2 i f f command1 re t u rn s t r u e (0 )command1 | | command2 # command2 i f f command1 re tu rn s fa ls e (non
zero )
command1 && command2 | | command3 # command2 i f f command1 re tu rn s tr ue# command3 i f f command1 re t u rn s fa l s e
#rm f $ t h i s f i l e && echo " $ t h i s f i l e was removed . " | | echo " $ t h i s f i l e n ot removed . "## c an ch ec k e x i t s t a t u s o f l a s t command u s in g $?echo $?
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 25 / 52
Miscellaneous Advanced Features Functions
F ti
-
7/29/2019 class10_Bash Shell scripting.pdf
22/46
Functions
bash has support for functions, general syntax:
#
# g e n er a l f u n c t i o n s y nt ax# o ne way:fu n ct i on _ n a me { # body o f f u n c t i o n i n s i d e br ac es
sta te me n ts ; # s ho ul d end w i t h s em ic ol on o r n ew li ne}## a n o t he r way :fu n c ti o n f u nc t io n am e ( ) # u si n g t he f u n c ti o n b u i l t i n r eq ui re s ( ){
sta te me n t1sta te me n t2. . .statementNre tu rn # o r an e x p l i c i t r e t ur n b u i l t i n
## h andy f u n c t i o n f o r DOS / w indows s h e l l u s er sd i r ( ){
l s
F - c o l o r = a u t o
l F - c o l o r = a lw ay s "$@" | l e s s
r # $@ i s $1 , $2 , . . .}
}
Return code is the the return code from the last statement in the
function.
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 26 / 52
Miscellaneous Advanced Features Functions
F ti P t
-
7/29/2019 class10_Bash Shell scripting.pdf
23/46
Function Parameters
You can pass arguments to functions (they behave like mini-scripts).
They become positional parameters that you can reference as $1, $2,...
## f un w it h f u n ct i on ( o r s c r i p t i t s e l f ) param eters#echo " My name i s $ 0 " # name o f t he s c r i p t
echo " I h ave $ # a rgu men ts" # n umber o f a rg um en ts i s $#i f [ n " $1 " ]then
echo " F i r s t ar gum ent i s $1 " # 1 s t a rg um en tf i## An o f t e n used f or m f o r s c r i p t s t o c he ck a rg um en t number#i f [ ! $# eq 2 ] ; th en
echo " U sage : $0 a r g1 a rg 2 "e x i t 1f i
Note that $0 remains unaffected by calling a function.
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 27 / 52
Miscellaneous Advanced Features Functions
Bit More Function Trivia
-
7/29/2019 class10_Bash Shell scripting.pdf
24/46
Bit More Function Trivia
Functions can be recursive, and can have their own local variables
through the local builtin.
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 28 / 52
Miscellaneous Advanced Features Shell Limits
Shell Limits
-
7/29/2019 class10_Bash Shell scripting.pdf
25/46
Shell Limits
bash shell limits are controlled through the ulimit builtin function.
Easiest way to see that is by example:[ bono : ~ ] $ u l i m i t acore f i l e s i z e ( bl oc ks , c ) u n li m i te ddata seg s i z e ( kbytes , d ) u n l i m it e df i l e s i z e ( bl oc ks , f ) u n l i m it e dpending s i g n a l s ( i ) 1024max l oc ke d memory ( k by tes , l ) 32max memory s i z e ( kbytes , m) u n l i m it e d
open f i l e s (
n ) 8 19 2p i p e s i z e (512 bytes , p ) 8POSIX message q ue ues ( b y te s , q) 819200s t a c k s i z e ( kbytes , s ) u n li m i te dcpu t i me ( seconds , t ) u n l i m it e dmax user processes (u ) u n l i m it e dv i r t u a l memory ( kbytes , v ) u n li m i te df i l e l o c k s (x ) u n li m i te d
The values can be numerical or one of hard, soft or unlimited. Theflags -H, -S can be used to specify the hard and soft limits. Note thatthe stack limit is of particular interest to us in HPC, and very frequently
needs to be increased from its (small) default value.
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 29 / 52
Miscellaneous Advanced Features Aliases
Aliases
-
7/29/2019 class10_Bash Shell scripting.pdf
26/46
Aliases
Aliases are basically keyboard shortcuts that you can set up and use -
they do not do any expansion, nor can they be recursive:
## f r om my ~ / . b a sh r c#export EDITOR= v i## A l i a se s
#a l i a s rm='rm i 'a l i a s mv= 'mv i 'a l i a s c p = ' c p i 'a l i a s xe= 'xemacs 'a l i a s myps= ' ps u jonesm Lf 'a l i a s m yj ob s = ' q s t a t a | g r ep j on es m 'a l i a s p r o j = ' cd / p ro j e ct s / j o ne sm 'a l i a s w i e n = ' cd / pr oj ec ts / jonesm/ d_wien / wien2k / u2 '
a l i a s mod= ' module 'a l i a s ml = 'mo du le l o a d 'a l i a s mu= ' module unload 'a l i a s m ls = ' module l i s t '
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 30 / 52
Miscellaneous Advanced Features Loading Other Scripts
Loading Other Scripts
-
7/29/2019 class10_Bash Shell scripting.pdf
27/46
Loading Other Scripts
You can load other bash scripts (or just files containing bashcommands) using the source command.
#
# l o a ds I n t e l c o mp i le r e nv ir on me nt. / o p t / i n t e l / c om po se rx e / b i n / c o m p i l e r v a r s . s h i n t e l 6 4
Sometimes you will see the shorthand . used instead of source (same
meaning).
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 31 / 52
Miscellaneous Advanced Features Regular Expressions
Regular Expressions
-
7/29/2019 class10_Bash Shell scripting.pdf
28/46
Regular Expressions
Regular expressions (REs) occur very frequently in UNIX, especially in
sed, awk, and grep. If you have done any significant work in Perl,of course, you are likely also well versed in REs.
REs are sets of (meta)characters used for pattern matching in text
searches and string manipulation, and contain one (or more) of:characters - retain their literal meaning
anchors - designate position in the text line, ^ (beginning) and $
(end)
modifiers - expand or narrow the range of text to search, includes ,brackets, and the backslash
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 32 / 52
Miscellaneous Advanced Features Regular Expressions
-
7/29/2019 class10_Bash Shell scripting.pdf
29/46
More on modifiers in REs:
*, matches any number of the character string or RE preceding it
., (dot) matches any non-newline character^, (carat) matches beginning of line, but also for negation (context dependent)
$, matches end of line
[,], brackets enclose a set of characters to match, including negation with ^ and rangeswith -
\, backslash escapes a special character (e.g. $)
\, escaped angle brackets match word boundaries
?, question mark matches zero or one of the preceding RE, used mostly for singlecharacters
+, matches one or more of the previous RE (like *, but does not match zero occurrences)
\{,\}, escaped curly braces indicate number of previous REs to match
(), parentheses enclose a group of REs
|, the or RE operator used in matching alternates
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 33 / 52
Miscellaneous Advanced Features Regular Expressions
RE Examples
-
7/29/2019 class10_Bash Shell scripting.pdf
30/46
RE Examples
Used in the context of grep just as an example:
g r ep " 2 22 4* " t e x t f i l e # m at ch es 222 f o l l o w e d by a ny number o f 4 s# e . g . 222 , 2224 , 2 2244 , . . .
grep " 2 4. " t e x t f i l e # m atc hes 24 f o l l o w e d by any c h a r ac t e r , b u t n o t 24grep " ^A" t e x t f i l e # matches any l i n e b eg i nn in g w i t h Agrep " A$ " t e x t f i l e # matches any l i n e e nd in g w i t h A
grep " ^$ " t e x t f i l e # m atc hes b l an k l i n e sgr ep " [ a bc123 ] " t e x t f i l e # matches any o f t he c h a r ac t e r a , b , c , 1 , 2 , 3g re p " [ ac13] " t e x t f i l e # same t h i n g u s i n g r a ng esg re p " [ ^ ac ] " t e x t f i l e # m atches a l l c h a ra c t e rs e xc ep t acgr ep " [ Yy ] [ Ee ] [ Ss ] " t e x t f i l e # c as e i n s e n s i t i v e match f o r YES , yes , Yes , . . .grep " \ $ \ $ " t e x t f i l e # match $ $grep " \ < yes \ > " t e x t f i l e # m atches o n ly d i s t i n c t word y es
REs are a rich topic in their own right ...
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 34 / 52
Miscellaneous Advanced Features Here Documents
Here Documents
-
7/29/2019 class10_Bash Shell scripting.pdf
31/46
Here Documents
So-called here documents are ways to pass text on-the-fly into
programs or other code blocks. Its another form of I/O redirection, buta very handy one for running programs:
. / a . ou t
-
7/29/2019 class10_Bash Shell scripting.pdf
32/46
Basic Job/Process Control
You have the ability to stop and continue your processes from a fairlylow level to more sophisticated tools like screen. The basics are
controlled through the notion of a job, created by putting a processasynchronously into the background using the & operator, or C-z
(control-z), the bg and fg commands place the process intobackground or foreground, jobs will list all jobs:
[ u 2 : ~ ] $ e m a c s t e s t . c^Z # c o n tr o lz suspends j ob , j o b i d g iv en i n s qu ar e b r ac k e ts[ 1 ] + Stopped emacs t e s t . c[ u 2 : ~ ] $ bg # p u t i n t o b ac kg ro un d , now r u n ni n g s e p a r a t e l y f ro m t e r m i n a l[1 ]+ emacs t e s t . c &[ u 2 : ~ ] $ jobs # l i s t s r un ni ng j ob s[ 1 ] + Running emacs t e s t . c &[ u 2 : ~ ] $ fgemacs t e s t . c # b r in g back t o f or eg ro un d i n t e r m i na l
[ u 2 : ~ ] $ k i l l %1 # d i e j o b 1 , d i e
n ot e % f o r j o b i d
kill terminates jobs and processes (no % sign for process ids), if you
have a stuck process (identified with the ps commands) you may needto specify a higher kill level like kill -9 pid.
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 36 / 52
Miscellaneous Advanced Features Job Control
Stuff Not Covered
-
7/29/2019 class10_Bash Shell scripting.pdf
33/46
Stuff Not Covered
In the context of a bash survival guide, I skipped quite a bit of stuff:
Argument processing (getopts, shift) ...
List-oriented data structures
Built-in functions (only talked about a small subset)
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 37 / 52
Miscellaneous Advanced Features Additional Resources
More Resources on bash
-
7/29/2019 class10_Bash Shell scripting.pdf
34/46
More Resources on bash
Man page for your particular bash (definitive for your version):man bash
bash Reference Manual:
http://www.gnu.org/software/bash/manualBeginners guide to bash (LDP):http://tldp.org/LDP/Bash-Beginners-Guide
Advanced bash Scripting (LDP):
http://tldp.org/LDP/abs
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 38 / 52
Miscellaneous Utilities (not just for bash) viewers
Viewers
http://www.gnu.org/software/bash/manualhttp://tldp.org/LDP/Bash-Beginners-Guidehttp://tldp.org/LDP/abshttp://tldp.org/LDP/abshttp://tldp.org/LDP/Bash-Beginners-Guidehttp://www.gnu.org/software/bash/manual -
7/29/2019 class10_Bash Shell scripting.pdf
35/46
e e s
You do not need (and often do not want) to use an editor just to look at
a file (or a long listing of results, files in a directory, etc.), soLinux/UNIX provides viewing utilities that you can use:
head - prints the first N (default 10) lines of a file, e.g., head-30 filename
tail - prints the last N (default 10) lines of a file, e.g., tail-30 filename
cat - catenates a file, i.e. just spits it out to standard output(-n shows line numbers, -t shows non- printing charactersand tabs)
more - file filter, with pagination commands like viless - alternative to more intended to be faster (does not
attempt to parse the whole file) and with more freedom ofmovement. Cf. man less
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 40 / 52
Miscellaneous Utilities (not just for bash) sed
sed
-
7/29/2019 class10_Bash Shell scripting.pdf
36/46
sed, the stream editor, is one of the oldest of the UNIX editors (and
therefore probably the least user-friendly). It has a lot of power,though, and you will still find it heavily utilized in shell scripts. Somevery simple examples:
[ u 2 : ~ ] $ echo ' H e l l o , w o r l d ! ' # s e l f e x pl an a to r y ( n ot e s i n g l e q uot es )#
[ u 2 : ~ ] $ echo ' H e l lo , w o rl d ! ' | sed s / Hel lo / Goodbye/Goodbye, world ![ u 2 : ~ ] $ echo ' H e l lo , w o rl d ! ' | sed " s / He l l o / Goodbye , cr u e l / "Goodbye , cru e l , wo rl d ! # n o t e q u ot e u sag e
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 41 / 52
Miscellaneous Utilities (not just for bash) grep
grep
-
7/29/2019 class10_Bash Shell scripting.pdf
37/46
g p
grep is a handy little utility to perform searches on files for lines
containing a given string (the origin of the name dates back to UNIXsoriginal command editor, ed, namely g/re/p.Simple examples:
[ u 2 : ~ ] $ w > t m p . w # dump o u t pu t o f w t o a t em po ra ry f i l e , c f . ' man w '[ u 2 : ~] $ g re p $USER tmp .w # my c u r r en t l o g i ns t o t h i s machine[ u 2 : ~ ] $ g re p j o eu s e r tmp . w # j oe us e r ' s l o g i n s t o t h i s machine[ u2 : ~ ] $ w | g r ep $USER # we d on ' t need no s t i n k i n ' t em po ra ry f i l e s . . .
man grep for handy options (notable, -i and -n).
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 42 / 52
Miscellaneous Utilities (not just for bash) awk
awk
-
7/29/2019 class10_Bash Shell scripting.pdf
38/46
awk is a very powerful utility - it harbors a complex programming
environment in its own right, but you are free to use only as much of itas you want or need to at a given time. The general syntax for awklooks like:
awk ' / p a t t e r n / { a c t i o n } ' f i l e
a given action will be performed on every record which matches thegiven /pattern/. A very handy list of the fields in the record are
given as variables, $0 (entire record), $1 (first field), $2 (second field).... Lets try an example:
[ u2 : ~ ] $ q s ta t a > t m p . q # r ec or d c u r r en t s t a t e o f t he queues
[ u 2 : ~ ] $ g re p x d ta s tmp . q # our f r i e n d grep[ u 2 : ~ ] $ awk ' / x d ta s / { p r i n t $0 } ' tmp . q # same t h i n g w i t h awk
# more i n t e r e s t i n g e xam ple[ u 2 : ~ ] $ awk ' BEGIN { c o r es = 0 } / x d t a s / { c o r es += $7 } END { p r i n t c o r es } ' t mp . q
# add nod es , j o b s ?
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 43 / 52
Miscellaneous Utilities (not just for bash) awk
Exercise
-
7/29/2019 class10_Bash Shell scripting.pdf
39/46
Modify the preceding awk example to also count and report the
number of jobs and nodes as well as cores.
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 44 / 52
Miscellaneous Utilities (not just for bash) find
find
-
7/29/2019 class10_Bash Shell scripting.pdf
40/46
find is quite a handy command for searching for files in complexdirectory layouts - see man find for various options, the very basic
syntax looks like:
f i n d [ p a t h ] [ e x p r e s s i o n ]
where the expression involves options for just about every possible file
property that you might imagine - here are a few examples:# f i n d a l l f i l e s h e r e i n w i t h s u f f i x . nc[ u 2 : / s c r a t c h / jo nes m ] $ f i n d . name \* . n c p r i n t# n ar ro w t h e s ea rc h down a b i t[ u 2 : / s c r a t c h / jo nes m ] $ f i n d . name 2010\*. nc p r i n t# l o n g l i s t ea ch one[ u 2 : / s c r a t c h / jo nes m ] $ f i n d . name 2010\*. nc exec l s l { } \ ;# show me f i l e s i n d an ge r o f b ei n g s c r ub be d f ro m my panas as s c r a t c h d i r e c t o r y[ u 2 : ~ ] $ f i n d / p anas as / s c r a t c h / jo nes m a t i me +23
Note that the xargs command is also helpful when you want to dosomething useful with copious amounts of output from a command like
find.
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 45 / 52
Miscellaneous Utilities (not just for bash) And many more
Other utilities
-
7/29/2019 class10_Bash Shell scripting.pdf
41/46
There is a zoo of other useful commands/utilities, here is a samplingbased on what I have found to be useful:
file determines a files type, typically text, data, or executable, but can also identifyother common file types (e.g., it will recognize windows file types).
ldd prints shared library dependencies, useful for checking to see that alldependencies are satisfied.
cut slices files in a similar fashion to awk, but does so in a less complex way. Youcan specify bytes (-b), delimiters (-d), character position (-c), or fields (-f) from
which to extract.paste merges lines from files (or standard input).
join more flexible version of paste for sources containing common fields.
sort handy sorting utility (lexical, -n numeric, etc.).
split splits files into pieces, by lines (-l) or by bytes (-b) or a maximum of some bytesper line (-C).
tr translates or deletes characters, very useful for rapidly cleaning up files(common example, tr -d \r < infile.csv > outfile.csv, toremove windows carriage returns from a file)
uniq omit (or report) repeated lines.
wc newline (-l), word (-w), character (-m), or byte (-c) counts.
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 46 / 52
Miscellaneous Utilities (not just for bash) More CLI Resources
More CLI Resources
-
7/29/2019 class10_Bash Shell scripting.pdf
42/46
List of CLI related cheat sheets:http://www.cyberciti.biz/tips/linux-unix-commands-cheat-sheets.html
The Linux documentation project:http://tldp.org/LDP/GNU-Linux-Tools-Summary/html/GNU-Linux-Tools-Summary.html
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 47 / 52
Examples Simple Tool Script
Simple Tool Script
http://www.cyberciti.biz/tips/linux-unix-commands-cheat-sheets.htmlhttp://tldp.org/LDP/GNU-Linux-Tools-Summary/html/GNU-Linux-Tools-Summary.htmlhttp://tldp.org/LDP/GNU-Linux-Tools-Summary/html/GNU-Linux-Tools-Summary.htmlhttp://www.cyberciti.biz/tips/linux-unix-commands-cheat-sheets.html -
7/29/2019 class10_Bash Shell scripting.pdf
43/46
Shell scripts are quite commonly used when deploying other tools -here is an example of one that is used to wrap a Java application
called Panoply1
1 [ u 2 : ~ ] $ l s p an op ly / P an op ly J2 c o l or b ar s j a r s o v er l ay s Panoply . j a r p ano pl y . s h README. t x t
3 [ u2 : ~ ] $ ca t panoply / PanoplyJ / panoply . sh4 # ! / b i n /sh5 #67 SCRIPT = ` r e a d l i n k f $0 `8 SCRIPTDIR=` dirname $SCRIPT`9
10 j a v a Xmx1GXms128m j a r $SCRIPTDIR / j a r s / Pan op ly . j a r "$@"
1http://www.giss.nasa.gov/tools/panoply
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 49 / 52
Examples Example PBS Script
Example PBS bash Script
http://www.giss.nasa.gov/tools/panoplyhttp://www.giss.nasa.gov/tools/panoply -
7/29/2019 class10_Bash Shell scripting.pdf
44/46
A more complex example script for a scientific application on thecluster, which makes use of various utilities that we have discussed.
1 #PBS
S / b i n / b ash2 #PBS q debug3 #PBS l 4 : IB2 : p pn =84 #PBS l w a l l t i m e = 0 1 :0 0 :0 05 #PBS M jonesm@ccr. buffalo .edu6 #PBS m e7 #PBS j oe8 #PBS N t i t a n _ 3 2 p9 #PBS o p b s _ s c r i p t . o u t
10 #PBS
W g r o u p _ l i s t = gmfg11 #12 # Se t e xe cu ta b l e name & mp ie xe c p a th13 # EXE = Exe cu ta b l e name14 # I NITDIR = d i r h o ld in g i n p u t & ex ec ut ab le15 # SAVEDIR = d i r t o p la ce o ut pu t f i l e s16 #17 module l oa d t i t a n / i n t e l m p i18 IN IT DI R=$PBS_O_WORKDIR
19 SAVEDIR=$PBS_O_WORKDIR20 #21 # Get l i s t o f p r oc e s so r s f ro m PBS22 p l i s t = ` cat $PBS_NODEFILE | sed " s / \ . ccr \ . b u ff a l o \ . e du / / " `23 u p l i s t = `ca t $PBS_NODEFILE | un iq | sed " s / \ . ccr \ . b u ff a l o \ . e du / / " `24 echo " u ni qu e nodes l i s t = " $ u p l i s t25 echo " f u l l p ro c es s l i s t = " $ p l i s t
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 50 / 52
Examples Example PBS Script
-
7/29/2019 class10_Bash Shell scripting.pdf
45/46
26 # S ta ge o u t i nd ex ed i n p u t f i l e s ( a nd common o nes t o o ) t o eac h27 # p ro ce ss or28 cd $PBS_O_WORKDIR29 NN=130 f o r p i n $ p l i s t ; do
31 l e t "NN = $NN + 1 "32 NUM= `echo $NN | awk ' { p r i n t f "%.4d" , $1} ' `33 echo " sta g i n g o u t funky$NUM . i n p to $p . . . "34 ssh $p " cp $ INITDIR / funky$NUM . i n p $PBSTMPDIR"35 done36 # f i l e s t h a t o nl y n eed t o appear o nce on a node , no m at te r37 # how many p ro cs / n od e38 f o r p i n $ u p l i s t ; do39 ssh $p " cp r $ INITDIR / * . da ta $PBSTMPDIR/ "
40 echo " Contents of "$PBSTMPDIR" on node "$p41 ssh $p " l s l $PBSTMPDIR/ "42 done43 NP= ` cat $PBS_NODEFILE | wc l `44 NP_UNIQ=` cat $PBS_NODEFILE | un iq | wcl `45 #46 # Run code fro m $PBSTMPDIR make s ur e t h a t you c opy back f i l e s47 # t h a t y ou n eed ( $PBSTMPDIR i s e ra s ed a t c o m p l e t i on o f t h e j o b )48 # N .B . you can a ls o run t i t a n from a s hared d i r e c t o ry w / o a l l
49 # t he e x t ra f i l e s ta gi ng , as l on g as t he i / o r eq ui re me nt s a re50 # r e as o n ab l y s m a l l ( a nd y ou do n o t ex ceed , s ay , a bo ut 16p )51 #52 cd $PBSTMPDIR53 mpirun np $NPROCS $EXE
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 51 / 52
Examples Example PBS Script
-
7/29/2019 class10_Bash Shell scripting.pdf
46/46
54 #55 # S tage back o ut pu t f i l e s56 #57 f o r p i n $ u p l i s t ; do58 echo " Conte nts of $PBSTMPDIR on $p: "59 ssh $p " l s l $PBSTMPDIR"60 s sh $p " c p $PBSTMPDIR / x dm f
*$SAVEDIR / ; cp $PBSTMPDIR / pe rf or m
*$SAVEDIR / ; \
61 cp $PBSTMPDIR/ output_summary * $SAVEDIR / ; cp $PBSTMPDIR/ pi l e h e i g ht * $SAVEDIR/ "62 done
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 52 / 52