UNIX Shell Scripting talk
Transcript of UNIX Shell Scripting talk
-
8/3/2019 UNIX Shell Scripting talk
1/52
UNIX Shell scripting
Netsoc
Stephen Shaw
2011
http://%[email protected]%3E/ -
8/3/2019 UNIX Shell Scripting talk
2/52
Getting started
SSH to one of our servers PuTTY: Enter login.netsoc.tcd.ie as the hostname Real operating systems: $ ssh [email protected] NX to cube if you want - all you need is a shell though
No netsoc account? CS: macneill.scss.tcd.ie Maths servers? Talk to an admin before you leave so you have an account for
next time
http://macneill.scss.tcd.ie/ -
8/3/2019 UNIX Shell Scripting talk
3/52
UNIX
Multi-user, multi-tasking operating system
Origins in the late 60s - UNICS
The ancestor of many modern operating systems: BSD AIX Solaris Mac OS X
-
8/3/2019 UNIX Shell Scripting talk
4/52
Kernels
In most operating systems, the kernel acts as an interfacebetween the machines hardware and the application software
running on it Linux is a kernel which was developed in the early 90s to
provide a free alternative to proprietary kernels
Generally the user doesnt interact directly with the kernel
-
8/3/2019 UNIX Shell Scripting talk
5/52
Shells
A shell is a user-friendly, high-level wrapper around thekernel
Some shells: sh
bash ksh tcsh csh
bash is one of the more popular shells
This talk will be based on bash
-
8/3/2019 UNIX Shell Scripting talk
6/52
chsh
Are you using bash?
echo $SHELL
If youre not using bash, you can switch to it by runningchsh -s /bin/bash
Log out, then log back in again
-
8/3/2019 UNIX Shell Scripting talk
7/52
Your prompt
You should see something like
1 stesh@cube:~$
This is called the prompt stesh - username cube
- hostname ~ - current working directory $ - privilege level
The format of the prompt is maintained in a variable calledPS1:
1 stesh@cube:~$ echo $PS1
2 ${debian_chroot:+($debian_chroot)}\u@\h:\w\$
Well use $ as a shorthand for the prompt
-
8/3/2019 UNIX Shell Scripting talk
8/52
Variables
All variables in bash are strings This is both a blessing and a curse
Variables are assigned values with =
Variables are evaluated with $
1 $ foo=bar2 $ echo $foo
3 ba r
4 $ echo zanzi${foo}
5 zanzibar
6 $ echo "zanzi$foo"
7 zanzibar
No spaces around the equals - otherwise its ambiguous(how?)
-
8/3/2019 UNIX Shell Scripting talk
9/52
Special variables
$RANDOM: random integer
$$: current PID
$?: exit status of last process exited
$!
: PID of last fork $@: argv
$0$9: 0th to 9th argument
$#: number of arguments
$SHELL: current shell $USER: current user
-
8/3/2019 UNIX Shell Scripting talk
10/52
Quotes
Quotes are very important in shell scripts Single quotes mean literally:
1 $ foo='bar'
2 $ echo '$foo'
3 $foo
Double quotes cause variable names in strings to be replacedwith their values:
1 $ today="Monday"
2 $ echo "todayis$today"
3 today is Monday
This opens up interesting security issues
-
8/3/2019 UNIX Shell Scripting talk
11/52
Backticks
Enclose a string in backticks, and bash will execute it andreturn a result:
1 echo $(whoami)
2 stesh
3 echo `uptime`
4 0 5: 07 :5 9 up 1 20 days , 1 6: 17 , 1 15 use rs , l oa d a ve ra ge :
0.46, 0.47, 0.42
$() can be easier to read
But many older versions of many shells dont support it
-
8/3/2019 UNIX Shell Scripting talk
12/52
stdin, stdout, stderr
Three standard data streams
stdin: Data going in (buffered)
stdout: Data coming out (buffered)
stderr: Warnings coming out (not buffered)
-
8/3/2019 UNIX Shell Scripting talk
13/52
cat
concatenate Copy stdin to stdout
Specify filenames as arguments, and cat will copy them tostdout one by one
Use it to concatenate files together On some systems, cat -n adds line numbers to each line
printed on stdout
tac is like cat, but it prints in reverse order:
1 $ echo "Stephen\nShaw" | tac2 Shaw
3 Stephen
-
8/3/2019 UNIX Shell Scripting talk
14/52
Pipes
Pipes make shell scripts really powerful
connect stdout of one process to stdin of another
1 $ ls / home | sort | head -n 5 # F ir st f iv e h om e f ol de rs by
alphabetical order
2 alxsoky
3 andyrew
4 arboroia
5 at_god
6 baran
7
8 $ ps -ef | grep emacs | grep -v grep | wc -l # How many
e ma cs u se rs a re t he re ? 9 4
-
8/3/2019 UNIX Shell Scripting talk
15/52
Redirects
< foo feeds stdin from foo
> foo redirects stdout to foo
2> foo redirects stderr to foo 2>&1 redirects stderr to stdout
1 $ mysql < my_database_backup.mysql
2 $ top > running_processes.mysql
-
8/3/2019 UNIX Shell Scripting talk
16/52
Fun with redirects
Silence error messages: find . 2> /dev/null
Record error messages: find . 2>&1 | less
Writing our first script quickly:
1 $ c a t
-
8/3/2019 UNIX Shell Scripting talk
17/52
Conditional execution
&&for conjunction and
||
for disjunction shells are like most programming languages in that they
shortcut boolean expressions
F
n
i=0 pi F no matter what each pi is
T ni=
0pi T no matter what each pi is
Abuse this to do conditional execution:
1 $ t r u e && echo "Hi$USER"
2 Hi stesh
3 $ f a l s e && echo "Hi$USER"
4 $ t r u e || echo "Hi$USER"
5 $ f a l s e || echo "Hi$USER"6 Hi stesh
7 $ ./configure && make
-
8/3/2019 UNIX Shell Scripting talk
18/52
if and exit codes
Processes have exit codes
They tell you something about the status of the process whenit ended
Success? Failure?
You exit a script with exit
exit followed by zero is true
exit followed by a non-zero positive integer is false
1 $ i f ( e x i t 0) ; th en e ch o 'yay!'; f i
2 yay!3 $ i f ( e x i t 1) ; th en e ch o 'yay!'; f i
T di i l l di i l
-
8/3/2019 UNIX Shell Scripting talk
19/52
Traditional-style conditionals
Programs have exit codes
So why not write a program which turns condition tests intoexit codes?
[ is such a program. It tests conditions on strings, as well ascharacteristics of files
1 $ i f [ - e / ho me / st es h ]; then
2 > ls /home/stesh
3 > e l s e
4 > echo "ohno!myhomedirectoryisgone!"
5 > f i
[ d l i
-
8/3/2019 UNIX Shell Scripting talk
20/52
[ and logic
condition true if
$p $p is not true$p -a $q $p is true and $q is true
$p -o $q $p is true or $q is true-z $str length of $str is zero-n $str length of $str is greater than zero$a = $b $a and $b are equal
$a != $b $a and $b differ
[ d fil
-
8/3/2019 UNIX Shell Scripting talk
21/52
[ and files
condition true if-e file file exists-f file file exists and is a regular file-d file file exists and is a directory-r file file exists and is readable by me
-w file file exists and is writable by me-x file file exists and is executable by me-p file file exists and is a pipe
You have to be careful using these file tests The condition is true as of when it was evaluated
Race conditions
L i
-
8/3/2019 UNIX Shell Scripting talk
22/52
Looping
while:
1 w h i l e [ -e $lock ]; do
2 > sleep 1
3 > done
for iterates over arguments separated by spaces
use $() to make things more readable
1 f o r i in 1 2 3; do
2 > echo $i
3 > done
4 1
5 26 f o r file in $(ls); do
7 > du -sh $i
8 > done
V b l
-
8/3/2019 UNIX Shell Scripting talk
23/52
Vocabulary
Now lets run through some fun programs we can glue
together into scripts
who
-
8/3/2019 UNIX Shell Scripting talk
24/52
who
who is logged in, and from where
1 $ who2 j gi lb er t pts /225 2011 -10 -26 2 1:4 1 ( 46 .7 .7 5. 13 8)
3 bunburya pts /38 2011 -09 -27 23:58 (: pts /14: S.0)
4 stesh :1010 2011 -07 -10 19:37 (spoon:s.0)
5 stesh pts /231 2011 -10 -26 00:11 (:1026.0)
6 scott :1006 2011 -08 -30 16:56 (89.126.1.54)
7 arboroia :1016 2011 -10 -25 14:51 (10.6.17.72)
8 .. .
When did we last boot?
1 $ who -b
2 system boot 2011-06-27 12:51
How many people are logged in?
1 $ who -q | grep "#"
2 # users=130
w
-
8/3/2019 UNIX Shell Scripting talk
25/52
w
who is logged in, and what are they running?
1 $ w
2 stesh pts/199 89.100.25.137 20:12 0.00s 0.06s 0.00s tmux a
3 stesh pts/228 :1026.0 Tue23 24:14m 0.67s 0.61s ssh spoon
4 stesh pts/230 :1026.0 Tue23 24:31m 0.06s 0.06s zsh
different on BSD Unixes and solaris:
1 $ w
2 USER TTY FROM LOGIN@ IDLE WHAT
3 stesh console - Mer18 6:53 -
4 stesh s000 - Mer19 1 ssh cube
w -h removes the header
last
-
8/3/2019 UNIX Shell Scripting talk
26/52
last
Login histories
1 $ who
2 mloc pts/129 104.76.534.53 Thu Oct 27 00:01 still logged
in
3 bunburya pts/222 88.151.27.232 Wed Oct 26 23:17 still
logged in
4 m lo c p ts / 19 3 2 02 .1 7. 56 .5 3 We d O ct 26 2 1: 35 g on e - no
l o g o u t
5 scott pts/58 89.116.2.54 Wed Oct 26 21:12 - 00:30 (02:12)
6 t1 pts/129 109.76.162.99 Wed Oct 26 22:16 - 00:06 (01:50)
7 .. .
If /var/log/wtmp isnt world-readable, this wont work withoutroot
finger
-
8/3/2019 UNIX Shell Scripting talk
27/52
finger
Look up information about a user
1 $ finger stesh
2 Login: stesh Name: Stephen Shaw3 Directory: /home/stesh Shell: /usr/bin/
zsh
4 .. .
5 $ finger finger
6 Login: finger Name: Kieran
Manning
7 Directory: /home/finger Shell: /bin/bash8 .. .
9 $ finger stephen # finger everyone called 'Stephen '
10 $ f in ge r - m s te sh # f in ge r s te sh in m or e d et ai l
run touch ~/.nofinger to prevent yourself getting fingered1
Some servers still allow fingers across the network:
1 $ finger @maths.tcd.ie
2 User Real Name What Idle TTY Host
Console Location
1but who doesnt want to get fingered?
uptime
-
8/3/2019 UNIX Shell Scripting talk
28/52
uptime
How long weve been up, and what the load averages are
1 $ uptime
2 0 0: 44 :0 3 up 1 21 day s , 1 1: 53 , 1 30 us er s , l oa d a ve ra ge :
0.71, 0.62, 0.56
ps
-
8/3/2019 UNIX Shell Scripting talk
29/52
ps
Get information about the processes that are currently running
ps varies widely between operating systems GNU ps:
1 $ ps -e # a ll p ro ce ss es
2 $ ps -U stesh # a ll s te sh ' s p ro ce ss es
3 $ ps -f # f ul l f or ma t
BSD ps:
1 $ ps au x # a ll p ro ce ss es
2 $ ps x # a ll my p ro ce ss es
Example: harvest passwords from silly people who place themon the command line:
1 $ w hi l e t ru e ;do p s -ef;done|grep "password" | gr ep -v grep
2 mysql -u sillyperson --password=RxFLo3YpEd
xargs
-
8/3/2019 UNIX Shell Scripting talk
30/52
xargs
Read command-line arguments from stdin and pass them tothe specified program
1 $ l s ~ | x a r g s d u - h # c al cu la te s iz es f or m y f il es
2 $ find /srv/webspace/$USER - type d | xa rgs chm od 755 # fi x
webspace permissions3 $ find /srv/webspace/$USER - type f | xa rgs chm od 644 # fi x
webspace permissions
if you dont specify a program, prints an argument list on
stdout
cp
-
8/3/2019 UNIX Shell Scripting talk
31/52
cp
Copy a file
1 $ cp /etc/motd.tail /etc/motd
2 $ cp -r /etc /var/backups/etc # r ec ur si ve ly c op y a
directory3 $ cp - a ~/ D oc s mn t/ sp oo n # p re se rv e a cc es s t im es a nd
ownership
4 $ cp -v /home /mnt/backupdrive # no tif y on s tde rr when a
c op y i s m ad e
mv
-
8/3/2019 UNIX Shell Scripting talk
32/52
mv
Move a file
1 $ mv /var/log/auth.log /var/log/auth.log.1
2 $ mv -i /etc/profile /etc/passwd # c on fi rm b ef or e m ov in g3 $ mv - n ne w. tx t o ld . txt # don ' t m ov e if o ld . txt e xi st s
4 $ m v - v # no tif y on s tde rr when a move is made
rm, rmdir
-
8/3/2019 UNIX Shell Scripting talk
33/52
,
remove a file or directory
1 $ rm / bi n/ rm # o op s
2 $ rm - r ~/ . Tr as h # r ec ur si ve ly r em ov e a d ir ec to ry3 $ rmdir ~/.Trash # r em ov e a d ir ec to ry , f ai ls i f no n - em pt y
4 $ rm - rf - -p re se rv e - ro ot / # R ef us e t o d es tr oy s la sh
grep,fgrep
-
8/3/2019 UNIX Shell Scripting talk
34/52
g p, g p
Print lines in a file which match a regular expression
1 $ grep root /etc/passwd
2 root:x:0:0:root:/root:/bin/bash
3 $ ps -e | grep tmux
4 3279 ? 00:06:15 tmux5 4 88 8 p ts / 18 3 0 0: 00 :0 0 t m ux
6 $ fgrep -i fail /var/log/auth.log # i gn or e c as e
7 $ la st | g rep -v n et so c # r ev er se t he m at ch
8 $ last | grep -e '(\d+)\.(\d+)\.(\d+)\.(\d+)' # u s e
extended regexes
wc
-
8/3/2019 UNIX Shell Scripting talk
35/52
Count things in a file
1 $ wc -l /var/log/sshd.log # c ou nt l in es
2 $ wc - m m yf il e .tx t # count characters
3 $ wc - b m yf il e .tx t # c ou nt b yt es
4 $ mv - w m yf il e .tx t # c ou nt t ok en s
5 $ g re p ":0:0" / et c/ p as sw d | wc - l # t oo r ?
Archiving and compressing
-
8/3/2019 UNIX Shell Scripting talk
36/52
g p g
tar - tape archive
1 $ tar -cf homebackup.tar /home/stesh # ar ch iv e my home
directory
2 $ tar -czf homebackup.tar /home/stesh # same , but with
compression
3 $ tar -xf homebackup.tar # r es to re f ro m a n a rc hi ve
4 $ gzip access.log # c om pr es s a f ile
5 $ gzip -9 access.log # h ig he st c om pr es si on l ev el ( b et we en
1 and 9)
6 $ gunzip access.log.gz # decompress
7 $ zcat access.log.gz # d ec om pr es s a nd o ut pu t t o s td ou t
pv
-
8/3/2019 UNIX Shell Scripting talk
37/52
p
Pipe viewer
Just like cat except it draws a progress bar on stderr
Monitor the flow of data through a pipe:1 $ pv b ac kup . tgz | tar x
2 0O 0:00:05 [ 0B/s] [ ]
sed and tr
-
8/3/2019 UNIX Shell Scripting talk
38/52
sed - Stream editor
modify input line-by-line
a silly example: replace all the colons in /etc/passwd withhyphens:
1 $ c a t /etc/passwd | sed "s/:/-/g"
tr - Transliterator
modify input character-by-character1 $ c a t ls / home | tr '\n ' ' ' # r ep la ce n ew li ne s w it h
spaces2 $ finger stephen | tr -s ' ' # ' sq ue ez e ' m ul ti pl e s pa ce s
into one
head and tail
-
8/3/2019 UNIX Shell Scripting talk
39/52
Output the first and last few lines of a file1 $ man ssh | head
2 $ h ea d - n 5 / etc / sh ad ow # f ir st 5 l in es3 $ last | tail -n 10 # l as t 10 l in es
4 $ tail -f /var/log/userweb.log # w at ch fo r ne w w ri te s
sort
-
8/3/2019 UNIX Shell Scripting talk
40/52
Sort lines of input1 $ who | sort
2 $ s or t - g m yf il e # sort numerically
3 $ s or t - r m yf il e # r ev er se o rd er 4 $ s or t - u m yf il e # do n ' t p ri nt d up li ca te s
5 $ d f - h | s o r t - h # s or t h um an - r e ad ab le q ua nt it ie s ( 1G , 2 K
, e tc .)
shuf
-
8/3/2019 UNIX Shell Scripting talk
41/52
Shuffle lines of input1 $ who | sort
2 $ shuf / etc /passwd | head -n 1 | cut -d ':' -f 1 | # a
random user
3 $ shuf /usr/share/dict/words | head -n 1 # a r and om word
from the dictionary
cut
-
8/3/2019 UNIX Shell Scripting talk
42/52
Tokenize lines of data on a given delimiter
modify input character-by-character1
$ cut - d ': ' - f 1 / etc / pa ss wd # l is t th e u se rn am es in / et c/passwd
2 $ ps - e f | c u t - d ' ' - f 2 , 3 , 4 # t he se con d , th ird , an d
forth space-delimited tokens
3 $ cut - c 100 ~/. plan # t he f ir st 1 00 c ha ra ct er s
comm, diff, uniq
-
8/3/2019 UNIX Shell Scripting talk
43/52
comm prints lines common to two files
diff shows the differences between two files
uniq shows the unique lines in a file1 $ ps - e f | c u t - d ' ' - f 1 | s o r t | u n i q2 $ comm /etc/ssh/ssh_config ~/.ssh/config
3 $ diff myfile.txt myfile.txt.old
comm and diff work on adjacent lines only
You get unexpected results if the input lines are not sorted
perl
-
8/3/2019 UNIX Shell Scripting talk
44/52
Perl is a general-purpose, interpreted programming language
It is used a lot in text processing and system administration Very powerful regular expressions
Regular expressions for mathematicians
-
8/3/2019 UNIX Shell Scripting talk
45/52
Formal language theory
Mathematicians and computational linguists use regularexpressions to define regular sets
The same expressive power as regular grammmars
All regular expressions have a generatively-equivalentfinite-state automaton
This is usually irrelevant for the purpose of shell scripting
Use to match patterns in text
Can also perform limited amounts of parsing
Some regular expressions
-
8/3/2019 UNIX Shell Scripting talk
46/52
Expression Recognizes
a a single occurrence ofa. a single occurrence of any character
a* zero or more occurrences ofaa+ one or more occurrences ofa
a|b a single occurrence a or ofb (but not both)ab a single a followed by a single b
ab? a single a, optionally followed by a single b
cron
-
8/3/2019 UNIX Shell Scripting talk
47/52
cron lets you schedule tasks to run at particular times
crontab -l to view your cron table
crontab -e to edit your cron table crontab -lu user to view users cron table (requires root)
1 $ c ro nt ab - l
2 # m h dom mon dow command
3
4 # h ou rl y b ac ku ps t o s po on
5 @hourly /home/stesh/bin/hourly -backups6
7 # d ai ly b ac ku ps f ro m CS
8 30 4 * * * /home/stesh/bin/daily-backups
Its often good to end a cron entry with 2>&1 >/dev/null
Otherwise cron daemon will send emails about your cronjob It is good manners not to schedule a big cron job during peak
hours
Notice how my big daily backup job runs at 4:30 in the
morning
nc
-
8/3/2019 UNIX Shell Scripting talk
48/52
netcat
copy stdin to stdout over a network
1 $ c a t m yf il e .t xt | nc - lp 9 99 9 # s er ve m yf il e .t xt on p or t
9999
2 $ nc localhost 9999 > myfile.txt.copy
3 $ nc -z spoon.netsoc.tcd.ie 22 # is port 22 open on sp oon ?
4 $ nc -z s poon 1 -1000 # w hich por ts be tw een 1 and 1000 are
o pe n o n s po on ?
nc is useful in all sorts of situations
the TCP/IP swiss army knife
Example: backup
-
8/3/2019 UNIX Shell Scripting talk
49/52
I want to upgrade a lot of packages on spoon, so I should takea backup of /etc/ in case something goes wrong.
I need to store the backup on a remote machine The remote machine isnt as physically secure as spoon.
Example: backup
-
8/3/2019 UNIX Shell Scripting talk
50/52
Use tar and gzip to consolidate /etc into an archive andcompress it.
Encrypt the archive using the GNU privacy guard (gpg) Use ssh to transfer the file securely to the remote machine
we can write a script to automate this
Example: backup
-
8/3/2019 UNIX Shell Scripting talk
51/52
1 #!/bin/bash
2 s e t -e # di e if an y c al l e xi st s w it h an e xc ep ti on
3 ln -s $$ lock || e x i t 1
4
5 i f [ ! - e e tc ba ck up . tgz ]; then
6 tar -czf etcbackup.tgz /etc
7 gpg -c etcbackup.tgz8 scp etcbackup.tgz.gpg prime.netsoc.tcd.ie:
9 f i
10
11 rm lock
thoughts?
Example: backup
-
8/3/2019 UNIX Shell Scripting talk
52/52
thoughts?
locking is important in admin-style scripts, especially cronjobs
make sure at most one instance of the script can run at anyone time
Be careful when using [ file tests
This implementation creates a few unnecessary files
We can condense it down to one line:
1 #!/bin/bash
2 t ar - c / et c | g zi p - -b es t | gp g - c | ss h p ri me . ne ts oc . tcd .
ie ">etcbackup.tgz.gpg"
We can have SSH accept stdin and pass it to stdout on theremote end