UNIX Shell Scripting talk

download UNIX Shell Scripting talk

of 52

Transcript of UNIX Shell Scripting talk

  • 8/3/2019 UNIX Shell Scripting talk

    1/52

    UNIX Shell scripting

    Netsoc

    Stephen Shaw

    2011

    http://%[email protected]%3E/
  • 8/3/2019 UNIX Shell Scripting talk

    2/52

    Getting started

    SSH to one of our servers PuTTY: Enter login.netsoc.tcd.ie as the hostname Real operating systems: $ ssh [email protected] NX to cube if you want - all you need is a shell though

    No netsoc account? CS: macneill.scss.tcd.ie Maths servers? Talk to an admin before you leave so you have an account for

    next time

    http://macneill.scss.tcd.ie/
  • 8/3/2019 UNIX Shell Scripting talk

    3/52

    UNIX

    Multi-user, multi-tasking operating system

    Origins in the late 60s - UNICS

    The ancestor of many modern operating systems: BSD AIX Solaris Mac OS X

  • 8/3/2019 UNIX Shell Scripting talk

    4/52

    Kernels

    In most operating systems, the kernel acts as an interfacebetween the machines hardware and the application software

    running on it Linux is a kernel which was developed in the early 90s to

    provide a free alternative to proprietary kernels

    Generally the user doesnt interact directly with the kernel

  • 8/3/2019 UNIX Shell Scripting talk

    5/52

    Shells

    A shell is a user-friendly, high-level wrapper around thekernel

    Some shells: sh

    bash ksh tcsh csh

    bash is one of the more popular shells

    This talk will be based on bash

  • 8/3/2019 UNIX Shell Scripting talk

    6/52

    chsh

    Are you using bash?

    echo $SHELL

    If youre not using bash, you can switch to it by runningchsh -s /bin/bash

    Log out, then log back in again

  • 8/3/2019 UNIX Shell Scripting talk

    7/52

    Your prompt

    You should see something like

    1 stesh@cube:~$

    This is called the prompt stesh - username cube

    - hostname ~ - current working directory $ - privilege level

    The format of the prompt is maintained in a variable calledPS1:

    1 stesh@cube:~$ echo $PS1

    2 ${debian_chroot:+($debian_chroot)}\u@\h:\w\$

    Well use $ as a shorthand for the prompt

  • 8/3/2019 UNIX Shell Scripting talk

    8/52

    Variables

    All variables in bash are strings This is both a blessing and a curse

    Variables are assigned values with =

    Variables are evaluated with $

    1 $ foo=bar2 $ echo $foo

    3 ba r

    4 $ echo zanzi${foo}

    5 zanzibar

    6 $ echo "zanzi$foo"

    7 zanzibar

    No spaces around the equals - otherwise its ambiguous(how?)

  • 8/3/2019 UNIX Shell Scripting talk

    9/52

    Special variables

    $RANDOM: random integer

    $$: current PID

    $?: exit status of last process exited

    $!

    : PID of last fork $@: argv

    $0$9: 0th to 9th argument

    $#: number of arguments

    $SHELL: current shell $USER: current user

  • 8/3/2019 UNIX Shell Scripting talk

    10/52

    Quotes

    Quotes are very important in shell scripts Single quotes mean literally:

    1 $ foo='bar'

    2 $ echo '$foo'

    3 $foo

    Double quotes cause variable names in strings to be replacedwith their values:

    1 $ today="Monday"

    2 $ echo "todayis$today"

    3 today is Monday

    This opens up interesting security issues

  • 8/3/2019 UNIX Shell Scripting talk

    11/52

    Backticks

    Enclose a string in backticks, and bash will execute it andreturn a result:

    1 echo $(whoami)

    2 stesh

    3 echo `uptime`

    4 0 5: 07 :5 9 up 1 20 days , 1 6: 17 , 1 15 use rs , l oa d a ve ra ge :

    0.46, 0.47, 0.42

    $() can be easier to read

    But many older versions of many shells dont support it

  • 8/3/2019 UNIX Shell Scripting talk

    12/52

    stdin, stdout, stderr

    Three standard data streams

    stdin: Data going in (buffered)

    stdout: Data coming out (buffered)

    stderr: Warnings coming out (not buffered)

  • 8/3/2019 UNIX Shell Scripting talk

    13/52

    cat

    concatenate Copy stdin to stdout

    Specify filenames as arguments, and cat will copy them tostdout one by one

    Use it to concatenate files together On some systems, cat -n adds line numbers to each line

    printed on stdout

    tac is like cat, but it prints in reverse order:

    1 $ echo "Stephen\nShaw" | tac2 Shaw

    3 Stephen

  • 8/3/2019 UNIX Shell Scripting talk

    14/52

    Pipes

    Pipes make shell scripts really powerful

    connect stdout of one process to stdin of another

    1 $ ls / home | sort | head -n 5 # F ir st f iv e h om e f ol de rs by

    alphabetical order

    2 alxsoky

    3 andyrew

    4 arboroia

    5 at_god

    6 baran

    7

    8 $ ps -ef | grep emacs | grep -v grep | wc -l # How many

    e ma cs u se rs a re t he re ? 9 4

  • 8/3/2019 UNIX Shell Scripting talk

    15/52

    Redirects

    < foo feeds stdin from foo

    > foo redirects stdout to foo

    2> foo redirects stderr to foo 2>&1 redirects stderr to stdout

    1 $ mysql < my_database_backup.mysql

    2 $ top > running_processes.mysql

  • 8/3/2019 UNIX Shell Scripting talk

    16/52

    Fun with redirects

    Silence error messages: find . 2> /dev/null

    Record error messages: find . 2>&1 | less

    Writing our first script quickly:

    1 $ c a t

  • 8/3/2019 UNIX Shell Scripting talk

    17/52

    Conditional execution

    &&for conjunction and

    ||

    for disjunction shells are like most programming languages in that they

    shortcut boolean expressions

    F

    n

    i=0 pi F no matter what each pi is

    T ni=

    0pi T no matter what each pi is

    Abuse this to do conditional execution:

    1 $ t r u e && echo "Hi$USER"

    2 Hi stesh

    3 $ f a l s e && echo "Hi$USER"

    4 $ t r u e || echo "Hi$USER"

    5 $ f a l s e || echo "Hi$USER"6 Hi stesh

    7 $ ./configure && make

  • 8/3/2019 UNIX Shell Scripting talk

    18/52

    if and exit codes

    Processes have exit codes

    They tell you something about the status of the process whenit ended

    Success? Failure?

    You exit a script with exit

    exit followed by zero is true

    exit followed by a non-zero positive integer is false

    1 $ i f ( e x i t 0) ; th en e ch o 'yay!'; f i

    2 yay!3 $ i f ( e x i t 1) ; th en e ch o 'yay!'; f i

    T di i l l di i l

  • 8/3/2019 UNIX Shell Scripting talk

    19/52

    Traditional-style conditionals

    Programs have exit codes

    So why not write a program which turns condition tests intoexit codes?

    [ is such a program. It tests conditions on strings, as well ascharacteristics of files

    1 $ i f [ - e / ho me / st es h ]; then

    2 > ls /home/stesh

    3 > e l s e

    4 > echo "ohno!myhomedirectoryisgone!"

    5 > f i

    [ d l i

  • 8/3/2019 UNIX Shell Scripting talk

    20/52

    [ and logic

    condition true if

    $p $p is not true$p -a $q $p is true and $q is true

    $p -o $q $p is true or $q is true-z $str length of $str is zero-n $str length of $str is greater than zero$a = $b $a and $b are equal

    $a != $b $a and $b differ

    [ d fil

  • 8/3/2019 UNIX Shell Scripting talk

    21/52

    [ and files

    condition true if-e file file exists-f file file exists and is a regular file-d file file exists and is a directory-r file file exists and is readable by me

    -w file file exists and is writable by me-x file file exists and is executable by me-p file file exists and is a pipe

    You have to be careful using these file tests The condition is true as of when it was evaluated

    Race conditions

    L i

  • 8/3/2019 UNIX Shell Scripting talk

    22/52

    Looping

    while:

    1 w h i l e [ -e $lock ]; do

    2 > sleep 1

    3 > done

    for iterates over arguments separated by spaces

    use $() to make things more readable

    1 f o r i in 1 2 3; do

    2 > echo $i

    3 > done

    4 1

    5 26 f o r file in $(ls); do

    7 > du -sh $i

    8 > done

    V b l

  • 8/3/2019 UNIX Shell Scripting talk

    23/52

    Vocabulary

    Now lets run through some fun programs we can glue

    together into scripts

    who

  • 8/3/2019 UNIX Shell Scripting talk

    24/52

    who

    who is logged in, and from where

    1 $ who2 j gi lb er t pts /225 2011 -10 -26 2 1:4 1 ( 46 .7 .7 5. 13 8)

    3 bunburya pts /38 2011 -09 -27 23:58 (: pts /14: S.0)

    4 stesh :1010 2011 -07 -10 19:37 (spoon:s.0)

    5 stesh pts /231 2011 -10 -26 00:11 (:1026.0)

    6 scott :1006 2011 -08 -30 16:56 (89.126.1.54)

    7 arboroia :1016 2011 -10 -25 14:51 (10.6.17.72)

    8 .. .

    When did we last boot?

    1 $ who -b

    2 system boot 2011-06-27 12:51

    How many people are logged in?

    1 $ who -q | grep "#"

    2 # users=130

    w

  • 8/3/2019 UNIX Shell Scripting talk

    25/52

    w

    who is logged in, and what are they running?

    1 $ w

    2 stesh pts/199 89.100.25.137 20:12 0.00s 0.06s 0.00s tmux a

    3 stesh pts/228 :1026.0 Tue23 24:14m 0.67s 0.61s ssh spoon

    4 stesh pts/230 :1026.0 Tue23 24:31m 0.06s 0.06s zsh

    different on BSD Unixes and solaris:

    1 $ w

    2 USER TTY FROM LOGIN@ IDLE WHAT

    3 stesh console - Mer18 6:53 -

    4 stesh s000 - Mer19 1 ssh cube

    w -h removes the header

    last

  • 8/3/2019 UNIX Shell Scripting talk

    26/52

    last

    Login histories

    1 $ who

    2 mloc pts/129 104.76.534.53 Thu Oct 27 00:01 still logged

    in

    3 bunburya pts/222 88.151.27.232 Wed Oct 26 23:17 still

    logged in

    4 m lo c p ts / 19 3 2 02 .1 7. 56 .5 3 We d O ct 26 2 1: 35 g on e - no

    l o g o u t

    5 scott pts/58 89.116.2.54 Wed Oct 26 21:12 - 00:30 (02:12)

    6 t1 pts/129 109.76.162.99 Wed Oct 26 22:16 - 00:06 (01:50)

    7 .. .

    If /var/log/wtmp isnt world-readable, this wont work withoutroot

    finger

  • 8/3/2019 UNIX Shell Scripting talk

    27/52

    finger

    Look up information about a user

    1 $ finger stesh

    2 Login: stesh Name: Stephen Shaw3 Directory: /home/stesh Shell: /usr/bin/

    zsh

    4 .. .

    5 $ finger finger

    6 Login: finger Name: Kieran

    Manning

    7 Directory: /home/finger Shell: /bin/bash8 .. .

    9 $ finger stephen # finger everyone called 'Stephen '

    10 $ f in ge r - m s te sh # f in ge r s te sh in m or e d et ai l

    run touch ~/.nofinger to prevent yourself getting fingered1

    Some servers still allow fingers across the network:

    1 $ finger @maths.tcd.ie

    2 User Real Name What Idle TTY Host

    Console Location

    1but who doesnt want to get fingered?

    uptime

  • 8/3/2019 UNIX Shell Scripting talk

    28/52

    uptime

    How long weve been up, and what the load averages are

    1 $ uptime

    2 0 0: 44 :0 3 up 1 21 day s , 1 1: 53 , 1 30 us er s , l oa d a ve ra ge :

    0.71, 0.62, 0.56

    ps

  • 8/3/2019 UNIX Shell Scripting talk

    29/52

    ps

    Get information about the processes that are currently running

    ps varies widely between operating systems GNU ps:

    1 $ ps -e # a ll p ro ce ss es

    2 $ ps -U stesh # a ll s te sh ' s p ro ce ss es

    3 $ ps -f # f ul l f or ma t

    BSD ps:

    1 $ ps au x # a ll p ro ce ss es

    2 $ ps x # a ll my p ro ce ss es

    Example: harvest passwords from silly people who place themon the command line:

    1 $ w hi l e t ru e ;do p s -ef;done|grep "password" | gr ep -v grep

    2 mysql -u sillyperson --password=RxFLo3YpEd

    xargs

  • 8/3/2019 UNIX Shell Scripting talk

    30/52

    xargs

    Read command-line arguments from stdin and pass them tothe specified program

    1 $ l s ~ | x a r g s d u - h # c al cu la te s iz es f or m y f il es

    2 $ find /srv/webspace/$USER - type d | xa rgs chm od 755 # fi x

    webspace permissions3 $ find /srv/webspace/$USER - type f | xa rgs chm od 644 # fi x

    webspace permissions

    if you dont specify a program, prints an argument list on

    stdout

    cp

  • 8/3/2019 UNIX Shell Scripting talk

    31/52

    cp

    Copy a file

    1 $ cp /etc/motd.tail /etc/motd

    2 $ cp -r /etc /var/backups/etc # r ec ur si ve ly c op y a

    directory3 $ cp - a ~/ D oc s mn t/ sp oo n # p re se rv e a cc es s t im es a nd

    ownership

    4 $ cp -v /home /mnt/backupdrive # no tif y on s tde rr when a

    c op y i s m ad e

    mv

  • 8/3/2019 UNIX Shell Scripting talk

    32/52

    mv

    Move a file

    1 $ mv /var/log/auth.log /var/log/auth.log.1

    2 $ mv -i /etc/profile /etc/passwd # c on fi rm b ef or e m ov in g3 $ mv - n ne w. tx t o ld . txt # don ' t m ov e if o ld . txt e xi st s

    4 $ m v - v # no tif y on s tde rr when a move is made

    rm, rmdir

  • 8/3/2019 UNIX Shell Scripting talk

    33/52

    ,

    remove a file or directory

    1 $ rm / bi n/ rm # o op s

    2 $ rm - r ~/ . Tr as h # r ec ur si ve ly r em ov e a d ir ec to ry3 $ rmdir ~/.Trash # r em ov e a d ir ec to ry , f ai ls i f no n - em pt y

    4 $ rm - rf - -p re se rv e - ro ot / # R ef us e t o d es tr oy s la sh

    grep,fgrep

  • 8/3/2019 UNIX Shell Scripting talk

    34/52

    g p, g p

    Print lines in a file which match a regular expression

    1 $ grep root /etc/passwd

    2 root:x:0:0:root:/root:/bin/bash

    3 $ ps -e | grep tmux

    4 3279 ? 00:06:15 tmux5 4 88 8 p ts / 18 3 0 0: 00 :0 0 t m ux

    6 $ fgrep -i fail /var/log/auth.log # i gn or e c as e

    7 $ la st | g rep -v n et so c # r ev er se t he m at ch

    8 $ last | grep -e '(\d+)\.(\d+)\.(\d+)\.(\d+)' # u s e

    extended regexes

    wc

  • 8/3/2019 UNIX Shell Scripting talk

    35/52

    Count things in a file

    1 $ wc -l /var/log/sshd.log # c ou nt l in es

    2 $ wc - m m yf il e .tx t # count characters

    3 $ wc - b m yf il e .tx t # c ou nt b yt es

    4 $ mv - w m yf il e .tx t # c ou nt t ok en s

    5 $ g re p ":0:0" / et c/ p as sw d | wc - l # t oo r ?

    Archiving and compressing

  • 8/3/2019 UNIX Shell Scripting talk

    36/52

    g p g

    tar - tape archive

    1 $ tar -cf homebackup.tar /home/stesh # ar ch iv e my home

    directory

    2 $ tar -czf homebackup.tar /home/stesh # same , but with

    compression

    3 $ tar -xf homebackup.tar # r es to re f ro m a n a rc hi ve

    4 $ gzip access.log # c om pr es s a f ile

    5 $ gzip -9 access.log # h ig he st c om pr es si on l ev el ( b et we en

    1 and 9)

    6 $ gunzip access.log.gz # decompress

    7 $ zcat access.log.gz # d ec om pr es s a nd o ut pu t t o s td ou t

    pv

  • 8/3/2019 UNIX Shell Scripting talk

    37/52

    p

    Pipe viewer

    Just like cat except it draws a progress bar on stderr

    Monitor the flow of data through a pipe:1 $ pv b ac kup . tgz | tar x

    2 0O 0:00:05 [ 0B/s] [ ]

    sed and tr

  • 8/3/2019 UNIX Shell Scripting talk

    38/52

    sed - Stream editor

    modify input line-by-line

    a silly example: replace all the colons in /etc/passwd withhyphens:

    1 $ c a t /etc/passwd | sed "s/:/-/g"

    tr - Transliterator

    modify input character-by-character1 $ c a t ls / home | tr '\n ' ' ' # r ep la ce n ew li ne s w it h

    spaces2 $ finger stephen | tr -s ' ' # ' sq ue ez e ' m ul ti pl e s pa ce s

    into one

    head and tail

  • 8/3/2019 UNIX Shell Scripting talk

    39/52

    Output the first and last few lines of a file1 $ man ssh | head

    2 $ h ea d - n 5 / etc / sh ad ow # f ir st 5 l in es3 $ last | tail -n 10 # l as t 10 l in es

    4 $ tail -f /var/log/userweb.log # w at ch fo r ne w w ri te s

    sort

  • 8/3/2019 UNIX Shell Scripting talk

    40/52

    Sort lines of input1 $ who | sort

    2 $ s or t - g m yf il e # sort numerically

    3 $ s or t - r m yf il e # r ev er se o rd er 4 $ s or t - u m yf il e # do n ' t p ri nt d up li ca te s

    5 $ d f - h | s o r t - h # s or t h um an - r e ad ab le q ua nt it ie s ( 1G , 2 K

    , e tc .)

    shuf

  • 8/3/2019 UNIX Shell Scripting talk

    41/52

    Shuffle lines of input1 $ who | sort

    2 $ shuf / etc /passwd | head -n 1 | cut -d ':' -f 1 | # a

    random user

    3 $ shuf /usr/share/dict/words | head -n 1 # a r and om word

    from the dictionary

    cut

  • 8/3/2019 UNIX Shell Scripting talk

    42/52

    Tokenize lines of data on a given delimiter

    modify input character-by-character1

    $ cut - d ': ' - f 1 / etc / pa ss wd # l is t th e u se rn am es in / et c/passwd

    2 $ ps - e f | c u t - d ' ' - f 2 , 3 , 4 # t he se con d , th ird , an d

    forth space-delimited tokens

    3 $ cut - c 100 ~/. plan # t he f ir st 1 00 c ha ra ct er s

    comm, diff, uniq

  • 8/3/2019 UNIX Shell Scripting talk

    43/52

    comm prints lines common to two files

    diff shows the differences between two files

    uniq shows the unique lines in a file1 $ ps - e f | c u t - d ' ' - f 1 | s o r t | u n i q2 $ comm /etc/ssh/ssh_config ~/.ssh/config

    3 $ diff myfile.txt myfile.txt.old

    comm and diff work on adjacent lines only

    You get unexpected results if the input lines are not sorted

    perl

  • 8/3/2019 UNIX Shell Scripting talk

    44/52

    Perl is a general-purpose, interpreted programming language

    It is used a lot in text processing and system administration Very powerful regular expressions

    Regular expressions for mathematicians

  • 8/3/2019 UNIX Shell Scripting talk

    45/52

    Formal language theory

    Mathematicians and computational linguists use regularexpressions to define regular sets

    The same expressive power as regular grammmars

    All regular expressions have a generatively-equivalentfinite-state automaton

    This is usually irrelevant for the purpose of shell scripting

    Use to match patterns in text

    Can also perform limited amounts of parsing

    Some regular expressions

  • 8/3/2019 UNIX Shell Scripting talk

    46/52

    Expression Recognizes

    a a single occurrence ofa. a single occurrence of any character

    a* zero or more occurrences ofaa+ one or more occurrences ofa

    a|b a single occurrence a or ofb (but not both)ab a single a followed by a single b

    ab? a single a, optionally followed by a single b

    cron

  • 8/3/2019 UNIX Shell Scripting talk

    47/52

    cron lets you schedule tasks to run at particular times

    crontab -l to view your cron table

    crontab -e to edit your cron table crontab -lu user to view users cron table (requires root)

    1 $ c ro nt ab - l

    2 # m h dom mon dow command

    3

    4 # h ou rl y b ac ku ps t o s po on

    5 @hourly /home/stesh/bin/hourly -backups6

    7 # d ai ly b ac ku ps f ro m CS

    8 30 4 * * * /home/stesh/bin/daily-backups

    Its often good to end a cron entry with 2>&1 >/dev/null

    Otherwise cron daemon will send emails about your cronjob It is good manners not to schedule a big cron job during peak

    hours

    Notice how my big daily backup job runs at 4:30 in the

    morning

    nc

  • 8/3/2019 UNIX Shell Scripting talk

    48/52

    netcat

    copy stdin to stdout over a network

    1 $ c a t m yf il e .t xt | nc - lp 9 99 9 # s er ve m yf il e .t xt on p or t

    9999

    2 $ nc localhost 9999 > myfile.txt.copy

    3 $ nc -z spoon.netsoc.tcd.ie 22 # is port 22 open on sp oon ?

    4 $ nc -z s poon 1 -1000 # w hich por ts be tw een 1 and 1000 are

    o pe n o n s po on ?

    nc is useful in all sorts of situations

    the TCP/IP swiss army knife

    Example: backup

  • 8/3/2019 UNIX Shell Scripting talk

    49/52

    I want to upgrade a lot of packages on spoon, so I should takea backup of /etc/ in case something goes wrong.

    I need to store the backup on a remote machine The remote machine isnt as physically secure as spoon.

    Example: backup

  • 8/3/2019 UNIX Shell Scripting talk

    50/52

    Use tar and gzip to consolidate /etc into an archive andcompress it.

    Encrypt the archive using the GNU privacy guard (gpg) Use ssh to transfer the file securely to the remote machine

    we can write a script to automate this

    Example: backup

  • 8/3/2019 UNIX Shell Scripting talk

    51/52

    1 #!/bin/bash

    2 s e t -e # di e if an y c al l e xi st s w it h an e xc ep ti on

    3 ln -s $$ lock || e x i t 1

    4

    5 i f [ ! - e e tc ba ck up . tgz ]; then

    6 tar -czf etcbackup.tgz /etc

    7 gpg -c etcbackup.tgz8 scp etcbackup.tgz.gpg prime.netsoc.tcd.ie:

    9 f i

    10

    11 rm lock

    thoughts?

    Example: backup

  • 8/3/2019 UNIX Shell Scripting talk

    52/52

    thoughts?

    locking is important in admin-style scripts, especially cronjobs

    make sure at most one instance of the script can run at anyone time

    Be careful when using [ file tests

    This implementation creates a few unnecessary files

    We can condense it down to one line:

    1 #!/bin/bash

    2 t ar - c / et c | g zi p - -b es t | gp g - c | ss h p ri me . ne ts oc . tcd .

    ie ">etcbackup.tgz.gpg"

    We can have SSH accept stdin and pass it to stdout on theremote end