Review• Please turn in your homework and practicals• sed

Today

• Regular Expressions– Again!– Review• grep & sed• Again! (The Revenge!)• awk

The Guy Who Draws XKCD• I think I have a Bash problem. What follows is an actual

command from my history.• cat /usr/share/dict/words | fgrep -v "'" | perl -ne

'chomp($_); @b=split(//,$_); print join("", sort(@b))." ".$_."\n";' | tee lookup.txt | perl -pe 's/^([^ ]+) .*/\1/g' | awk '{ print length, $0 }' | sort -n | awk '{$1=""; print $0}' | uniq -c | sort -nr | egrep "^[^0-9]+2 " | awk '{ print length, $0 }' | sort -n | awk '{$1=""; print $0}' | perl -pe 's/[ 0-9]//g' | xargs -i grep {} lookup.txt | perl -pe 's/[^ ]+ //g' | tail -n2

• It’s just so hard to bite the bullet, admit that the problem has grown in scope, and move it to its own Perl/Python script. (P.S. The Guinness Book is wrong. “Conservationalists” is not a real word.)

Quick Searching

• The grep utility is used for matching• Printing to STDOUT – an ‘eyeball’ test or

redirection to another file• grep ‘static string’ file >> redirection• grep ‘metacharacters’ file• And grep starts at the first line, at the beginning

of the row, reads across the row, and as soon as it matches its regex, prints

• Doesn’t modify, only needs to match once

Replacement

• The sed utility is used for ‘substitutions’• Can either execute once (by default), or across

all entries (globally)• Can also print• sed ‘s/from/to/g’ file• sed -n ‘/regex/p’ file

awk is Awkward

• Final regex utility is awk• For use when your data is evenly formatted• Teams.txt last two lines– Seattle Mariners– There might be some guys called the Cubs too– No even formatting

• First few lines– City Teamname– Even formatting

Example - /etc/passwd

• smithj:x:561:561:Joe Smith:/home/smithj:/bin/bash• smith:*:100:100:8A-74(office):/home/smith:/usr/bin/sh• jsmith:x:1001:1000:John Smith:/home/jsmith:/bin/sh

• Between each entry is a :• Username:Password:UID:GID:Name:Homedir:Shell• Empty entries are still “bounded” (ie, :: indicates one

empty value)

awk Examples

• awk -F ‘:’ ‘{ print $1 }’ /etc/passwd• awk - command• -F ‘:’ - -F sets our ‘delimiter’ to the : character• ‘{ print $1 }’ - action• /etc/passwd - file to use• Prints the first column in the /etc/passwd file– Numbering starts at 1; 0 is the whole file– /etc/passwd is ‘delimited’ by : and empty values still

have a : around them

Advanced awk

• I will not quiz you on this• The more powerful awk commands are in

scripts– #!/bin/bash– awk ‘\– BEGIN { print “File\tOwner” } \– { print $0, “\t”, $3} \– END { print “ – DONE –” } \– ‘

Case Study

• I used awk yesterday• Something close to • awk -F ‘,’ ‘{ print $1\tprint $7}’ example.csv

| sort \• | uniq -c | sort -nr• Counting audit findings so that we could see

where to concentrate

Unfortunately

• I had to reimage my laptop over the weekend• This wiped out our vm on my system• I have basic commands, but they’re not

guaranteed to work• There may be one question on awk on the

final

Finally

• I have yet to see my son play T-ball• I will be “sick” a week from today (Wednesday,

June 4th) so I can see his last game

• All late/resubmitted work is due June 9th

• Final is Monday, June 16th during normal class time

• Just under 50 questions, similar to quizzes

Own Study

• Regex’s• Grep, sed and awk