Stupid Unix TricksorLove Your Command Line!

Liz HenryBlogHer 2009

The command line is like your trusty robot friend always waiting for you to tell it what to do!

How do I get to the command line?

MacOS or Linux: TerminalWindows: HyperTerminal or PuTTY, CygWin

You can get these slides from my blog or we'll download them at end of session, so don't worry about taking notes!

Navigation: Where are you?

pwdPrint(current) working directory

lslist files in the directory you're in. Type this a lot!

cd change directory (blank will take you to your home dir)

mkdir dev Make a directory called dev in your current working directory

cd devchange into that directory! whee!

lsWhat's in there? Probably nothing since you just made it.

pwdwhere are you again?

up arrow keyPush it a few times. It shows the commands you last typed.

Navigation: Where are you?

echo $SHELLWhat command line shell am I in? tcsh, csh, bash?

whoamiSo existential! What's my username?

hostnameWhat's the name of the computer or server I'm on?

ping google.com Are you on the net? (Control-C to stop this)

ifconfig Really, how about that network!? Cryptic...

traceroute google.comWhere am I really? In relation to somewhere else?

easy challenge: Poke around and see what other environment variables you can echo! Hint, type printenv

harder challenge: change your prompt to be in color and tell you what directory you're in (look up later and try it)

Look around some more!

man ps

alias woman=manalias wom=manwom ls Q to quit reading man pages

alias ls = ls -lah

ps

ps -x Whoa!

ps -x | morePipes the output into a paging program Q to quit

ps -x | grep TermPipe into grep to search for specific process

topInfo about top processes (Q to quit)

Get remote files

First use curl or wget to download two sample text files.

curl -O http://bookmaniac.org/stuff/slides/example.tar.zipor wget http://bookmaniac.org/stuff/slides/example.tar.zip

lsThere's the file, tarred and zipped uptar -xvzf example.tar.zip Uncompress the files!lsReassure yourself...cat bloglist.txtTake a look at the file contentscat stats.txt

ls -lahFSrFancy! sorts by file size.

Challenge for later: Download Wordpress and unzip it from the command lineDownload some WP themes and unzip them!

Challenge: Download and install wget if you don't have it. Then, try downloading an entire web site with its files and images 2 levels deep:wget --wait=9 --recursive --level=2 http://example.org/

Manipulate text files

sort bloglist.txtThis sorts on the first field, alphabeticallysort stats.txt

sort stats.txt > tmp.txt Sorts into a new file called tmp.txt

sort stats.txt | uniq > tmp.txt Sorts, then pipes output to uniq, then writes to a file

sort -t',' -k 2,2 stats.txtSorts with comma as field separator, on field 2

sort -t',' -k 2,2 -nrg stats.txt -nrg means general numerical sort, reverse order

There are many more tricks and oneliners, try out all the options for sort, and search for sort examples unix or sort examples command line

Manipulate text files

Manipulate text files Awk is weird but fun

awk '{print $1}' bloglist.txtPrints first field, field separator is a blank space

awk -F"," '{print $1}' stats.txtPrint first field with a comma as field separator

awk -F"," '{print $2 $1}' stats.txtPrint both fields, the other way round

awk -F"," '{print $2 "," $1}' stats.txt Do that again but with a comma separator

awk -F"," '{print $2 "," $1}' stats.txt | sort -nr Pipe it through sort, numerical, reversed!

awk -F"," '{print $2 "," $1}' stats.txt | sort -nr | mail -s 'Stats from me' lizhenry@gmail.comEmail me the results! (Stats from me is the subject line)

Manipulate text files

Manipulate text files grep and regular expressions

Even grep baby talk is incredibly useful! Regular expressions are fantastic.

* is a wildcard. It means match anything

grep Sarah bloglist.txtShows any lines where Sarah appears in that filegrep Sarah *.txtShows any lines containing Sarah in any file in this directorygrep "S" *.txtLines containing a capital S

grep -r sidebar *.php Look in this directory, and all the ones below it, for any php files that contain the pattern sidebargrep -v widget *.cssLook in all the css files in this directory for the word widget

grep -r sidebar *.php | grep -v test As above, but then send the output into another grep filter that will return every line that doesn't match the pattern test

Manipulate text files comm, sort, and join

comm bloglist.txt stats.txtCompare 2 files. Output is in 3 columnsCol 1: lines unique to file 1Col 2: lines uniq to file 2Col 3: lines that appear in both files

head -1 stats.txt >> bloglist.txtStick the 1st line from stats at the end of bloglist.

comm bloglist.txt stats.txtNow there is a line in the 3rd column!

comm -12 bloglist.txt stats.txtSuppress column 1 and column 2.

sort -t"," -k2,2 bloglist.txt > blogs-sorted.txt Sort on urlsort stats.txt > sorted-stats.txtSort this file on url too

Manipulate text files Apache logs

if on blogherista server acct,cdcd exampleslsmore access.logA sample bit of an Apache log file.

awk '{print $1}' access.log | sort | uniq -c | sort -nrpull out the IP numbers of people accessing the site sort them and unique them, counting (-c) the times they appear and sort again numerically, reversed, to see who visits your site most often

Try this with the other fields too. What files are accessed most often? What's the most commonly used browser?

nslookup 195.240.190.227Spy on whoever it is a little.whois 195.240.190.227 Some ISP in the Netherlands is downloading my music.

Challenge: Go through every example on the fabulous Awk one-liners page!http://www.catonmat.net/blog/wp-content/uploads/2008/09/awk1line.txt

Other Useful Stuff to Try

scp filename.txt youraccountname@remoteserver.example.com:path/to/files/ Copy some files from where you are to a remote account.

tar -cvzf target-filename.tar.gz file1.txt file2.txt *.html *.jpg Compress and zip some files.

ftp -i youraccountname@ftp.remoteserver.example.com Handy. -i means you don't have to type y y y at every file it's uploading.

make a bunch of aliases in your .bash_profile or other shell profile.

Understand the drwxrwxrwx stuff from the ls -a command. And learn to use chmod.

Try a lot of grep examples.

Do the Bash for Beginners tutorial.

Have fun!