Chapter 3 UNIX Utilities II

By C. Shing

ITEC Dept

Radford University

Objectives

• Understand how to filter files• Understand how to sort files• Understand how to search for files• Understand how to compare files• Understand how to archive, transform and

compress/uncompress files• Understand how to schedule processes• Understand hard and soft links• Understand how to attach different file systems• Understand how to find every character in file

Filters (Find file Content)

• grep (Global or Get Regular Expression and Print), egrep (Extended grep), fgrep (Fast grep)

• uniq

• awk (Aho, Weinberger, Kernighan): C control structures

• sed (Stream Editor)

Filters

I. grep, egrep, fgrep: filter out lines that do/don’t have specified pattern in the file content

1. grep• Syntax: grep -option pattern filename

Option: i: ignore case l: list files containing the pattern n: display line # v: displays lines that don't match the pattern w: match to whole word only

Pattern: regular expression

Filters- grep

Pattern Explain

. match any single character

* The preceding item match 0 or more characters

^a begins with a

a$ end with a

Filters- grep

Pattern Explain

\. match the . character (Do not use meta-character meaning of .)

[a-z] match any character within a to z range

[^a-s] tv match any word not starting or continue within a to s range but end with tv

Filters- grep

• Examples:– grep smith /etc/passwd

show all lines that have smith pattern– grep -wn lower /usr/include/*

use word: lower and show line #– grep -ni lower /usr/include/*

ignore case and show line number– ls -l | grep ^d

show directory– ls -l | grep ‘^..w’

show files that owner can write– cut -d: -f5 /etc/passwd | grep -i ' smith$' | more

show under field 5, any line has word end with smith case insensitive

Filters- egrep

2. egrep• Syntax: egrep -option pattern filename

or grep –e -option pattern filename Option:

i: ignore case l: list files containing the pattern n: display line # v: displays lines that don't match the pattern w: match to whole word only

Pattern: extended regular expression

Filters- egrep

Extended (extra) Pattern

Explain

+ The preceding item match 1 or more characters

? The preceding item match 0 or 1 character

| or

() put regular expression in ()

Filters- egrep

• Examples:– egrep sm.?ith: /etc/passwd

all lines contain 0 or 1 of any character after sm– egrep -wn stdio \| STDIO /usr/include/stdio.h

any line containing stdio or STDIO– egrep -ni to+ /usr/include/* | grep too

all lines contain too– ls -l /devices| egrep –v ^d

show all non-directory files(device has major #: routine #, minor #: index in the

routine)– cut -d: -f5 /etc/passwd | egrep -i ' smith.+$' | more

any line containing smith followed by 1 or more characters

Filters- fgrep

3. fgrep• Syntax: fgrep -option string filename

Option: i: ignore case l: list files containing the pattern n: display line # v: displays lines that don't match the pattern w: match to whole word only

Filters- fgrep

• Examples:– fgrep smith /etc/passwd– fgrep –wn lower /usr/include/*

Filters- uniq

II. uniq: filter out duplicated adjacent lines• Syntax: uniq -option [infile [outfile]]

Option: number: # of fields to be skipped (delimiter: space) c: precedes # of duplications

Filters- uniq

• Examples:– cut –d: -f4 /etc/passwd | uniq -c

list # of same adjacent GID lines

Compare - cmp

• cmp: find 1st byte that is different• Syntax: cmp -option file1 file2

Option:l: list offset and line number of 1st mismatched byteS: all output inhibited

Compare - diff

• diff, sdiff: display differences so file1 can be converted to file2 (sdiff shows actual lines involved)

• Syntax: diff -option file1 file2

or

sdiff -option file1 file2 Option:

i: ignore case

Compare - diff

• Output form:1. File1Line# a File2Line#Start, File2Line#Stop

>

2. File1Line#Start, File1Line#Stop d lineCount

<

3. File1Line#Start, File1Line#Stop c File2Line#Start, File2Line#Stop

<

---

>

Compare - diff

• Example:– diff /etc/auto_net_rucs /etc/auto_net_rucs2– diff /etc/auto_net_rucs /etc/auto_net_ruacad

Sort

• Sort: in ascending order• Syntax: sort –option file

Option: t: c-delimiter

default - space or tab +n1: n1 is the index number for the starting sort field -n2: n2 is the stopping sort field index number (non-

inclusive) b: ignore leading blanks f: ignore case for sort fields n: sort fields in numerical order M: sort fields in month order r: sort in reverse (descending) order +n3, -n4: more comparison when equal in using +n1,-n2

Sort

• Examples:– sort +0 -1 -b /usr/include/stdio.h

starts from and to the 1st field (index=0), ignore leading spaces and sort /usr/include/stdio.h in ascending order (delimiter is space or tab).

– sort -t: +4 -5 -b +0r /etc/passwd | moreUse : as the delimiter of fields, sort in ascending order according the Real Name (index=4) in /etc/passwd, ignore leading spaces; if sorted fields are equal, then sort by the username in descending order

– sort -t: +2 -3n /etc/passwd | moresort by uid (numerically)

Search (Find file/directory name/attribute)

• Find: search the expression from the path• Syntax: find path expression

Search

Expression Explain

-name pattern pattern can use meta-characters such as *

-type ch File type: ch can be d, b, c

-user username or use uid

-group groupname or use gid

-atime -daycount accessed within the previous # of days

-perm oct Permission with oct number

SearchExpression Explain

-mtime -daycount modified within the previous # of days

-ctime -daycount changed within the previous # of days

-print print matched in screen

-ls list file attributes

-exec command executes command when found. If specify {} in command, it is replaced by all found items end with \;

SearchExpression Explain

expr1 -a expr2 expr1 and expr2

expr1 -o expr2 expr1 or expr2

-maxdepth levels Default is recursively to leaf

Search

• Examples:– find ~ -name '*.c' -print | more

It recursively goes down from your home directory to find any c files and print on screen.

– find . -mtime -14 –print | more

It displays all files that has been modified within the past 2 weeks.

- find ~ -perm 4701 –ctime -7

superuser needs check any executable set to suid within last 7 days(maybe hacker)

Search

• Examples:– (find / -name handler.py > /dev/tty) > &

/dev/null

This finds the python file and sends errors to /dev/null, and results are displayed on the terminal, so you don’t see error in screen

– find / -name handler.py >& errorfile

The error will goes to the file errorfile

Search (Cont.)

• Examples: (Cont.)– find . -name a.out -ls -exec rm {} \;

It recursively goes down from the current directory to delete a.out files and print those names on screen. (This rm command may execute 100 times if there are 100 a.out files found)

Note: a more efficient way is to use xargs command as below: (Linux only) find . -name a.out -ls | xargs rm

(This will put all files found in the parameters of the command rm and execute once)

If the command line parameters of rm has limited to 10, then we can use find . -name a.out -ls | xargs –n10 rm

(This will execute the command 10 times, each time it removes 10 files.)

Search (Cont.)

• Examples: (Cont.)– find . \( -name '*.c' -o -name '*.java' \) -

print | moreIt recursively goes down from the current directory to find any c files or java files and print on screen.

Filters - awk

III. awk: scan every line that matches certain criteria (Programmable Text Processing ) or no condition, perform some actions to them

Use C syntax and C library function

• Syntax: each line in form - [condition][{action}]– awkPgm (use –f): line in file

• awk -option -f awkPgm file – Command: line (in quotes)

• awk -option '[condition][{action}]' file

Option: Fc: c – Fields delimiter

tab or space - default delimiter

Filters - awk• Condition

Condition Explain

BEGIN/END pair 1st line before loop is right after the BEGIN, 1st line after loop is right after the END

expression combination of logical operators, relational operators and regular expressions

/regular expr/ Lines having reg. expr pattern

/regular expr1/

, /regular expr2/

the range between lines having reg. expr1 and reg. expr2

Filters - awk

• Action

Action Explain

if (conditional) statement [else statement]

while (conditional) statement

for (expr; conditional; expr) statement

break

continue

var = expr

Filters - awk

• Action

Action Explain

print [list of expr]

printf format[,list of expr]

next

exit

list of statements

Filters - awk

• Examples:(all necessary Unix format files(Do not use DOS format) are in the folder)

1. awk -F: '{print NF, $0}' /etc/passwd|morewhere NF: # of fields (predefined variable),

$0: entire line

2. awk -F: -f awk1 /etc/passwd|moreIt prints 1st, 3rd fields and number of fields (no spaces in between)

Filters - awk

3. awk -F: -f awk2 /etc/passwd | moreIt prints out 1st, 3rd fields and number of fields (with spaces in between) for line 2 and 3(NR: record number)

4. awk -F: -f awk3 /etc/passwd | moreIt prints out line # along with each line and count total # of line s and fields.

5. awk -F: -f awk4 /etc/passwd | moreIt prints out from the last field back to the 1st field for each line.

Filters - awk

6. awk -F: -f awk5 /etc/passwd |moreIt prints out the line that have the pattern (t followed by one or more characters and end with e).

7. awk -F: -f awk6 /etc/passwd |moreIt prints out the line that contains "smith".

8. awk -F: -f awk7 /etc/passwd |moreIt prints out the line that contains "Smith:".

Filters - awk

9. awk -F: -f awk8 /etc/passwd |moreIt prints out all lines in between (inclusively) the beginning line that contains "root" to the ending line that contains "local“

10.awk -f awk9 awktestlibIt shows awk library functions.

Filters - awk

11. Example of using if .. else

Filters - sed

IV. sed: scan every line and perform simple editing action on lines that match certain criteria• Syntax: sed -option scriptfile filename

Option: f: scriptfile is file of command lines

no option: use command line in quotes n: suppress print duplication caused by p e: multiple instructions in command line

Filters - sedCommand Explain

s/expr/str file substitute the first occurrence of the expr by str in the file

s/expr/str/g file substitute all occurrences of the expr by str in the file

/expr1/, /expr2/w newfile

Select all lines such that the 1st one contains expr1 and the last line has expr2 and save them into a newfile

/expr/d file delete every line that contains expr in the file

n1,n2d file delete from line n1 to line n2 in the file

Filters - sedCommand Explain

n1q file Quit after reading the first n1 lines of the file

n1,n2p file prints duplication from line n1 to line n2 in the file (suppress the duplication: must use -n option)

Address i\

text

Insert text before line #address

Address a\

text

Append text after line #address

AddrRange c\

text

Change line #address range by text

Filters - sed

• Examples:(all necessary Unix format files(Do not use DOS format) are in the folder)

1. sed 's/^/\$ /' awk1 > awk1.indentThis gives a $ at the beginning of each line in the file awk1 and creates output file awk1.indent.

2. sed 's/^ *//' awk3This removes leading spaces in each line of the file awk3.

3. sed -f sed1 awk3This removes all lines that contains $ in the file awk3.

4. sed '2,4d' awk9This removes line 2 to line 4 in the file awk9.

Filters - sed5. sed '$d' awk9

This removes last line in the file awk9.6. sed -n '2,4p' awk9

This prints from line 2 to line 4 in the file awk9.7. sed 5q awk9

This prints from line 1 to line 5 in the file awk9. 8. sed –f sedinsert awk9

This inserts 1st and last line.9. sed –f sedchange awk9

This changes the last line.10. sed –f sedreplace awk9

This replace all “printf” by “fprintf”11. sed ‘1,$s/the/THE/g’ awk9

replace the word ‘the’ by ‘THE’ in the entire file awk912. sed –n –e ‘1,2p’ –e ‘$p’ awk9

prints line 1,2 and the last line of the file awk913. sed –n –e ‘/<TABLE>/,/<\/TABLE>/w table.html’ –e ‘/<FRAME>/,/<\/FRAME>/w

frame.html’ entire.htmlextract table and frame parts of the entire.html and save them into individual files

Archive

• cpio (not in MS-DOS): small files

• tar (also available in MS-DOS) : backup on tape - single volume

• dump (or ufsdump): incremental backup can store in many volumes – restore (or ufsstore): recover

Archive - cpio

• cpio• Syntax: cpio –option [file][directory]

Option:o: takes standard input and creates output cpio

format files with those filenames v: displays those filenames i: takes a cpio format file from standard input and

creates those filenames d: creates directory t: creates table of contents p: copies files into the directory l: creates links (instead of copying) in the directory

Archive - cpio

• Examples:

– mkdir tmpcp *.c tmpcd tmpls *.c | cpio -ov > backupIt creates a cpio format output file conatining all c files

– ls -l backuprm -f *.clscpio -i < backupIt recreates all original files from the cpio file.ls -lcd ..rm -rf tmp

Archive - tar

• tar• Syntax: tar –option [tarfilename] filelist

Option:c: creates tar format file f: tarfile r: unconditionally append to the tarfile t: creates table of contents v: verbose output x: extract from tarfile Z: compress file

Archive - tar• Examples:

– tar -cvf tarfile *.ctar all c files to the current firectory

– tar -tvf tarfileread current contents of tarfile

– tar -rvf tarfile *.txtunconditionally adds all txt files to the end of the tarfile

– tar -tvf tarfile | moremkdir tmpmv tarfile tmpcd tmp

– tar -xvf tarfile `tar -tf tarfile | grep filecp.c `It only untar the file filecp.c from the tarfile.

– tar czf tarcompressfileachive and then compress file

Archive – dump/restore

• dump• Syntax: dump level f option dumpfile filesystem

where dumpfile is defaulted to /etc/rmt0

Option:v: verify each volumn w: displays filesystems that need to be dumped

Archive – dump/restore

• Examples:– dump 0 fv /dev/rmt0 /dev/da0

Archive – dump/restore

• restore• Syntax: restore -option f dumpfile files

where dumpfile is defaulted to /etc/rmt0

Option:t: show table of contents x: restore only files specified i: interactive mode

Archive - dump/restore

• Examples:– restore -x f /dev/rmt0 file1.c file2.c

Transform

• tr: translate from string1character set to string2 character set, respectively. If len(string2)<len(string1), pad the last character in string2 as long as possible

• Syntax: tr –option string1 string2 Option:

c: complement of string1 set d: string1 set deleted s: show once for duplicated characters

Tee

• Duplicates input and save it in a file and creates standard output

Example:

who | tee who.txt | wc –l

This saves users in the file who.txt and output number of users in screen

Transform

• Example– tr “[:lower:]” “[:upper:]” < /usr/include/stdio.h |

more (Solaris)

change from lower case to upper case– tr “a-d” “A-D” < /usr/include/stdio.h | more – tr –cs “a-z” “\015” < /usr/include/stdio.h | more

for non-(a-z) character, replace by new-line character only once

Compress/Uncompress

• Compress/Uncompress (System V)• Syntax:

– compress -option file– uncompress file.Z Option:

c: send result to screen, instead of replacing the old file by a .Z file

v: verbose output

Compress/Uncompress

• Examples:– compress –v filename– uncompress –v filename.Z

Compress/Uncompress

• gzip/gunzip • Syntax:

– gzip -option file– gunzip file.gz Option:

c: send result to screen, instead of replacing the old file by a .gz file

f: force

GNU free compress program

Compress/Uncompress

• zip/unzip (Linux)• Syntax:

– zip -option file.zip file1 file2 …– unzip file.zip Option:

r: recursively descend down its tree to the leaves

PKZIP/PKUNZIP compress program

Schedule Process

• Periodic execution: crontabonly users in cron.allow can use if the file existsall users in cron.deny are not allowed to use if the file exists

• One time execution: atsimilar for files: at.allow and at.deny

• One time execution when CPU is available: batche.g. batch < job.sh

Schedule Process - crontab

• crontab: schedule a crontab process periodically

• Syntax: crontab -option Option:

l: lists contents of a registered crontab file e: update and reregister the registered crontab file r: unregister the registered crontab file

Schedule Process - crontab

• A crontab file must be in the Unix form (not DOS):

– minute hour day month weekday command See the file crontab.cron in the folder

Note:1. Avoid to use * for minute field2. All fields:

– List: e.g. 2,4,6– Range: e.g. 2-6/2 (as above)

2-6

Schedule Process - crontab

• Examples:– crontab crontab.cron

This registers the crontab file– crontab -e

This can edit the crontab file and reregister it– crontab -l

This lists the contents of the crontab file– crontab -r

This unregister the crontab file.

Schedule Process - at

• at: schedule a one-time process• Syntax:

– at time < shellscriptThis starts to run the shellscript at the future time.

– at -l

This checks all future jobID.– at -r jobID

This removes the future jobID.

Schedule Process - at

• Examples: (See the file at.csh in the folder)– at 4:30pm < at.csh

This program at.csh will be executed at 4:30pm today.– at now +1 day < at.csh

This program at.csh will be executed at exactly one day from now.

– at 5pmThis sets interactive mode to type in all shell command and finish it be ctrl-d

Unix File System

• Introduction

Link - hard

• Hard link• Syntax: ln filename aliasename

Hard link will creates an alias called aliasname having the same inode number (same file)

Link – hard (Cont.)

• Limitations:1. Can’t link file across different file systems

2. Can’t link directory

Link - hard

• Examples: link directory– mkdir temp– ln *.c temp

This creates hard links to all c files and put them in temp directory

– ls –li– cd temp– ls -li

Link - soft

• Soft link• Syntax: ln -option filename[directory] softlink

Option:s: soft link

Soft link creates a different file (different inode), it can link to different file systems

Note: This is similar to short-cut in Windows.

Link - soft

• Example: usually used to link directory– ln -s /usr/include myincludedir– cd myincludedir– ls -l

Link - soft

• Example: link file– ln file1 file2– ln –s file1 file3– cat file2– cat file3– ls –i file*– rm –f file2– ls –i file*– rm –f file1– ls –i file*– ls –l file*– cat file3file3 becomes a dangling linkCheck their inode and file content in file3

Attach/Detach File System – mount/umount

• Mount/umount: attach/detach file system• Syntax:

– mount -o options devicename mountpoint Option:ro: read only rw: read and write (default)

– mount

list currently mounted devices– umount mountpoint

detach file system

Attach/Detach File System – mount/umount

• Examples: Use jump drive on Red Hat Linux 8

– mkdir /usb– mount /dev/sd0 /usb– mount – cp * /usb– umount /usb

Raw File Content - od

• od: dump non-text file content in octal form• Syntax: od –option filename

Option:c: in character form s[n]: search string of length n (default 3) x: in hexadecimal formb: in Octal form (Linux)

Raw File Content - od

• Examples:– od –c java1.class | more

look at characters in bytecode file java1.class– od –cxs5 a.out | more

look at characters (string length at least 5, also in hexadecimal form) in executable file a.out

– od –bc a.out | more

Misc.

• whoami• cut• ul: transform an underline sequence file to an ASCII

text file– man cut | ul –tdumb > cut.txt

• spell: spell checker• crypt keyvalue < origFile > cryptFile

To encrypt the file origFile.

• crypt keyvalue < cryptFileTo decrypt the file cryptFile.

Misc.

• time commandFind user, system, and elapsed times, CPU % for running the command.

• paste file1 file2

Align 2 files side by side

• nohup command

Do not kill process created by the command when logout

Misc.

• dfcheck usage of mounted file systems

• dos2unix dosfilename unixfilenameconvert dos file to unix file

• unix2dos unixfilename dosfilenameconvert unixfile to dosfile

• last username: show last login time• strings executable: examine executable string contents• dd : create bit by bit image copy• paste file1 file2: put file2 side by side with file 1

Perl

• Introduction

Reference

• Appendix A