Post on 23-Dec-2015
Unix Comp-145
LECTURE 7: UNIX UTILITIES AND FILTERS
BASED ON: S. DAS, “YOUR UNIX: THE ULTIMATE GUIDE”, 2ND EDITION, MCGRAW HILL, 2006
BROOKDALE COMMUNITY COLLEGE 111/4/2009 rwj
BROOKDALE COMMUNITY COLLEGE 2
UNIX OPERATING SYSTEM
• PIPE FACILITY TO CREATE NEW FUNCTIONS• ACCESS & MANIPULATE FILES• EDITING AND DISPLAY CMDS: CUT/PASTE,
HEAD, TAIL• ADDITIONAL TOOLS
11/4/2009 rwj
BROOKDALE COMMUNITY COLLEGE 3
Pipe Facility to Create New Functions
• PIPE FACILITY COVERTS THE OUTPUT OF ONE FUNCTION OR COMMAND INTO INPUT FOR ANOTHER FUNCTION
• EXAMPLECommand1 [args] | command2 [args] | command3
11/4/2009 rwj
COMMANDS THAT ACCESS & MANIPULATE FILES
$ find$ pg$ more$ less$ wc$ diff$ sort$ uniq
BROOKDALE COMMUNITY COLLEGE 411/4/2009 rwj
$ head$ tail$ cut$ paste$ tr$ grep
FILE LEVEL CONTENT LEVEL
BROOKDALE COMMUNITY COLLEGE 511/4/2009 rwj
• RECURSIVELY EXAMINE A DIRECTORY TREE TO IDENTIFY FILES MATCHING SOME CRITERIA AND THEN INVOKE SOME ACTION.
• SYNTAXfind [pathlist] [selectionCriteria] [action]
• EXAMPLE$ find / -name a.out -print
$ find . -name “*.c” –print OR find . -name ‘*.c’ –print./q6.c./testit/q7.c$
• USEFUL OPTIONS:-name xx xx=name of file looking for.-print Prints selected file on standard output.-inum # Select files matching # where # is a file’s Inode number.
find COMMAND
BROOKDALE COMMUNITY COLLEGE 611/4/2009 rwj
pg COMMAND
• A TERMINAL PAGING PROGRAM ON UNIX AND UNIX-LIKE SYSTEMS FOR PERUSING THROUGH TEXT FILES
• CAN BE USED TO PAGE THROUGH THE OUTPUT OF A COMMAND VIA A PIPE
• ITS INTERFACE IS SIMILAR TO vi, BUT COMMANDS ARE DIFFERENT
• more, & less ARE MORE COMMONLY USED PAGING PROGRAMS
$ pg fileNameNOTES:1. pg is NOT available on this UNIX platform in this lab, instead use more fileName
BROOKDALE COMMUNITY COLLEGE 7
more COMMAND
• A TERMINAL PAGING PROGRAM ON UNIX AND UNIX-LIKE SYSTEMS FOR PERUSING THROUGH TEXT FILES
• USED TO DISPLAY (NOT MODIFY) THE CONTENT OF A FILE 1 SCREEN AT A TIME
• VERY BASIC PAGER, ORIGINALLY ALLOWED ONLY FORWARD NAVIGATION THROUGH A FILE, SOME NEWER IMPLEMENTATIONS SUPPORT LIMITED BACKWARD MOVEMENT$ more [options] [fileName]command | more [/c] [/p] [/s] [/tn] [+n]
more [[/c] [/p] [/s] [/tn] [+n]] < [Path]fileName
more [/c] [/p] [/s] [/tn] [+n] [Path]fileNames11/4/2009 rwj
BROOKDALE COMMUNITY COLLEGE 811/4/2009 rwj
more COMMAND
• Useful Options-num THIS OPTION SPECIFIES AN INTEGER WHICH IS THE SCREEN SIZE (IN LINES).
-d PROMPTS THE USER WITH THE MESSAGE "[PRESS SPACE TO CONTINUE, 'Q' TO QUIT.]" AND WILL DISPLAY "[PRESS 'H' FOR INSTRUCTIONS.]" INSTEAD OF RINGING THE BELL WHEN AN ILLEGAL KEY IS PRESSED.
-l PREVENT A PAUSE AFTER ANY LINE THAT CONTAINS A FORM FEED, USUALLY TREATS ^L (FORM FEED) AS A SPECIAL CHARACTER.
-f CAUSES MORE TO COUNT LOGICAL, RATHER THAN SCREEN LINES (I.E., LONG LINES ARE NOT FOLDED).
-p DO NOT SCROLL. INSTEAD, CLEAR THE WHOLE SCREEN AND THEN DISPLAY THE TEXT.
BROOKDALE COMMUNITY COLLEGE 9
more COMMAND
• Useful Options (cont’d)
-c DO NOT SCROLL. INSTEAD, PAINT EACH SCREEN FROM THE TOP, CLEARING THE REMAINDER OF EACH LINE AS IT IS DISPLAYED.
-s SQUEEZE MULTIPLE BLANK LINES INTO ONE.
-u SUPPRESS UNDERLINING.
+/ THE +/ OPTION SPECIFIES A STRING THAT WILL BE SEARCHED FOR BEFORE EACH FILE IS DISPLAYED. (EX.: more +/PREAMBLE AMENDMENT1.TXT)
+num START AT LINE NUMBER num
11/4/2009 rwj
BROOKDALE COMMUNITY COLLEGE 10
less COMMAND
• A TERMINAL PAGING PROGRAM ON UNIX AND UNIX-LIKE SYSTEMS FOR PERUSING THROUGH TEXT FILES
• USED TO DISPLAY (NOT MODIFY) THE CONTENT OF A FILE 1 SCREEN AT A TIME
• SUPPORTS AN EXTENDED CAPABILITY OVER more BY OF ALLOWING BOTH FORWARD AND BACKWARD NAVIGATION THROUGH A FILE
$ less [options] [fileName]
11/4/2009 rwj
BROOKDALE COMMUNITY COLLEGE 1111/4/2009 rwj
less COMMAND
• Useful Options-g HIGHLIGHTS JUST THE CURRENT MATCH OF ANY SEARCHED STRING. -I CASE-INSENSITIVE SEARCHES. -M SHOWS MORE DETAILED PROMPT, INCLUDING FILE POSITION. -N SHOWS LINE NUMBERS (USEFUL FOR SOURCE CODE VIEWING). -S DISABLES LINE WRAP ("CHOP LONG LINES"). LONG LINES CAN BE
SEEN BY SIDE SCROLLING. -? SHOWS HELP. ng JUMP TO LINE NUMBER n. DEFAULT IS THE START OF THE FILEnG JUMP TO LINE NUMBER n. DEFAULT IS THE END OF THE FILE
BROOKDALE COMMUNITY COLLEGE 12
wc COMMAND
• SHORT FOR WORD COUNT, DISPLAYS 3 SEPARATE COUNTS ABOUT A FILE’S CONTENT IN THIS ORDER:
[# of lines] [# of words] [# of characters] filename
• COMMAND SYNTAX AND RESULTS$ wc [-c | -m | -C ] [-l] [-w] [ fileName ... ] $ command | wc
7 15 67 [fileName] – NO FILE NAME IF INPUT WAS A COMMAND
$
11/4/2009 rwj
BROOKDALE COMMUNITY COLLEGE 1311/4/2009 rwj
wc COMMAND (CONT’D)
• OPTIONS-c COUNT BYTES. -m COUNT CHARACTERS. -C SAME AS -m. -l COUNT LINES. -w COUNT WORDS DELIMITED BY WHITE SPACE CHARACTERS OR
NEW LINE CHARACTERS. DELIMITING CHARACTERS ARE EXTENDED UNIX CODE (EUC) CHARACTERS FROM ANY CODE SET DEFINED BY iswspace()
fileName NAME OF FILE TO WORD COUNT.
BROOKDALE COMMUNITY COLLEGE 1411/4/2009 rwj
diff COMMAND
• COMPARES 2 FILES OR 2 DIRECTORIES AND PRINTS THE LINES THAT ARE DIFFERENT, WHICH MAY BE THE ENTIRE FILES. PRODUCES "DELTA" -- LINES THAT ARE CHANGED OR ABSENT IN
EITHER OF FILES• CAN IDENTIFY WHICH LINES IN ONE FILE THAT
HAVE TO BE CHANGED TO MAKE IT IDENTICAL TO THE OTHER.diff [-b] [-i] [-t] [-w] [-c] [-C] [-e] [-f] [-h] [-
n] [-D string] [-l] [-r] [-s] [-S name] [fileone filetwo ] OR [directoryone directorytwo]
BROOKDALE COMMUNITY COLLEGE 1511/4/2009 rwj
diff COMMAND (CONT’D)
• OPTIONS-e option PRODUCES A SCRIPT OF ONLY a (APPEND), c (CHANGE), AND d
(DELETE) COMMANDS FOR THE EDITOR ed, WHICH CAN BE USED TO CREATE A MODIFIED FILE2 FROM FILE1.
-i IGNOR CASE OF LETTER-t EXPAND TABS INTO SPACES ON OUTPUT-w IGNORE ALL WHITE SPACES (BLANKS AND TABS)-l DISPLAY LONG OUTPUT LISTING-r RECURSIVELY DESCEND THROUGH SUBDIRECTORIES-s DISPLAY FILES THAT ARE THE SAME-S name BEGIN THE DIRECTORY COMPARISON WITH FILE NAME
BROOKDALE COMMUNITY COLLEGE 1611/4/2009 rwj
diff COMMAND (CONT’D)
• OPTIONS (CONT’D)-c[n] DISPLAYS A COMPARISON WITH N LINES OF CONTEXT. THE DEFAULT
VALUE OF N IS 3. THE OUTPUT BEGINS WITH THE IDENTIFICATION AND CREATION DATES OF EACH FILE. EACH CHANGE ENCOUNTERED IS SEPARATED WITH A DOZEN
ASTERISKS (*). LINES REMOVED FROM FILE1 ARE PRECEDED BY A HYPHEN (-). LINES REMOVED FROM FILE2 ARE PRECEDED BY A PLUS (+). LINES CHANGED FROM FILE1 TO FILE2 AND VICE VERSA ARE
PRECEDED WITH AN EXCLAMATION MARK (!).
.
BROOKDALE COMMUNITY COLLEGE 17
cmp COMMAND
• COMPARES TWO FILES AND TELLS YOU WHICH LINE NUMBERS ARE DIFFERENT.
• SYNTAXcmp [-c] [-i N] [-l] [-s] [-v] firstFile secondFile
11/4/2009 rwj
c Output differing bytes as charactersi Ignore differences in the first N bytes of inputl Write the byte number (decimal) and the differing bytes
(octal) for each differences Write nothing for differing files; return exit statuses onlyv Output version info
Compare file1 to file2: cmp file1.txt file2.txt file.txt file2.txt differ: char 1011, line 112
BROOKDALE COMMUNITY COLLEGE 1811/4/2009 rwj
sort COMMAND
• Sorts the lines in a text file using primary and/or secondary “keys” (columns of data).
• COLUMN DATA DELIMITED BY WHITE SPACE• SYNTAX
sort [-b] [-d] [-f] [-i] [-k #][-m] [-M] [-n] [-r]
[-u] [-tdelimiter][+fields] fileName [-o outfile]
command | sort
• EXAMPLE:o SORT A FILE USING THE 5TH COLUMN AND OUTPUT THE RESULT
INTO A NEW FILE.sort –k 5 –t: fileName >> sortedFile
BROOKDALE COMMUNITY COLLEGE 1911/4/2009 rwj
sort COMMAND (CONT’D)
• White Space is displayed to represent one or more unprintable characters.
o EXAMPLES INCLUDE: CTL-H(S) SPACE CHARACTER(S) TAB CHARACTER(S)
BROOKDALE COMMUNITY COLLEGE 2011/4/2009 rwj
sort COMMAND (CONT’D)
• Useful options:-f Case insensitive sort.-k # Sorts based on #th row.
-f #1,#2 Starts sort based on #1st row (field) and ends on #2nd row (field).
-m list Merges sorted file into list.
-n Sorts numerically.
-r Reverses sort order (z to a, instead of a to z).
-u Removes repeated lines (aka, leaves only unique lines in file.
-tdelimiter Changes the default value of delimiter to identify fields (columns) in output file.
-o outfile Places output in file name.
BROOKDALE COMMUNITY COLLEGE 21
• DISPLAYS THE FIRST TEN LINES OF A FILE, UNLESS OTHERWISE STATED
• SYNTAXhead [-#] [-n #] fileName
• EXAMPLE$ head -25 fileName
11/4/2009 rwj
-# The number of lines to display. -n# The number of lines to display.
head COMMAND
BROOKDALE COMMUNITY COLLEGE 2211/4/2009 rwj
• DISPLAYS A FILE’S THE LAST 10 LINES, OR THE # PROVIDED
• USEFUL FOR MONITORING LOGFILE GROWTH• SYNTAX
tail [-#] [-n #] [-f] [-r] [-l] [-b] [-c] fileName
• EXAMPLE tail -25 fileName
-# The number of lines to display. -n# The number of lines to display.-f Permits opening of files currently in use, e.g., logfiles-r Copies lines from the specified starting point in the file in reverse
order. The default for r is to print the entire file in reverse order.
tail COMMAND
BROOKDALE COMMUNITY COLLEGE 2311/4/2009 rwj
• CUTS A COLUMN OF DATA (I.E., FIELDS) OUT OF EACH LINE IN A DELIMITED TEXT FILE.
• SYNTAXcut [[-b] [-c] [-f] list] [-n] [-d delim] [-s] [fileName]command | cut
• EXAMPLE cut –f3 fileName
-b The list following -b specifies byte positions (for instance, -b1-72 would pass the first 72 bytes of each line).
-c Specifies columns to be extract.-f Fields (columns) assumed to be separated in the file by a delimiter
character-d <ch> The character following -d is the field delimiter (-f option only).
Default is tab. Space or other characters with special meaning to the shell must be quoted.
cut COMMAND
BROOKDALE COMMUNITY COLLEGE 2411/4/2009 rwj
• MERGE CORRESPONDING OR SUBSEQUENT LINES OF FILES IN COLUMN’D FORMAT INTO A DELIMITED TEXT FILE.
• PERMITS VIEWING CONTENTS SIDE BY SIDE AS WELL AS REORGANIZING A DELIMITED FILE.
• SYNTAXpaste [-s] [-d list] [fileName]paste fileName1 fileName2 | lpr
• EXAMPLE$ paste fileName1 fileName2 fileName3 > fileName4
paste COMMAND
BROOKDALE COMMUNITY COLLEGE 2511/4/2009 rwj
• OPTIONS-s Concatenate all of the lines of each separate input file in
command line order. The NEWLINE character of every line except the last line in each input file will be replaced with the TAB character, unless otherwise specified by the -d option.
-d <ch> When the option is not specified:• The NEWLINE characters in the file specified by the last file will not be
modified. • Default delimiter is TAB.• If a backslash character appears in list, it has the following semantics:
\n Newline character. \t Tab character. \\ Backslash character. \0 Empty string (not a null character). If \0 is immediately followed
by the character “x” or “X”, the results are unspecified.
NOTE: Any character follow the backslash, the results are unspecified.
paste COMMAND (CONT’D)
BROOKDALE COMMUNITY COLLEGE 2611/4/2009 rwj
uniq COMMAND
• LISTS UNIQUE LINES IN THE INPUT.• SYNTAX
uniq [-c | -d | -u ] [ -f fields ] [ -s char ] [-n] [+m] [inputFile [ outputFile ] ]
• EXAMPLEuniq –u inputFile OR uniq –c inputFilecut –d: -f3 inputFile | sort | uniq –c > outputFile
• USEFUL OPTIONS:-c Count of the frequency of occurrence of all lines along with the line.-u Select only the unique lines found in the input. Suppress the lines that
are repeated.-d Selects only the repeated lines found in input, suppresses unique lines.
NOTE: If 2 input files are included in the command, uniq overwrites the content of the 2nd file.
BROOKDALE COMMUNITY COLLEGE 2711/4/2009 rwj
tr COMMAND
• MANIPULATES INDIVIDUAL CHARACTERS IN A FILE OR STREAM.
• SYNTAXtr [-c] [-d] [-s] [string1] [string2] [< inputFile] [> outputFile]
• EXAMPLEtr ‘:/’ ‘~-’ < inputFile | head –n 3
Changes all ‘:’ to ‘~’ and all ‘/’ to ‘–’ in input1 and displays top 3 lines on std output.
head –n 3 inputFile | tr ‘[a-z]’ ‘[A-Z]Reads and transfers top 3 lines in inputFile to tr which changes all lower case characters to
upper case and displays output on std output.
tr –d ‘:/’ < inputFile | head –n 3Strips all “:” and “/” characters from input1, passes results to HEAD which puts top 3 lines
to std output
tr -cd '\11\12\40-\176' < input1 > output1 Strips all non printable characters from input1 and puts results into output1
BROOKDALE COMMUNITY COLLEGE 2811/4/2009 rwj
tr COMMAND (CONT’D)
• USEFUL OPTIONS:-d ‘st’ Deletes characters found in ‘st’ contained in inputFile.
-s ‘st’ Squeeze characters found in ‘st’ contained in inputFile. Way to eliminate multiple spaces
-c Complement characters.
NOTE: If 2 input files are included in the command, uniq overwrites the content of the 2nd file.
BROOKDALE COMMUNITY COLLEGE 2911/4/2009 rwj
grep COMMAND
• FINDS TEXT WITHIN A FILE OR OUTPUT OF A COMMAND.
• SYNTAXgrep [options] PATTERN [FileName...]
grep [options] [-e PATTERN | -f FileName] [FileName...]
• PATTERNS FOR SEARCHING. Matches any single character. * Wild character. Example C* would find CC or CAT if in the input... {} Matches any character contained within the bracket. ^ Metacharacter that represents anchors the search to the beginning of
the line, e.g., ^T searches for any line starting with a T. $ Metacharacter that represents the end of the line. If use ‘\.$’ then it
would pull up any lines that ended with . (period) \ Means to take the next character as a literal, e.g., search for the string
“D\ D” in a file using ‘D\\ D’.
BROOKDALE COMMUNITY COLLEGE 3011/4/2009 rwj
grep COMMAND (CONT’D)
• TO SEARCH FOR META-CHARACTERS E.G., & ! . * ? AND \, PRECEDE IT WITH A BACKSLASH (\).
• EXAMPLES:grep \\$ file1Search for a $ (dollar sign) in the file named file1
grep -E "abc|xyz" file1Display all lines in the file1 file that match either the string abc or xyz.
grep -v "^#" file1Display all lines that do not match a pattern
grep "^[a-zA-Z]" file1Display every line in file1 whose first character is a letter
ls -l *.ps | grep SepDisplay files that end in “.ps” and created in the month of September