Chapter Five

Advanced File Processing

2

Objectives

Use the pipe operator to redirect the output of one command to another command

Use the grep command to search for a specified pattern in a file

Use the uniq command to remove duplicate lines from a file

3

Objectives

Use the comm and diff commands to compare two files

Use the wc command to count words, characters and lines in a file

Use the manipulate and format commands: sed, tr, and pr

4

But first…

We need to get a common file we can all work on Type the following commands:wget –U “” http://wildbill.org/rose/gettysburg.txt …and hit enterwget –U “”

http://wildbill.org/rose/Fall07/Chapter05.zip

…and hit enter

5

Advancing YourFile Processing Skills

The select commands, which extract data

6

Advancing YourFile Processing Skills

The manipulation and transformation commands alter and transform into useful and appealing formats data

7

Using the Select Commands

Select commands: grep, diff, uniq, comm, wc

Using Pipes – The pipe operator (|) redirects the output of one command to the input of another command An example would be to redirect the

output of the ls command to the less command

The pipe operator can connect several commands on the same command line

8

Using PipesUsing pipe operators and connecting commands is useful when viewing directory information

ls /etc | sort –r | less

9

Using the grep Command

Used to search for a specific pattern in a file, such as a word or phrase

grep’s options and wildcard support allow for powerful search operations

You can increase grep’s usefulness by combining with other commands, such as head or tail

10

Using the grep Command

grep can take input from other commands and also be directed to provide input for other commands

grep IBM /etc/termcap | head

11

Using the uniq Command

Removes duplicate lines from a file

It compares only consecutive lines, therefore uniq requires sorted input

uniq has an option that allows you to generate output that contains a copy of each line that has a duplicate

12

Using the comm Command

Used to identify duplicate lines in sorted files

Unlike uniq, it does not remove duplicates, and it works with two files rather than one

It compares lines common to file1 and file2, and produces three column output Column one contains lines found only in file1 Column two contains lines found only in file2 Column three contains lines found in both files

13

Using the diff Command

Attempts to determine the minimal changes needed to convert file1 to file2

The output displays the line(s) that differ

The associated codes in the output indicate that in order for the files to match, specific lines must be added or deleted

14

Using the wc Command

Used to count the number of lines, words, and bytes or characters in text files

You may specify all three options in one issuance of the command

If you don’t specify any options, you see counts of lines, words, and characters (in that order)

15

Using the wc Command

The options for the wc command:

–l for lines

–w for words

–c for characters

16

Using the Manipulate and Format Commands

These commands are: sed, tr, pr

Used to edit and transform the appearance of data before it is displayed or printed

17

About sed

sed is a UNIX editor that allows you to make global changes to large files

Minimum requirements are an input file and a command that lets sed know what actions to apply to the file

sed commands have two general forms Specify an editing command on the command

line Specify a script file containing sed commands

18

Formatting Output

The awk command is used to prepare formatted output

For the purposes of developing a new file-processing application, we will focus primarily on the printf action of the awk command

19

Translating CharactersUsing the tr command

tr copies data from the standard input to the standard output, substituting or deleting characters specified by options and patterns

The patterns are strings and the strings are sets of characters

A popular use of tr is converting lowercase characters to uppercase

20

Using the pr Command toFormat Your Output

pr prints specified files on the standard output in paginated form

By default, pr formats the specified files into single-column pages of 66 lines

Each page has a five-line header, its latest modification date, current page, and five-line trailer consisting of blank lines

21

Using a Shell Script toImplement the Application

Shell scripts should contain: The commands to execute Comments to identify and explain the

script so that users or programmers other than the author can understand how it works

Use the pound (#) character to mark comments in a script file

22

Running a Shell Script

You can run a shell script in virtually any shell that you have on your system

The Bash shell accepts more variations in command structures that other shells

Run the script by typing sh followed by the name of the script, or make the script executable and type ./ prior to the script name

23

Chapter SummaryThe UNIX file-processing commands can be organized into two categories: (1) select and (2) manipulation and transformationThe uniq command removes duplicate lines from a sorted fileThe comm command compares lines common to file1 and file2, and produces output that shows the variances between the twoThe diff command attempts to determine the minimal set of changes needed to convert file1 into file2

24

Chapter Summary

The tr command copies data read from the standard input to the standard output, substituting or deleting characters specifiedThe sed command is a file editor designed to make global changes to large filesThe pr command prints the standard output in pages

25

Chapter Summary

Shell programs should contain commands to execute programs and comments to identify and explain the programs. The pound (#) character denotes comments

Write shell scripts in stages so that you can test each part before combining them into one script. Using small shell scripts and combining them in a final shell script file is an effective way to develop applications