Some announcements - eecs.yorku.ca file−Shell functionalities (common) −Bourn shell and...

1

Some announcements

• Labtest II next Tuesday (Aug 11) …y ( g )

• Plan: a review lecture on Aug 18, same time, same location− To be confirmed,

− I will keep you updated

Plan for last week, today and next week

• Unix

− Introduction (file system, process, security …)( y , p , y )

− Utilities

− Shell functionalities (common)

− Bourn shell and shell script

2

Last timeOverview of UNIX

StructureFil t File systems

Processes Pipes Shells (briefly) UNIX Philosophy UNIX’s History ….

Utilities Basic cat, rmdir ….. Advanced grep, sort, cut, find, …..

Unix System Structure

user c programsscripts

shell & utilities

kernel

scripts

lsksh

gccfind

open()fork()exec()

hardware

3

UNIX Shell (briefly)• Shell is the (command line) user interface to the operating system

− between you and the raw Unix os

• Functionality:• Functionality:

− Execute other programs

− Manage files, processes

• Interactively

− When you log in, you interactively use the shell

• Scripting

− A set of shell commands that constitute an executable program: a script

datacalecho hello

Files and Processes

• A file is a collection of data that is usually stored on disk, although some files are stored on tape.

• When a program is started, it is loaded from disk into RAM. When a program is running, it is called a process.

RAM DISK

inter-processcommunication

4

Absolute and Relative Pathnames• A process may also unambiguously specify a file by

using a pathname relative to its current working directory.

• UNIX file system supports the following special fieldsthat may be used when supplying a relative pathname:

Field Meaning

. current directory

.. parent directory Same in DOS

cd ..

choose wisely cd../../../../../../../../asssignment1 ???

Relative Pathnames

• Relative Pathnames (from /home/huiwang)

readme.txt or ./readme.txt

../tim/readme.txt

../../bin/readme.txt

/

readme.txt

readme.txt

binhome

timhuiwang

readme.txt

5

Unix Security

• Processes and files have an owner and may be protected against unauthorized access.

• A set of users can form a group. A user can be a member of multiple groups.− You are in group: ugrad, submit, labtest

− use groups to check

• A special user (id 0, name root) has complete control.

• Each user has a primary (default) group.− ugrad (undergrad)

File Permissions (Security)• File permissions are the basis for file security. They are given in

three clusters. In the example, the permission settings are “rw-r--r--”:

1 - rw- r-- r-- 1 glass cs 213 Jan 31 00:12 heart final1 rw r r 1 glass cs 213 Jan 31 00:12 heart.final

User (owner) Group Others

rw- r-- r-- clusters

10

Each cluster of three letters has the same format:

Read permission Write permission Execute permissionr w x

6

File and Directory Permissions UNIX provides a way to protect files based on users and groups.

Three sets of permissions:

permissions for owner

permissions for group

permissions for other

Three types of permissions:

read process may read contents of file

write process may write contents of file

execute process may execute file (used for running script)

Same types and sets of permissions as for files apply to directories:

read process may read the directory contents (i.e., list files)

write process may add/remove files in the directory

execute process may open files in directory or subdirectories ???

Changing File Permissions: examples

Requirement Change parameters

Add group write permission g+w

Remove user read and write permission u-rw

Add execute permission for user, group, a+xand others. Give the group read permission only. g=r

Add writer permission for user, and u+w, g-rremove group read permission.

12

7

100010001

Processes

Ending a program

When a process ends, there is a return code (exit status) p , ( )associated with the process outcome

This is a non-negative integer 0 means success

anything larger than 0 represent various kinds of failure

The return value is passed to the parent process

8

Communication• Processes can communicate using a

number of means:− passing arguments, environment− read/write regular filesread/write regular files− exit values− inter-process communication with

shared queues, memory and semaphores− signals− pipes− sockets: different machines

A i i di d d t h l th t ll t• A pipe is a one-way medium-speed data channel that allows two processes on the same machine to talk.

• If the processes are on different machines connected by a network, then a mechanism called a “socket” may be used instead. A socketis a two-way high-speed data channel.

Pipeline

Th i thi b t i li i th t bl• The nice thing about pipelines is that many problems can be solved by such an arrangement of processes.

• Each process in the pipeline performs a set of operationsupon the data and then passes the results on to the next process for further processing.

Process 1 Process 2 Process 3

A B Cwho | sort

who | wc -l

who | sort | wc -l

9

Pipeline Example

abacus

abacus

abacus

abacus

3 abacus

1 bottlesort uniqbottle

abacus

abacus

bottle

1 bottlesort uniq

sort file uniq terminal

U W T

wc -l

S

sort xFile123 | uniq | wc -l

Pipe-Equivalent Communication Using a File

Could we use a file instead of a pipe? YES.

Run first program, save output into file

sort xFile123 | uniq | wc -l

Run second program, using file as input ….

sort file122 > tmp; uniq tmp > tmp2; wc –l tmp2

Disadvantages:

process 1 process 2 process 3

Disadvantages:

Unnecessary use of the disk Slower Can take up a lot of space

Makes no use of multi-tasking

Pipe is very similar, but does not involve the external device

all mechanisms stay in the realm of the operating system

10

Unix Utilities

• Standard UNIX comes complete with at least 200 small utility programs, usually including:

− shells,

− editors,

− a C compiler,

− matching with regular expressions, e.g., grep

− searching, e.g., find,

− a sorting utility, e.g. sortg y g

− software development tools,

− text-processing tools, etc. awk, sed

Basic utilities Introduces the following utilities, listed in in groups:

Generalmanclearcaldatemail

Directoryls -ldtSrmkdirrmdircdpwd

Filecatmoreheadtailcp -r

File printlplprlprmlpqlpstat

20

(quota –v)p

mvrm -rchmodwc -lcchgrpchown

11

Utilities II – advanced utilitiesIntroduces utilities for power users, listed in

alphabetical order:

t datawkbiffcmpcryptdiffdump

odtartimetrulcompresscpio

grepgunzipperlsedsortsuumount

egrepgziplnmount

croncrontabfgrepfind

uncompressuniqwhoami

gREp , eREep RE: regular expression

Pattern Maning Example

c Non-special, matches itself 'tom'

\c Turn off special meaning '\$'

$ End of line 'ab$'$ End of line ab$

^ Start of line ^ab

. Any single character '..nodes'

[…] Any single character in [] '[tT]he'

[^…] Any single character not in [] '[^tT]he'

R* Zero or more occurrences of R 'e*'

R+ One or more occ rrences of R (egrep) 'e+'

Don’t get confused

R+ One or more occurrences of R (egrep) 'e+'

R? Zero or one occurrences of R (egrep) 'e?'

R1R2 R1 followed by R2 '[st][fe]'

R1|R2 R1 or R2 (egrep) 'the|The'

22Don’t get confused with UNIX

metacharacter (file name wildcards)ls file*.c *.java cp file?.sh .

12

We introduce about thirty useful utilities.

Section Utilities

Filtering files egrep fgrep grep uniq

Introduces utilities for power users, grouped into logical sets

Utilities II – advanced utilities

Filtering files egrep, fgrep, grep, uniqSorting files sort Extracting fields cutComparing files cmp, diff Archiving files tar, cpio, dump Searching for files find Scheduling commands at, cron, crontabProgrammable text processing awk, perlHard and soft links lnSwitching users sugChecking for mail biff Transforming files compress, crypt, gunzip, gzip,

sed, tr, ul, uncompressLooking at raw file contents odMounting file systems mount, umountIdentifying shells whoamiDocument preparation nroff, spell, style, troffTiming execution of commands time23

Removing Duplicate Lines: uniq

The uniq utility displays a file with all of its identical adjacentlines replaced by a single occurrence of the repeated line.

Here’s an example of the use of the uniq utility: Here s an example of the use of the uniq utility:

$ cat animals # look at the test file. cat snake monkey snake dolphin elephant dolphin elephant

t l h t

$ uniq animals # filter out duplicate adjacent lines.

cat snake monkey snake

goat elephant pig pigpig pigmonkey pig pig pig

24

ydolphin elephant goat elephant pig pigmonkey pig pig pig

13

Removing Duplicate Lines: uniq

The uniq utility displays a file with all of its identical adjacentlines replaced by a single occurrence of the repeated line.

Here’s an example of the use of the uniq utility: Here s an example of the use of the uniq utility:

$ cat animals # look at the test file. cat snake monkey snake dolphin elephant dolphin elephant

t l h t

$ uniq -c animals # filter out duplicate adjacent lines.

1 cat snake 1 monkey snake

goat elephant pig pigpig pigmonkey pig pig pig

25

y2 dolphin elephant 1 goat elephant 2 pig pig1 monkey pig 1 pig pig

How about un-adjacent lines? sort and then uniq

Sort

• sorts a file in ascending or descending order based on one or more fields.one or more fields.

• Individual fields are ordered lexicographically, which means that corresponding characters are compared based on their ASCII values.

-t: separator is : instead of blank tab-r descending instead of ascending-r descending instead of ascending

-n numeric sort-f ignore case-M month sort (3 letter month abbreviation)

14

Sort Examples

sh-3.00$ cat ftoSortJohn Smith 1222 26 Apr 1956Tony Jones 1012 20 Mar 1950Tony Jones 1012 20 Mar 1950John Duncan 1111 20 Jan 1966Larry Jones 1223 20 Dec 1946Lisa Sue 1222 15 Jul 1980

sh-3.00$ sort ftoSortJohn Duncan 1111 20 Jan 1966John Smith 1222 26 Apr 1956

sh-3.00$ sort –r ftoSortTony Jones 1012 20 Mar 1950Lisa Sue 1222 15 Jul 1980John Smith 1222 26 Apr 1956

Larry Jones 1223 20 Dec 1946Lisa Sue 1222 15 Jul 1980Tony Jones 1012 20 Mar 1950

Lisa Sue 1222 15 Jul 1980Larry Jones 1223 20 Dec 1946John Smith 1222 26 Apr 1956John Duncan 1111 20 Jan 1966

sh-3.00$ sort -k3 –n ftoSort # +2 -3 start 0Tony Jones 1012 20 Mar 1950John Duncan 1111 20 Jan 1966John Duncan 1111 20 Jan 1966John Smith 1222 26 Apr 1956Lisa Sue 1222 15 Jul 1980Larry Jones 1223 20 Dec 1946

sh-3.00$ sort -k3 -k4 -n ftoSort # +2 -3 +3 -4Tony Jones 1012 20 Mar 1950Tony Jones 1012 20 Mar 1950John Duncan 1111 20 Jan 1966Lisa Sue 1222 15 Jul 1980John Smith 1222 26 Apr 1956Larry Jones 1223 20 Dec 1946

28

15

Sort Examples

sh-3.00$ sort -k5 ftoSort # +4 -5 John Smith 1222 26 Apr 1956L J 1223 20 D 1946Larry Jones 1223 20 Dec 1946John Duncan 1111 20 Jan 1966Lisa Sue 1222 15 Jul 1980Tony Jones 1012 20 Mar 1950

sh-3.00$ sort -k5 -M ftoSort # +4 -5 John Duncan 1111 20 Jan 1966

LexicographicallyMonths not sorted correctly

John Duncan 1111 20 Jan 1966Tony Jones 1012 20 Mar 1950John Smith 1222 26 Apr 1956Lisa Sue 1222 15 Jul 1980Larry Jones 1223 20 Dec 1946

-M enablesmonths to be sorted correctly

sort + uniq

• uniq is a little limited but we can combine it with sortsort | uniq -csort | uniq c

• counts number of times line appears in file

• output would now be:3 abacus

1 bottle

30

16

sort + uniq

• To understand:

abacus

abacus

bottle

abacus

abacus

abacus

abacus

bottlesort uniq

31

abacus bottle

Comparing Files: cmp, diff There are two utilities that allow you to compare the contents of two files:

cmp, which finds the first byte that differs between two files diff, which displays all of the differences and similarities between two files

Testing for sameness: cmp Testing for sameness: cmp

The cmp utility determines whether two files are the same.

$ cat lady1 # look at the first test file. Lady of the night, I hold you close to me, And all those loving words you say are right.

$ cat lady2 # look at the second test file$ cat lady2 # look at the second test file. Lady of the night, I hold you close to me, And everything you say to me is right.

$ cmp lady1 lady2 # files differ. lady1 lady2 differ: char 48, line 3$ _32

17

File Differences: diff

The diff utility compares two files and displays a list of editing changes that would convert the first file into the second file.

$ diff lady1 lady2 # compare lady1 and lady2.

3c3

< And all those loving words you say are right.

…

> And everything you say to me is right.

$ _

33

cut deal with fields (columns)• Used to split lines of a file

• A line is split into fields• A line is split into fields

• Fields are separated by delimiters

• A common case where a delimiter is a space:− Default is tab, (not “ ”) need to set it for blank

− cut -f3 -d” ”

• hello there world

34

field

delimiter

18

sh-3.00$ cat ftoSort # assuming tab as delimiterJohn Smith 1222 26 Apr 1956Tony Jones 1012 20 Mar 1950John Duncan 1111 20 Jan 1966Larry Jones 1223 20 Dec 1946Lisa Sue 1222 15 Jul 1980

sh-3.00$ cut -f 1 ftoSortJohnTonyJohnLarryLisa

sh-3.00$ cut -f 1,3 ftoSortJohn 1222Tony 101John 1111Larry 1223Lisa 1222

indigo 399 % cut -f 1-3 ftoSortJohn Smith 1222Tony Jones 101John Duncan 1111Larry Jones 1223Lisa Sue 1222

find Utility

• find pathList expression

• finds files starting at pathList• finds files starting at pathList

• finds files descending from there

• Allows you to perform certain actions − e.g., deleting the files

− e.g., copying , mv, rm …..

find . –name “*.c” -exec mv {} {}.bak \;

find . –name “*.class” -exec rm {} \;

“Find all the c files and make a backup of them/rename to xx.bak“

“Find all the Java class files and delete them”

19

Find Utility

• -mtime count true if the file has been modified within count days

• -atime count− true if the file has been accessed within count days

• -ctime count− true if the contents of the file have been modified within

count days or any of its file attributes have been modified• -exec command

t if th it d 0 f ti th d− true if the exit code = 0 from executing the command. command must be terminated by \; If {} is specified as a command line argument it is replaced by

the file name currently matched

Find Examples• $ find / -name x.c

− searches for file x.c in the entire file system

• $ find . -mtime 14 − lists files modified in the last 14 days

• $ find . -name '*.bak' -ls -exec rm # “*.bak”− search for all back files

$ fi d ' ? ' {} {} # “ ? ”• $ find . -name 'a?.c' -exec cp {} {}.# “a?.c”− Find all a?.c and then cp it to a?.c.bak

a1.c a1.c.bak a2.c a2.c.bak a3.c a3.c.bak

20

Find Examples• $ find / -name x.c

− searches for file x.c in the entire file system

• $ find . -mtime 14 -ls− lists files modified in the last 14 days

• $ find . -name '*.bak' -exec rm {} \;− ls and then remove all files that end with .bak

• $ find . -name ‘a2.c’ -exec cp {} a2.c.bak \;

$ fi d ‘ ? ’ {} {} b k \• $ find . -name ‘a?.c’ -exec mv {} {}.bak \;− Find ax.c and then rename it to ax.c.bak

• $ find . -name ‘*.c’ -exec cp {} {}.2015Su \;− Find x.c and then cp it to x.c.2015Su

Transforming Files: tr tr copies the standard input to the standard output with

substitution or deletion of selected characters input characters from string1 are replaced with the

corresponding characters in string2 (the last character in t i 2 i d f ddi if )string2 is used for padding if necessary)

tr –cds string1 string2

the –d option results in deletion from the input string of all characters specified in string1

the –s option condenses all repeated characters in the input string to a single character

with the –c option, string1 is substituted with its complement; i.e., all characters that are not in the original string1

40

21

Transforming Files with tr: Examples

$ tr xyz abcthis is a x world y zthis is a a world b cthis is a a world b c^D

$ cat aFilea lot of space

$tr alo xyz < aFile # transfer a to x, l to y# and o to z#

x yzt zf spxce

41

Transforming Files with tr: Examples

# transfer all lower case character to upper case$ tr [a-z] [A-Z] [:lower:] [:upper:]A LOT OF SPACE

# transfer all spaces into dots$ tr " " .a...lot.....of......space

# transfer all spaces into dots and compact the# dots

$ tr " " . | tr -s "."a.lot.of.space

$ tr 0-9 “*” < input.txt #replace digits with *

42

22

We introduce about thirty useful utilities.

section Utilities

Filtering files egrep fgrep grep uniq

Introduces utilities for power users, grouped into logical sets


Filtering files egrep, fgrep, grep, uniqSorting files sort Extract fields cutComparing files cmp, diff Archiving files tar, cpio, dump Searching for files find Scheduling commands at, cron, crontabProgrammable text processing awk, perlHard and soft links lnSwitching users su gChecking for mail biff Transforming files compress, crypt, gunzip, gzip,

sed, tr, ul, uncompressLooking at raw file contents odMounting file systems mount, umountIdentifying shells whoamiDocument preparation nroff, spell, style, troffTiming execution of commands time


grep/egrep

sort

grep -w –i ^[Tt]he file123

sort –t: -k4 -r -n/M file

Regular Expression

sort

uniq

cut

awk

tr

/

uniq -c

cut -d” ” -f2,3

awk –F”:” ‘{print $1,$2}’

tr a-z A-Z < filename

Start column 1, default delimiter:

blank/tab

Start column 1Default delimiter: tab

cmp/diff

ln

find

cmp/diff

ln

find . –name “*.c” -exec

cp {} {}.bak \;

Start column 1Default delimiter blank/tab

23

Plan for last week, today and next week

• Unix

− Introduction (file system, process, security …)( y , p , y )

− Utilities

− Shell functionalities (common)

− Bourn shell and shell script

UNIX Shells

Ch 4 Unix shells “UNIX for Programmers and Users”

Third Edition, Prentice-Hall, GRAHAM GLASS, KING ABLES

24

INTRODUCTION

A shell is a program that is an interface between a user and A shell is a program that is an interface between a user and the raw operating system.

It makes basic facilities such as multitasking and piping easyto use, and it adds useful file-specific features such as wildcardsand I/O redirection.

There are four common shells in use:

· the Korn shell · the C shell · the Bourne shell· the Bash shell (Bourne Again Shell)

SHELL FUNCTIONALITY

- This part describes the common core of functionality thatall four shells provide.

- The relationship among the four shells:

Common Common

Bourne shell

Korn shellC shell

Bourne Again Shell

Commoncore

Commoncore

25

SHELL FUNCTIONALITY

- A hierarchy diagram to illustrate the features shared by the four shells

Shell functions

Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing subsitution

Local Environment Conditional Unconditional

Displaying Information : echo

The built-in echo command displays its arguments to standard output and works like this:

Shell Command: echo {arg}*

echo is a built-in shell command that displays all of itsarguments to standard output.

By default, it appends a new line to the output. t fi use –n to fix

sh-3.00$ echo hello world # echo “hello world”hello worldsh-3.00$ echo -n hellohellosh-3.00$

26

SHELL OPERATIONS

Commands range from simple utility invocations like:

$ ls$ ls

to complex-looking pipeline sequences like:

$ cat compact123 | sort | uniq | wc -l

- a command with a backslash(\) character, and the shell will allow you to continue the command on the next line:

$ echo this is a very long shell command and needs to \be extended with the line-continuation character. Note \that a single command may be extended for several lines.

$ _

METACHARACTERSSome characters are processed specially by a shell and

are known as metacharacters.

All four shells share a core set of common metacharacters, whose meanings are as follow:

Symbol Meaning

> Output redirection; writes standard output to a file.>> Output redirection; appends standard output to a file.< Input redirection; reads standard input from a file. * File substitution wildcard;* File-substitution wildcard;

matches zero or more characters.? File-substitution wildcard;

matches any single character.[…] File-substitution wildcard;

matches any character between the brackets.

CSE1020 lab tour. Don’t confuse with RE

27

Symbol Meaning

`command` Command substitution; replaced by the output from

Shell functions

Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background Command Commands Processing substitution


command Command substitution; replaced by the output fromcommand.

$ Variable substitution. Expands the value of a variable.

& Runs a command in the background. | Pipe symbol; sends the output of one process to the

input of another. ; Used to sequence commands. % echo hello; wc lyrics

|| Conditional execution;executes a command if the previous one fails.

&& Conditional execution; executes a command if the previous one succeeds.

(…) Groups commands.

# All characters that follow up to a new line are ignoredby the shell and program(i.e., used for a comment)

\ Prevents special interpretation of the next character.<<tok Input redirection; reads standard input from script up to tok.

- When you enter a command,the shell scans it for metacharacters and processes them specially.

When all metacharacters have been processed,the command is finally executed.

To turn off the special meaning of a metacharacter, precede it by a backslash(\) character. # Also “ ” ‘ ‘ (later)

Here’s an example:

$ echo hi > file # store output of echo in “file”. $ cat file # look at the contents of “file”. hi hi $ echo hi \> file # inhibit > metacharacter. hi > file # > is treated like other characters. $ cat file # look at the file again. Not writtenhi

$ echo 3 \* 4 = 12$ echo 3 + 2 = 5 # this is a comment

28

- When you enter a command,the shell scans it for metacharacters and processes them specially.

When all metacharacters have been processed,the command is finally executed.

To turn off the special meaning of a metacharacter, precede it by a backslash(\) character. # Also “ ” ‘ ‘ (later)


$ echo hi > file # store output of echo in “file”. $ cat file # look at the contents of “file”. hi hi $ echo hi \> file # inhibit > metacharacter. hi > file # > is treated like other characters. $ cat file # look at the file again. Not writtenhi

$ echo 3 \* 4 = 12$ echo 3 + 2 = 5 # this is a comment

Redirection > >> < <<

The shell redirection facility allows you to:

Shell functions


Local Envirnment Conditional Unconditional

1) store the output of a process to a file (output redirection) 2) use the contents of a file as input to a process (input redirection)

Output redirection

To redirect output, use either the “>” or “>>” metacharacters.

The sequence

$ a.out > fileName

$ echo “new line” > fileName$ echo “new line” >> fileName

sends the standard output of command to the file with name fileName.

Difference?

29

• Input Redirection

Input redirection is useful because it allows you to prepare a processinput beforehand and store it in a file for later use.

To redirect input, use either the ‘<‘ or ‘<<‘ metacharacters.

The sequence

$ a.out < fileName

executes command using the contents of the file fileNameas its standard input.

If the file doesn’t exist or doesn’t have read permission, an error occurs.

• FILENAME SUBSTITUTION ( WILDCARDS )

All shells support a wildcard facility that allows you to select files

Shell functions



- All shells support a wildcard facility that allows you to select files that satisfy a particular name pattern from the file system.

- The wildcards and their meanings are as follows: ls *.c cp a?.c

Wildcard Meaning

* Matches any string, including the empty string.

? Matches any single character.

[..] Matches any one of the characters between the brackets.A range of characters may be specified by separating

a pair of characters by a hyphen. [a-d] [0-9]

Don’t confuse with Regulation Expression grep a*b file123.* grep a?.c file123?

30

- Prevent the shell from processing the wildcards in a string by surrounding the string with single quotes(apostrophes) or doublequotes.

-A backslash(/) character in a filename must be matched explicitly A backslash(/) character in a filename must be matched explicitly. - Here are some examples of wildcards in action:

$ ls *.c # list any text ending in “.c”. a.c b.c cc.c

$ ls ?.c # list text for which one character is followed by “.c”.a.c b.c

$ cp /cs/dept/course/2014-15/S/2031/xFile? .

$ cp /cs/dept/course/2014-15/S/2031/xFile* .

$ cp /cs/dept/course/2014-15/S/2031/xFile[12] .

$ ls [ac]* # list any string beginning with “a” or “c”. a.c abcd a3.pdf cc.c ce3.doc

$ ls [A-Za-z]* # list any string beginning with a letter. a.c b.c cc.c

$ ls dir*/*.c # list all files ending in “.c” files in “dir*”# directories ( that is, in any directories beginning

with “dir” ). dir1/d.c dir2/g.c

$ ls */*.c # list all files ending in “.c” in any subdirectory. dir1/d c dir2/g cdir1/d.c dir2/g.c

$ ls *2/?.? ?.? ---> list all files with extensions in “2*” directoriesand current directory.

a.c b.c dir2/f.d dir2/g.c$ _

31

• PIPES

Shell functions



- Shells allow you to use the standard output of one processas the standard input of another process by connecting the processes

together using the pipe(|) metacharacter.

- The sequence

$ command1 | command2

th t d d t t f d1 t “fl th h” t causes the standard output of command1 to “flow through” to the standard input of command2.

- Any number of commands may be connected by pipes.

A sequence of commands changed together in this wayis called a pipeline.

$ head -4 /etc/passwdroot:eJ2S10rVe8mCg:0:1:Operator:/:/bin/cshnobody:*:65534:65534::/:daemon:*:1:1::/:sys:*:2:2::/:/bin/csh

Or, awk -F: ‘{ print $1 }’ sys:*:2:2::/:/bin/csh

$ cat /etc/passwd | cut –d “:” -f 1 | sort audit bin daemon glass ls Pipe cut Pipe sort Terminalingresnews news nobody rootsync sys tim$ cat /etc/passwd | grep –w Wang | wc -l ???

Lets do it

32

Pipeline Example

abacus

abacus

abacus

abacus3 abacus

bottle

abacus

abacus

bottle

1 bottlesort uniq

sort file uniq terminal

U W T

wc -l

S

sort xFile123 | uniq | wc –l

COMMAND SUBSTITUTION used very very … heavily in script!

A command surrounded by grave accents (‘) - back quote - is executed,

Shell functions

Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background CommandCommands Processing substitution


A command surrounded by grave accents ( ) back quote is executed,and its standard output is inserted in the command’s place in the entirecommand line. Any new lines in the output are replaced by spaces.

For example:

$ echo the date today is ‘date`, right?the date today is Sun Jul 26 08:57:44 EDT 2015, right? $ _

$ who ---> look at the output of who. posey ttyp0 Jan 22 15:31 ( blackfoot:0.0 ) glass ttyp3 Feb 3 00:41 ( bridge05.utdalla ) huynh ttyp5 Jan 10 10:39 ( atlas.utdallas.e )

$ echo there are ‘who | wc -l` users on the systemthere are 3 users on the system

$echo there are ‘wc –l classlist` students in the classthere are 71,students in the class

$echo there are ‘cat classlist | grep –w Wang` students with name Wanthere are 1,students with name Wang

$x=`wc –l classlist` # x get value 54 (talk later)

33

COMMAND SUBSTITUTION used very very … heavily in script!

A command surrounded by grave accents (‘) - back quote - is executed,

Shell functions

Built-in Scripts Variables Redirection Wildcards Pipes Sequence Subshells Background CommandCommands Processing substitution


A command surrounded by grave accents ( ) back quote is executed,and its standard output is inserted in the command’s place in the entirecommand line. Any new lines in the output are replaced by spaces.

For example:

$ echo the date today is ‘date`, right?the date today is Sun Jul 26 08:57:44 EDT 2015, right? $ _

$ who ---> look at the output of who. posey ttyp0 Jan 22 15:31 ( blackfoot:0.0 ) glass ttyp3 Feb 3 00:41 ( bridge05.utdalla ) huynh ttyp5 Jan 10 10:39 ( atlas.utdallas.e )

$ echo there are ‘who | wc -l` users on the systemthere are 3 users on the system

$echo there are ‘cat calsslist | wc –l` students in the classthere are ,44 students in the class

$echo has ‘cat classlist | grep –w Wang | wc -l`students with name Wanghas 1,students with name Wang

$x=`wc –l 2031classlist` # x get value 44 (talk later)

• SEQUENCES ;

If you enter a series of simple commands or pipelines separated bysemicolons the shell will execute them in sequence from left to right

Shell functions


Local EnvirnmentConditional Unconditional

semicolons, the shell will execute them in sequence, from left to right.

This facility is useful for type-ahead(and think-ahead) addicts who liketo specify an entire sequence of actions at once.


$ date; pwd; ls # execute three commands in sequence. Mon Feb 2 00:11:10 CST 1998 /home/glass/wild a.c b.c cc.c dir1 dir2 $ _

$ date > date.txt; ls; pwd > pwd.txt a.c b.c cc.c date.txt dir1 dir2

- Each command in a sequence may be individually I/O redirected as well:

34

• Conditional Sequences && ||

Every UNIX process terminates with an exit value

Shell functions



- Every UNIX process terminates with an exit value. By convention, an exit value of 0 means that the process completed successfully, and a nonzero exit value indicates failure. (opposite to C)

- All built-in shell commands return a value of >=1 if they fail. You may construct sequences that make use of this exit value:

1) For a series of commands separated by “&&” tokens, the next command is executed only if the previous command returns

an exit code of 0. i.e., successful

2) For a series of commands separated by “||” tokens, the next command is executed only if the previous command returns

a non-zero exit code. fails

- For example, if gcc compiles a program without fatal errors,it creates an executable program called “a.out” and returns an exitcode of 0; otherwise, it returns a non-zero exit code.

$ gcc myprog.c && a.out # if gcc successful, do a.out

$ gcc myprog.c || echo “compilation failed.”

$ grep huiwang classlist && echo “found in the list”

return 0 if match, return 1 otherwise

35

• Background Processing &

Shell functions



g g

- If you follow a simple command, pipeline, sequence of pipelines, or group of commands by the “&” metacharacter, a subshell iscreated to execute the commands as a background process

- The background process runs concurrently with the parent shell and does not take control of the keyboard.

- Background processing is therefore very useful for performing severaltasks simultaneously, as long as the background tasks do not requireinput from the keyboard.

• SHELL PROGRAMS: SCRIPTS

- Any series of shell commands may be stored inside a regular text file

Shell functions



Any series of shell commands may be stored inside a regular text filefor later execution.

A file that contains shell commands is called a script. batch file in Windows

Before you can run a script, you must give it execute permission by using the chmod utility.

chmod u+x filename echo hello world

To run it, you need only to type its name.

- Scripts are useful for storing commonly used sequences of commands,and they range in complexity from simple one-liners to fully blown

programs.

date

36

• SHELL PROGRAMS: SCRIPTS

one for the Bash shell and the other for the Korn shell.

$ cat > script.sh # create the bash script. #! /bin/sh# This is a sample sh script # This is a sample sh script. echo “hello world”echo -n “the date today is “ # in sh, -n omits new linedate # output today’s date.^D # end of input.

$ chmod u+x script.sh # make the scripts executable.

$ script.sh # execute the shell script. hello worldThe date today is Sun Feb 1 19:50:00 CST 2004

• VARIABLES substitution $

A h ll ki d f i bl

Shell functions



- A shell supports two kinds of variables: local and environment variables.

local: user defined, positional

Both kinds of variables hold data in a string format. Both kinds of variables hold data in a string format.

the child shell gets a copy of its parent shell’s environment variables, but not its local variables.

Environment variables are therefore used for transmitting useful information between parent shells and their children.

E h ll h f d fi d i i bl h

37

• Environment VARIABLES (for your reference)

- Here is a list of the predefined environment variables that are common to all shells:

Name Meaning

$HOME the full pathname of your home directory

$PATH a list of directories to search for commands

$MAIL the full pathname of your mailbox

$USER your username

$SHELL the full pathname of your login shell

$TERM the type of your terminal

-several common built-in local variables that have special meanings:

Name Meaning

Built-in local variables (must know!)

$$ The process ID of the shell.

$0 The name of the shell script ( if applicable ).

$1..$9 $n refers to the nth command line argument( if applicable ).

$* A list of all the command-line arguments.

$ myscript we are the world$0 $1 $2 $3 $4

$*

38

#! /bin/sh

echo the name of this script is $0echo the first argument is $1echo a list of all the arguments is $*echo the script places the date into a temporary file called $1.$$ date > $1.$$ # redirect the output of date.

$ script.sh paul ringo george john # execute the script. the name of this script is script.shthe first argument is paul

$ $$ pls $1.$$ # list the file. rm $1.$$ # remove the file.x=5echo the value of variable x is $x

$ variable substitution

the first argument is paula list of all the arguments is paul ringo george john the script places the date into a temporary file called paul.2431paul.2431value of variable x is 5.$ _

• QUOTING

There are often times when you want to inhibit the shell’s wildcard-substitution * ? [], variable-substitution $, and/or command-substitution ` ` mechanisms.

The shell’s quoting system allows you to do just that.

- Here’s the way that it works:

1) Single quotes(‘ ’) inhibits wildcard substitution,variable substitution, and command substitution.

2) Double quotes(“ ”) inhibits wildcard substitution only.

3) When quotes are nested, it’s only the outer quotes that haveany effect

39

• QUOTING

- The following example illustrates the difference between the two different kinds of quotes:

$ echo 3 * 4 = 12 # remember, * is a wildcard. 3 a.c b b.c c.c 4 = 12

$ echo “3 * 4 = 12” # double quotes inhibit wildcards. 3 * 4 = 12

$ echo ‘3 * 4 = 12’ # single quotes inhibit wildcards. 3 * 4 = 12

another way?$ echo 3 \* 4 = 12 # backslash inhibit a metacharacter3 * 4 = 12

$ name=Graham

- By using single quotes(apostrophes) around the text,we inhibit all wildcarding and variable and commandsubstitutions:

$ name=Graham # assign value to name variable$ name=Graham # assign value to name variable

$ echo ‘3 * 4 = 12, my name is $name - date is `date`’3 * 4 = 12, my name is $name - date is ‘date’$ _

- By using double quotes around the text, we inhibit wildcarding,but allow variable and command substitutions:

inhibitedinhibitedinhibited

$ echo “3 * 4 = 12, my name is $name - date is `date`”3 * 4 = 12, my name is Graham - date is Mon Feb 2 23:14:56 CST $ -

inhibited inhibitedinterpreted

40

- TERMINATION AND EXIT CODES

Every UNIX process terminates with an exit value.

By convention an exit value of 0 means that the processBy convention, an exit value of 0 means that the processcompleted successful, and a nonzero exit value indicates failure.

All built-in commands return an exit value of 0 if they succeedreturn an non-zero is they fail.

In the Bash, Bourne and Korn shells, the special shell variable$? always contains the value of the previous command’s$? always contains the value of the previous command sexit code.

In the C shell, the $status variable holds the exit code.

date, ls, grep, find ….. Every command has a exit code (return status)Look for man

Matching 0 No matching 1 No such file 2

-In the following example, the date utility succeeded, whereasthe gcc utilities failed:

$ date # date succeeds. Mon Feb 2 22:13:38 CST 1998 $ echo $? # display its exit value $ echo $? # display its exit value. 0 # indicates success.

$ gcc prog.c # compile a nonexistent program. cpp: Unable to open source file ‘prog.c’.

$ echo $? # display its exit value. 1 # indicates failure.

41

-In the following example, the grep utility succeeded or failed:

$ grep Wang 2031classlist$ echo $? # display its exit value. 0 # indicates success

Matching 0 No matching 1 No such file 2

0 # indicates success.

$ grep Hui 2031classlist$ echo $?1 # indicates failure (not matching).

$grep Wang 2031classlistXgrep: classlistX: No such file or directory$ echo $?2 # indicates failure (not such a file).

Used in scriptingLook for manman grep | grep –w “exit”

Try it

- Any script that you write should always explicitly return an exitcode.

To terminate a script, use the built-in exit command, which works as follows: which works as follows:

Shell Command: exit number

exit terminates the shell and returns the exit value numberto its parent process.

If number is omitted, the exit value of the previous commandis used.

42

SummaryShell functions


< >>>

* ? [ ] | ;&

` `$

• Covered core shell functionality− Built-in commands − Redirection < > >>− Wildcards * ? [ ] filename substitution


&& ||$0 $1-9 $*

Wildcards ? [ ] filename substitution− Pipes |− Background processing &− Command substitution ` `− Variables $ variable substitution− Quotes “ ” ‘ ’

Some announcements - eecs.yorku.ca file−Shell functionalities (common) −Bourn shell and...

Documents

Transcript of Some announcements - eecs.yorku.ca file−Shell functionalities (common) −Bourn shell and...