Unit -3 Shell Programming

© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63 By Narinder Kaur, Asstt. Prof

Unit -3

Shell Programming

© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63. By Narinder Kaur, Asstt. Prof

What is the Shell

• Shell is a user level program.

• It is an command language interpreter that executes commands read from the standard input device (keyboard) or from a file.

• It is not part of system kernel, but uses the system kernel to execute programs, create files etc.

• In MS-DOS, Shell name is COMMAND.COM which is also used for same purpose, but it's not as powerful as Linux Shells are!


• Sh (1971) – Developed by Ken Thompson - Introduced the concept of pipes, filters and redirection Lacked the ability to script.

• Bourne (1977) – Developed by Stephen Bourne- Introduced the concepts of scripting, control flows, string literals and command substitution - Lacked the ability to define functions

• Korn (1978) – Developed by David Korn (also called ksh)- Includes features from other shells- Includes advanced features from Ruby and Python languages like floating point arithmetic and arrays- File name completion- History feature

Common Shells


• C (1978) – Developed by Bill Joy (also called csh)- Provides C language feel- Advanced feature is Command history

• Tenex C (1983) – Developed by Ken Greer (also called tcsh)- Command line editing feature- Command completion

• Bourne Again Shell – Developed by Brijan Fox (also called bash)- It combines the features of Korn and C shells- Introduced some environment variables- Default shell in Linux.- specific shell variables.- some built-in commands like set and man.

Common Shells


The file /etc/shells lists the shells available under Linux. It contains a list of all the installed shells, available to all users The first entry acts as the default shell

Note : that each shell does the same job, but each understand a different command syntax and provides different built-in functions.

/etc/shells


.login: Login Initialization File

.tcshrc: TCSH shell configuration file

.logout: Logout file

Tcsh

.bash_profile: Login initialization file

.bashrc: BASH shell configuration file

.bash_logout: Logout script

Bash

ScriptsShell

Csh .cshrc .login (executed only at login)

.logout (executed only at logout)

Login/Logout Scripts


1. Command Interpretation

2. Providing a scripting language

3. A process that provides runtime environment

Shell: Responsibilities


Simple architecture of a Shell

User

Kernel

Lexical Analysis Expansion Execution


The Shell1. Issues the prompt and waits for a command to be

entered2. Analyses the input and parses it3. Scans the command line for meta-characters and

performs expansions to give a simple command4. Passes command line to the kernel for execution5. Waits for command to execute6. On completion gives the prompt again; and the cycle

continues

Interpretation cycle for shell commands


• Control the environment provided by the shell

• Set via any of the following activities After user logs in through login scripts As a result of scripts executed after login Interactively by user

• Also known as Environment / System variables• .bash_profile initialization file contains definitions and

assignments of parameter variables.

Shell: Variables


• Each shell variable is used for a specific purpose

• A shell variable is typically entered in uppercase

• To get its value, you precede the shell variable name with a $, as in $PWD

• The set command shows the complete list

Shell: Variables


Examples of shell variables:

PWD – the most recent working directory set with the cd command

OLDPWD – the previous working directory set with the cd command

BASH – the full path of the bash shell

HOME – each user’s home directory, typically /home/username, where username is a user’s login name

HOSTNAME – the current hostname of a Linux system

Learning about shell variables


• Expansion is the process of using meta characters and special symbols to change the given text

• Simple examples: Variable Expansion

$HOME expands into current user’s home directory,$BASH expands to give the full path of the bash shell

• The types of expansion discussed: Parameter and variable Command substitution

Expansion


• Parameter expansion substitutes values for parameter or variable names

• Parameters and variables are names that contain data values• Example:

echo $x will display the contents of a variable named x Notice=“Meeting will be held tomorrow” Echo $Notice

Use of braces {…} allows one to mix variables and numbers. Example:

echo ${EmployeeName} will display an employee’s name on the screen

Parameter and Variable Expansion


Command substitution allows you to substitute the output of a command in place of the command itself

Two forms: Use $(command)

Example: echo "Today is " $(date) Surround the command with a single back quote as in

`command`Example : Mycmd = `ls *.c`

echo $Mycmd

Expansion – Command Substitution


• Normally shells are interactive i.e. shell accept command from you (via keyboard) and execute them.

• But if required a set of related commands can be run by storing them to text file and telling the shell to execute this text file.

• This is known as shell script.

• Definition: "Shell Script is series of command written in plain text

file. Shell script is just like batch file is MS-DOS but have more power than the MS-DOS batch file."

Shell Script


• Shell script can take input from user, file and output them on screen.• Useful to create our own commands.• Multiple commands can be bound by means of control constructs to

define a logical flow of control.

• Shell scripts execute commands sequentially.

• You can alter the sequential order with decisions, loops and functions

• Shell scripts

• combine programming logic with operating system commands

• help in automating job duties

Why to write shell script?


• Use any editor like vi or gedit to write shell script.• After writing shell script set execute permission for your

script as follows• Syntax:

chmod permission script-name Examples:

$ chmod +x script-name$ chmod 755 script-name

• Note: This will set read write execute(7) permission for owner, for group and other permission is read and execute only(5).

Creating shell scripts


syntax:

$ sh script-name or$ ./script-name

Examples:

$ sh scr or$ ./scr

Executing shell scripts


Syntax: variable name=value

Rules for naming variables: Variable name must begin with Alphanumeric character or

underscore character (_) Don't put spaces on either side of the equal sign when

assigning value to variable Variables are case-sensitive, just like filename in Linux NULL variable (variable which has no value at the time of

definition) can be defined as:$ vech=

Do not use ?,* etc, to name your variable names Variable value can be accessed using $vname

User defined variables


Syntax: expr op1 math-operator op2

Examples: $ expr 1 + 3$ expr 2 - 1$ expr 10 / 2$ expr 20 % 3$ expr 10 \* 3$ echo `expr 6 + 3`

Shell Arithmetic


In Linux when a particular command/shell script is executed, it returns two type of values, signifying success or failure

This value, returned via return statement, is known as Exit Status.

These values can be used to check whether command or shell script executed successfully or not. If return value is zero (0), command is successful. If return value is nonzero, command is not successful or some

sort of error executing command/shell script.

$? can be used to find the exit status of last executed command.

Exit Status


For e.g. (This example assumes that unknownfile does not exist )

$ rm unknownfile

It will show error as followsrm: cannot remove `unkownfile': No such file or directoryand after that if you give command

$ echo $? it will print nonzero value to indicate error

The value of $? can be used in scripts for decision making

Exit status : example


A blank is a space or tab used to separate items

A word (token) is a sequence of characters considered to be a single unit to the shell

A name (identifier) is a word consisting of letters, numbers and underscore

A meta character is a character that, when unquoted, separates words

A Control Operator is a token that performs control function.

Shell Grammar - I


Metacharacter: Examples

| - the pipe symbol allows you to pass output to another command

& - the ampersand allows you to run a process in the background

; - the semicolon allows you to sequence commands < - less than redirects input > - greater than redirects output

Shell Grammar - II


Control operators: || - executes a command depending upon failure of

another && - executes a command depending upon success of

another

A reserved word has special meaning to the shell and cannot be used for another purpose. Example: if, then, else, fi

Shell Grammar - III


The && Operator Causes a command to execute only if the preceding command

completes successfully (exit status of 0) Example: rm file4.txt && ls

The ls command will only execute if rm file4.txt completes successfully

The || Operator Causes a command to execute only if the preceding command

completes unsuccessfully (exit status of 1) Example: rm file4.txt || pwd

The ls command will only execute if rm file4.txt completes unsuccessfully

Command Separators


Logical -a: and -o: or !: not

Comparison -eq, -ne, -lt, -gt, -le, -ge: Numerical Comparison =, != : String comparison -z: check string against null

Operators

© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63 By Narinder Kaur, Asstt. Prof

Decision Structures


The shell requires you to follow strict syntax when implementing the if statement

The syntax of the if statement follows:if [condition]then

Statementselse

Statementsfi

if statements can be nested if required

if statement


The optional elif clause allows you to further test an if statement

if [condition]then

Statementselif [condition]then Statementselse

Statementsfi

elif clause in an if statement


Another decision structure

Use when a decision is based upon multiple inputs

Syntax:case word in

Pattern1) statements;;Pattern2) statements;;

esac

Can use * to match all patterns, ? to match a single character or […] for a range.

The case statement


Looping Structures


A set of statements is executed till a condition evaluates to true

while [ condition ]do

statementsdone

The while statement


Use the break command to end a loop prematurely.

Example:count=1while [condition]do

statementsif [ $count –gt 3 ]then

break fistatements

done

The while statement


A set of statements is executed till a condition does not evaluate to true

until [ condition ]do

statementsdone

The until statement


The for statement syntax in the shell takes two forms: Word list form:

for variable in listdo

statementsdone

Arithmetic expression form:for ((var=init; test; incr))do

Statementsdone

The for statement


• Functions are self-contained blocks of code

• Functions can accept values and return a result

• Function code can be reused

• Functions reduce redundancy

• Functions provide for modular code

Understanding functions


• General format of a function in the shell:function function-name (){

list}

• Function name is required

• Either the keyword “function” or the parentheses are required but can use both

• The left and right braces are required

• The commands in list are any valid shell commands

Components of a function


Calling a function: Done by using its name

Example:function DisplayHello (){

echo "hello"}DisplayHello

Function call


• Functions are usually placed at the beginning of the script or prior to the function call

• A function called prior to its definition will generate a “command not found” error

• Functions are executed in context of the current shell (unlike shell scripts)

• Functions can be exported to subshells using export -f

Function basics


• On execution the parameters become the positional parameters

• $0 remains unchanged

• Variables are shared between function and its caller

• return returns control back to the caller & the positional parameters are restored to their original value

• Variables local to the function can be declared using local keyword

Function parameters


function testFunc (){ for i in $* do echo $i done}

for i in $*do echo $idonetestFunc 1 2 3

Function : Example


• Use the return command to cause a function return an exit status

• General format: return n

• Where n is the return status you specify

• This value can be checked using $?

• Typically 0 for success and non-zero for indicating cause of failure

The return statement


• Array is a variable containing a series of elements

• Refer to elements using an integer called a subscript

• Subscripts begin with 0

• Use an array when you have values that are similar in nature

• All the data in the array can be accessed by referencing the whole array variable

• A single element can be accesses by using its subscript value

Arrays


• Use the declare -a command to declare an array

• Can use typeset -a command too

• Example to declare an array named courses: declare -a courses declare -ia codes

Declaring an array


• After declaring an array, you need to assign values to it

• Example to set the first to elements of the courses array: courses[0]="MCA" courses[1]="B Tech."

• Another method is to assign multiple elements as in: courses=("MCA" "B Tech. ") codes=(10 20 30)

• Space acts as the element separator

Assigning values to array elements


• Syntax to refer to a single element of an array: ${array-name[subscript]}

• Example to display the sixth element (subscript 5) in the courses array: echo ${courses[5]}

• Using array elements as rvalue: course=${courses[5]}

• The expression $courses[5] is treated as $courses followed by [5]

Using array elements


• Reference all of the array elements: echo ${array-name[*]} or echo ${array-name[@]} Example: echo ${courses[*]} or echo ${courses[@]}

• Determine the number of array elements: echo ${#array-name[*]} or echo ${#array-name[@]} Example: echo ${#food[*]} or echo ${#food[@]}

• Determine an element’s length: echo ${#array-name[subscript]} Example: echo ${#food[2]} will give the length of 3rd element in the

array.

Similarly echo ${#array} will give the length of the first lement in the array i.e. ${#array[0]}

Operations on an array


• It is possible to use arithmetic operations and variable name substitution to determine a subscript number

• Reading i-th element from user: read -p "Enter $i the element" arr[$i]

• Displaying i-th element to user: echo ${arr[$i]} " "

Getting values from the user


• Search and Replace array elements

• Eg : echo ${unix[@] / mandriva / RedHat}

will replace ‘mandriva’ with ‘RadHat’. But this will not permanently replace the array relements.



• Add an element to an existing bash array :

array=(“${array[@]}” “new item”)

• Copying an array

newarray = (“${oldarray[@]}”)



• Concatenate two arrays :

array3 = (“${array1[@]}” “${array2[@]}”)

• Deleting an entire array :

unset arraynameecho ${#array[@]} # would give 0.



which <command>

Can be used to locate executables that are in one of the directories contained in the PATH variable

It prints the full path of the executable that would get executed in the command was given on the command prompt.

which -a command prints all the occurrences of the command; not just the first one

which command


• Used for matching patterns• Pattern: built using regular expressions• Options:

-i: Ignore case -v: Don't display matching lines -n: Display line numbers along with lines -c: Display count of occurrences -l: Display list of filenames that contain the pattern that is searched in a set of

files• Variations

egrep : Equivalent to grep -E: use Extended Regular Expressions fgrep: Equivalent to grep -f: read patterns from file

grep


A*: Look for 0 or more occurrences of A . : Look for any one character (?) .* : Look for any number of characters Â : Look for A in beginning of string A$ : Look for A at the end of string Â$ : Look for A [abc]: Look for a single character a, b or c [a-d]: Look for a range [â-d]: Look for a character not in this range range

Basic regular expressions


• A+ : Matches one or more occurrences of A• A? : Matches zero or one occurrence of A• exp1 | exp2 : Matches exp1 or exp2• ( e1 | e2 ) e3 : Matches e1e3 or e2e3

• Use egrep or grep -E to make use of Extended Regular Expressions

Extended regular expressions


• A multipurpose tool • Combines work of several filters• Performs non-interactive operations on a data stream (file or input from a

pipeline)• Uses instructions to work on text• An instruction combines:

An address for selecting lines of input Action to be taken on them

• General form:• sed options 'address action' file(s)

• The 'address action' pair is enclosed in single quotes

Stream Editor - sed


Addresses in sed can be specified in two ways: By specifying line numbers: single or range By specifying a pattern enclosed in / /

Actions (commands) in sed are defined by sed's internal family of command. Can be used for

Substitution (s) Global substitution (g) Duplication (p)Deletion (d)Stop searching (q)Append (a)Replace lines (c)

Addressing and actions


Command: sUsage:

sed 's/exp1/exp2/' filename First occurrence of exp1 will be changed to exp2 in all the lines of textsed 's/exp1/exp2/g' filename All occurrences of exp1 will be changed to exp2 in all the lines of textsed '---s/exp1/exp2/' filename Change will be made in lines addressed by ---sed 's/^/exp2/' filename Will prefix exp2 in beginning of all linessed 's/$/exp2/' filename Will prefix exp2 at the end of all lines

Text Editing : Substituting text


• Some examples :

• $ sed ‘s/mr./Mr./g’ < file1 > file2will replace all occurrences of mr. with Mr. and place it in file2

• $ sed ‘1,3 s/mr./Mr./’ file will make changes only in lines from 1 to 3

• $ sed ‘5,$ s/yahoo/google/’ filewill replace yahoo with google in lines from 5 till end

• Echo “hello how are you” | sed ‘s/h/H/g’will give ouptput as : Hello How are you

Text Editing : Substituting text


‘q’ option is used to stop searching after a particular line.

sed '3q' <filename': Prints and quits after 3rd line sed -n '3p' <filename : Prints the third lineAddressing rules:

Num : Single line Num1, Num2: Range of lines $: Last line ! for negation: Used before command

Use -e for combining multiple commandssed -n -e '1,2p' -e '5,6p' <filename>

Will print lines from 1 to 2 and 5 to 6

Line Addressing : Example


• For appending linessed ‘/pattern/ a\line to append’

filename• e.g.

• sed ‘3 a\From here the 4th line begins’ fileThe above command will append a line after 3rd line.

• For duplicating linessed ‘/^$/p’ file

will duplicate all empty lines.sed ‘1,5 p’ file

will duplicate lines from 1 to 5

Appending and duplicating lines


‘c’ option is used to replace lines in a filesed ‘/pattern/ c\new line’ file

Examples: sed ‘1 c\This is a new line’ filename

will replace the first line with new line

Text replacement


Command: dCan be used with line / context addressing

sed '/pattern/d' filenamesed '1,3d' filename

Will remove lines from 1 to 3

Sed ‘1, /^ $/d’ fileWill remove header of a file until the first blank line.

Text Editing : Deleting lines


• An advanced filter• Available in Linux as gawk (GNU awk)• Named after its authors:

Aho, Weiberger and Kernigan

• Important Features Operates at field level, i.e. Can easily access, transform and format individual

fields Can work on extended regular expressions (ERE) Has C-like programming contructs Supports user defined variables Allows usage of arrays and functions

The awk


• Like sed, awk can be used to filter out lines (records) on basis of Line number; or Pattern

• Syntax• awk options 'selection_criteria {action} files(s)

• Selecting on basis of pattern (fixed or regular expressions)• awk '/pattern/ {print}' filename

• Selecting on basis of line number• awk 'NR==n1 {print}' filename

• awk 'NR==n1 {print}' filename // prints range of lines

Simple filtering using awk


• awk identifies fields using special variables $1..$n• $0 signifies the entire record• These variables are different from the varibles as used by shell to

idntify command line paramerters

• awk 'NR==2 { print $3, $4 }' file

• Specify field separator using -F • awk -F “:” 'NR==2 { print $3, $4 }' file

Processing individual fields


awk uses C-like format specifiers; as used in the printf function to generate formatted output.

Syntax:awk '{ printf “%5d %10s”, $2, $5 }' filename

Output The 2nd field as a 5 column wide number The 5th field as a 10 column wide string

The output generated by individual print commands can be redirected to different files / commands enclosed in double quotes

awk '{ printf “%5d %10s”, $2, $5 > “f1” }' filenameawk '{ printf “%5d %10s”, $2, $5 | “wc” }' filename

Generating formatted output


Computation can be generated on numeric fields using the basic arithmetic operators: + , - , * , / , %

These operators can be used for both Providing selection criteria Generating Output

The operations for incrementing / decrementing (pre as well as post) are also available.

Arithmetic Operators


User defined variables in awk Are created automatically without declaration Are initialized to 0 to NULL depending upon context.

Can be used in statements for performing arithmetic, comparison or output operations

As awk commands become more and more complex; the -f option can be used to fetch an awk program from a file:

awk -f <awk command file name> <input file name>

User defined variables


The built-in variable available with awk are listed as follows:

NR: Cumulative number of lines readFS: Input field separatorOFS: Output field separatorNF: Number of fields in current lineFILENAME: Current input fileARGC: Number of arguments in command lineARGV: List of arguments

Built-in Variables


Two special sections available with awk can be used for initialization and cleanup respectively.

One can use BEGIN for giving initial values to variables Printing headers; etc

Important because the commands given in {} are executed once for each line of input

Syntax:awk 'BEGIN{--} {--} END {--}' filename

BEGIN and END sections


Awk provides various functions for performing string and arithmetic operations

The parameters are passed using C-like syntaxSpecial case: parenthesis can be omitted when no parameters are

being passedCommon functions:

int (x) sqrt (x) Length / length (x) substr (str, start, len) index (s1, s2): look for s2 in s1 split (str, arr, ch): split str and store into arr, use ch as delimiter system(“cmd”)

Using functions


tr flags set1 set2 mail diff, cmp, comm cut uniq split sort head, tail join

Few Additional Commands


• The tr command allows you to translate or delete characters• Reads from Standard Input and write to Standard Output

– “-t” : translates characters of its input that is in first set to the corresponding character in second set

– “-d” : deletes those characters of its input that appear in the first set.– “-s” : replaces multiple consecutive occurrences of characters that are in set

one with a single occurrence• Symbols that can be used are:

– Escape sequences, ranges, character sequences, character classes– Character Classes: alnum, alpha, digit, lower, upper...– Example: tr [:lower:] [:upper:] to translate lower case letters to

uppercase

Using tr


• Can be used to send / receive mails.

• Sending mail : mail <username>

• Reading Mail : mail Prints one line header of each message

• Interactive Commands# to read a particular mail+, - to move back & forth

d to delete; u undeleter to replyq to quit

Using mail


Gives a formatted output after comparing Two files

Gives commands that can be applied to convert one file to other

A file and a directoryCompares the given file with a file in the given

directory having the same name. Two directories

Compares corresponding files in both directories“-r” for recursive comparisonDiff <file1> <file2>

Using diff


• Compares files • Silent if files are same• If they are different: reports the byte and line number at which first

difference occurs.• Options

“-l” Print byte number & differing byte values for each difference “-s” Silent mode, gives no output just returns status “-i n:m” skip first n bytes of file 1 and m bytes of file 2 -”n” Compare at most n bytes Can specify offsets for both files if required.

Cmp <options> <file1> <file2>

Using cmp


Compare sorted files

Three sets of outputs Lines unique to first file Lines unique to second file Lines in both files “-n” to suppress nth column of output.

comm <file1> <file2>

Using comm


Extract sections of a file based upon a delimiter

Default delimiter: TAB

Options “-f” specify field list “-c” specify column list “-d” specify delimiter “-s” suppress lines that don't contain the delimiter

Examplecut -f1,4 -d”:” -s datafile

Using cut


Extract unique / duplicate lines from a sorted fileOptions

“-i” : ignore case “-f” : ignore n fields while comparing “-s” : ignore n characters while comparing “-c” : print repeat count “-d” : print only duplicate entries “-u” : print only unique entries

Exampleuniq -f2 -cd datafile

Using uniq


Split a file and generate sub-files

Options “-a” : specify suffix length (default 2) “-b” : specify bytes per output file (b:512, k:kilo, m:mega) “-l” : specify lines per output file Can give a prefix for files to be generated as the last option

Examplesplit -b k <filename> <prefix>

Using split


Generate sorted output after reading a set of filesOptions

“-b” : ignore leading blanks “-f” : fold (ignore) case “-n” : numeric sort “-r” : reverse sort “-c” : check whether input is sorted “-k” : specify sort keys “-t” : specify separator “-o” : specify output file

Examplesort -k4,4 -k3 datafile -o outputfile

Using sort


Extract top N bytes / characters from a set of files.

Options: “-c N” : First N bytes “-n N” : First N lines “-q” : Quiet mode – Don't print file name headers “-v” : Verbose mode – Print file name headers

Example:head -n 5 datafile

Using head


Extract top N bytes / characters from a set of files.

Options: “-c N” : First N bytes “-n N” : First N lines

If N is written as +3 => extract all after 3rd line / byteIf N is written as -3 => extract last 3 lines / bytes

“-q” : Quiet mode – Don't print file name headers “-v” : Verbose mode – Print file name headers

Example:tail -n 5 datafile

Using tail


• Perform join operation on two files treating each line as a record and each column as a field.

• A white space separates fields by default.

• If join fields are not specified then the operation is performed using first field of each file.

• An output record (line) is generated for each pair of input records (lines) with identical join fields.

Using join


Options: “-1 N -2 M” : perform join using Nth field of first file and Mth field of

second file. “-a SIDE” : print unpairable lines from SIDE “-v SIDE : same as “a”; suppresses joined lines “-e TEXT” : replace empty fields with TEXT “-i” : ignore case “-t CHAR” : specify field separator “-o” : specify output format

Example:join -1 2 -2 1 f1 f2 -o 1.2,1.3,2.3

Using join


Short QuestionQ.1: Write a shell script which renames all .txt files as .text

files.Q.2: Explain different type of shells in Linux. Write a shell script

to generate Fibonacci series?Q.3:Write a shell script which takes a name as parameter and

returns the PID (s) of the processes with that name.Q.4: What are shell programming functions. Write a function

that adds two numbers.Q.5: Explain the usage of the test command for checking the

type of a file.


Long Questions

• Write a shell script which takes a name as parameter and returns the PID(s) of processes with that name.

• Write a shell program to evaluates an arithmetic expression like: ./calculator 10 + 20 ,where 10 is op1 and + is operator and 20 is op2. Using command line argument and operator may be (+,-,*,/ )

• Write a shell script for copying files. where the source file exists and has read permission . Check for it else display error message. Also check the target file should not exist in the directory


References:

• Sumitabha Das, “Unix Concepts and Applications”, TMH, 4th Ed., 2009.

• Arnold Robbins, “Linux Programming by Examples The Fundamentals”, Pearson Education, 2nd Ed., 2008.

Unit -3 Shell Programming

Documents

Transcript of Unit -3 Shell Programming