Unix Shell Programming Workshop

35
Introduction to Unix Introduction to Unix Shell Shell & & Scripting with Scripting with csh/tcsh csh/tcsh Brief Unix History Brief Unix History Unix Shell & Flavor Unix Shell & Flavor CSH/TCSH Scripts CSH/TCSH Scripts

description

Lot of commands covered for the Unix Shell Programming Workshop

Transcript of Unix Shell Programming Workshop

Page 1: Unix Shell Programming Workshop

Introduction to Unix Shell Introduction to Unix Shell & &

Scripting with csh/tcshScripting with csh/tcshBrief Unix HistoryBrief Unix HistoryUnix Shell & FlavorUnix Shell & FlavorCSH/TCSH ScriptsCSH/TCSH Scripts

Page 2: Unix Shell Programming Workshop

Unix Family TreeUnix Family Tree

Page 3: Unix Shell Programming Workshop

Unix ArchitectureUnix Architecture

Page 4: Unix Shell Programming Workshop

Unix History and MotivationUnix History and Motivation The first version of Unix came from AT&T in the The first version of Unix came from AT&T in the

early 1970s (Unix is old!).early 1970s (Unix is old!). Unix was developed by programmers and Unix was developed by programmers and for for

programmersprogrammers.. Unix is designed so that users can extend the Unix is designed so that users can extend the

functionalityfunctionality To build new tools easily and efficientlyTo build new tools easily and efficiently To customize the shell and user interface. To customize the shell and user interface. To string together a series of Unix commands to create To string together a series of Unix commands to create

new functionality.new functionality. To create custom commands that do exactly what we To create custom commands that do exactly what we

want.want.

Page 5: Unix Shell Programming Workshop

What is Shell?What is Shell?

Shell isShell is Command Interpreter that turns text that you Command Interpreter that turns text that you

type (at the command line) in to actions:type (at the command line) in to actions: User Interface: take the command from userUser Interface: take the command from user

Programming Shell can doProgramming Shell can do Customization of a Unix sessionCustomization of a Unix session ScriptingScripting

Page 6: Unix Shell Programming Workshop

Customization of a SessionCustomization of a Session Each shell supports some customization.Each shell supports some customization.

User promptUser prompt Where to find mailWhere to find mail Shortcuts (alias)Shortcuts (alias)

The customization takes place in The customization takes place in startupstartup files files Startup files are read by the shell when it Startup files are read by the shell when it

starts upstarts up The Startup files can differ for different shellThe Startup files can differ for different shell

Page 7: Unix Shell Programming Workshop

Types of ShellsTypes of Shells Interactive vs. Non-interactive; login or notInteractive vs. Non-interactive; login or not Interactive login shell started after loginInteractive login shell started after login Non-interactive shellNon-interactive shell

Present when shell script is runningPresent when shell script is running Just inherits parent’s environmentJust inherits parent’s environment

Interactive non-login shell startedInteractive non-login shell started Started from a command line Started from a command line Copies parent environment then invokes Copies parent environment then invokes

~/.bash_rc (or ~/.cshrc or ~/.tcshrc)~/.bash_rc (or ~/.cshrc or ~/.tcshrc)

Page 8: Unix Shell Programming Workshop

Popular ShellsPopular Shells

shsh Bourne Shell Bourne Shell ksh ksh Korn Shell Korn Shell csh,tcshcsh,tcsh C Shell (for this course)C Shell (for this course) bash bash Bourne-Again Shell Bourne-Again Shell

Page 9: Unix Shell Programming Workshop

Families of ShellsFamilies of Shells

Page 10: Unix Shell Programming Workshop

Flavors of Unix ShellsFlavors of Unix Shells Two main flavors of Unix ShellsTwo main flavors of Unix Shells

Bourne (or Standard Shell): sh, ksh, bash, zshBourne (or Standard Shell): sh, ksh, bash, zsh• FastFast• $ for command prompt$ for command prompt

C shell : csh, tcshC shell : csh, tcsh• better for user customization and scriptingbetter for user customization and scripting• %, > for command prompt%, > for command prompt

To check shell:To check shell: % echo $SHELL (shell is a pre-defined variable)% echo $SHELL (shell is a pre-defined variable)

To switch shell:To switch shell: % exec shellname (e.g., % exec bash)% exec shellname (e.g., % exec bash)

Page 11: Unix Shell Programming Workshop

Startup filesStartup files sh,ksh: sh,ksh: /etc/profile /etc/profile (out-of-the-box login shell settings)(out-of-the-box login shell settings)/etc/profile.local (addtnl. local system settings)/etc/profile.local (addtnl. local system settings)~/.profile ~/.profile (addtnl. user customized settings)(addtnl. user customized settings)~/.kcshrc~/.kcshrc (non-login shell user customization)(non-login shell user customization)

bash:bash:/etc/profile (out-of-the-box login shell settings) /etc/profile (out-of-the-box login shell settings) /etc/bash.bashrc (out-of-box non-login settings)/etc/bash.bashrc (out-of-box non-login settings)/etc/bash.bashrc.local (global non-login settings)/etc/bash.bashrc.local (global non-login settings)~/.bash_profile~/.bash_profile (login shell user customization) (login shell user customization)~/.bashrc~/.bashrc (non-login shell user customization)(non-login shell user customization)

~/.bash_logout (user exits from interactive login shell)~/.bash_logout (user exits from interactive login shell) csh/tcsh:csh/tcsh:

/etc/login/etc/login (out-of-the-box login shell settings) (out-of-the-box login shell settings)/etc/csh.login (non-login shell customizations)/etc/csh.login (non-login shell customizations)/etc/csh.login.local (global non-login settings)/etc/csh.login.local (global non-login settings)~/.login: (login shell user customizations)~/.login: (login shell user customizations)~/.cshrc: (non-login shell user customizations)~/.cshrc: (non-login shell user customizations)~/.cshrc.logout: (non-login shells at logout)~/.cshrc.logout: (non-login shells at logout)~/.logout: (read by login shells at logout)~/.logout: (read by login shells at logout)

Page 12: Unix Shell Programming Workshop

Some Special Keys Under tcshSome Special Keys Under tcsh

Ctrl-U = Delete everything on the command-Ctrl-U = Delete everything on the command-lineline

Ctrl-A = Move cursor to the frontCtrl-A = Move cursor to the front Ctrl-E = Move cursor to the endCtrl-E = Move cursor to the end Ctrl-P = Set the current command-line to the Ctrl-P = Set the current command-line to the

previous commandprevious command Ctrl-N = Set the current command-line to the Ctrl-N = Set the current command-line to the

next commandnext command TAB = Filename completionTAB = Filename completion

Page 13: Unix Shell Programming Workshop

Don’t forget your Best FriendDon’t forget your Best Friend

% man command (e.g., % man ls)% man command (e.g., % man ls) shows information about the commandshows information about the command usually space or usually space or Enter for more informationEnter for more information q to quitq to quit % man man% man man

Page 14: Unix Shell Programming Workshop

Create a shell scriptCreate a shell script Creating a simple shell scriptCreating a simple shell script

A shell script is a file that contains commands that the shell can A shell script is a file that contains commands that the shell can execute.execute.

• Any commands you enter in response to a shell prompt.Any commands you enter in response to a shell prompt. A utilityA utility A compiled programA compiled program Another shell scriptAnother shell script

• Control flow commands Control flow commands Run a shell scriptRun a shell script

Enter the script filename on the command lineEnter the script filename on the command line The shell interprets and execute the commands one after anotherThe shell interprets and execute the commands one after another

Why shell script?Why shell script? Simply and quickly initiate a complex series of tasks or a repetitive Simply and quickly initiate a complex series of tasks or a repetitive

procedure.procedure.

Page 15: Unix Shell Programming Workshop

Shell programmingShell programming Make the file executableMake the file executable

When you create a shell script using a editor, When you create a shell script using a editor, does it have execute permission typically? does it have execute permission typically? • Example: Example: (Make sure you are using tcsh/csh (Make sure you are using tcsh/csh

script!...)script!...)willow> echo $SHELLwillow> echo $SHELL/bin/tcsh/bin/tcshwillow> ./testwillow> ./test./test: Permission denied../test: Permission denied.willow> ls -l testwillow> ls -l test-rw-r--r-- 1 student ums 33 Sep 18 16:33 test-rw-r--r-- 1 student ums 33 Sep 18 16:33 testwillow> chmod +x testwillow> chmod +x testwillow> ./testwillow> ./testThis is Test!This is Test!

Page 16: Unix Shell Programming Workshop

Invoking a Shell scriptInvoking a Shell script Give the shell a command on the command line Give the shell a command on the command line

The shell forks a processThe shell forks a process• Which creates a non-interactive duplicate of the shell process Which creates a non-interactive duplicate of the shell process

The newly forked process attempts to exec the commandThe newly forked process attempts to exec the command• If the command is an executable program If the command is an executable program

Exec succeedsExec succeeds System overlays the newly created subshell with the executable System overlays the newly created subshell with the executable

programprogram• The command is a shell scriptThe command is a shell script

Exec failedExec failed The command is assumed to be a shell scriptThe command is assumed to be a shell script The subshell runs the commands in the shell.The subshell runs the commands in the shell.

Page 17: Unix Shell Programming Workshop

Invoking a Shell scriptInvoking a Shell script The shell itself is program The shell itself is program

It can be run as a command in a shell and It can be run as a command in a shell and also accepts arguments. Note: Let’s find your also accepts arguments. Note: Let’s find your default shell executing “echo $SHELL”default shell executing “echo $SHELL”

willow> echo $SHELLwillow> echo $SHELL/bin/tcsh/bin/tcsh

To run a shell script To run a shell script Which does not have executable permissionWhich does not have executable permission

Ex: willow>tcsh testEx: willow>tcsh test Run the script with different shell other than Run the script with different shell other than

your interactive shellyour interactive shellEx: willow>sh testEx: willow>sh test

Page 18: Unix Shell Programming Workshop

Invoking a Shell script Invoking a Shell script Put special characters on the first line of a shell script Put special characters on the first line of a shell script

To tell OS checks what kind of file it is before attempting to To tell OS checks what kind of file it is before attempting to exec itexec it

To tell which utility to use (sh, csh, tcsh, …)To tell which utility to use (sh, csh, tcsh, …) Special sequenceSpecial sequence

The firsts two character of a script are #! The firsts two character of a script are #! Then followed by the absolute pathname of the program that Then followed by the absolute pathname of the program that

should execute the script should execute the script Ex:Ex:

willow> more testwillow> more test#!/bin/tcsh#!/bin/tcsh# This line will not run since it is commented out...# This line will not run since it is commented out...echo 'This is Test!‘echo 'This is Test!‘

Page 19: Unix Shell Programming Workshop

Make a comment #Make a comment #

Comments make shell scripts easier to Comments make shell scripts easier to read and maintainread and maintain

Pound sign (#) start a comment line until Pound sign (#) start a comment line until the end of that line as second line in the end of that line as second line in previous example, exceptprevious example, except #! In the first line. #! In the first line. Or inside quotesOr inside quotes

Page 20: Unix Shell Programming Workshop

Parameters and VariablesParameters and Variables A shell parameter is associated with a value that is A shell parameter is associated with a value that is

accessible to the user.accessible to the user. Shell variablesShell variables

• Names consist of letters, digits and underscoresNames consist of letters, digits and underscores By convention, environment variables use UPPERCASEBy convention, environment variables use UPPERCASE

• User created variables (create and assign value)User created variables (create and assign value)• Keyword shell variablesKeyword shell variables

Have special meaning to the shellHave special meaning to the shell Being created and initialized by the startup fileBeing created and initialized by the startup file

Positional parametersPositional parameters• Allow you to access command line argumentsAllow you to access command line arguments

Special parametersSpecial parameters• Such asSuch as

The name of last commandThe name of last command The status of most recently executed commandThe status of most recently executed command The number of command-line argumentsThe number of command-line arguments

Page 21: Unix Shell Programming Workshop

Positional ParametersPositional Parameters The command name and arguments are the The command name and arguments are the

positional parameters. positional parameters. Because you can reference them by their position on Because you can reference them by their position on

the command linethe command line $0 : Name of the calling program$0 : Name of the calling program $1 - $9 : Command-line Arguments$1 - $9 : Command-line Arguments

• The first argument is represented by $1The first argument is represented by $1• The second argument is represented by $2The second argument is represented by $2• And so on up to $9And so on up to $9• The rest of arguments have to be shifted to be able to use The rest of arguments have to be shifted to be able to use

$1- $9 parameters. $1- $9 parameters.

Page 22: Unix Shell Programming Workshop

Positional ParametersPositional Parameters Example:Example:

•Change directory to your assigned numbered subdirectory

willow> cd 1

•List the directory contents, confirming display_5args

willow> ls -l display_5args

•Change mode of display_5args to executable

willow> chmod +x display_5args

•Execute the script

willow> ./display_5args 1 2 3 4 5

you are running script ./display_5args with parameter 1 2 3 4 5

Page 23: Unix Shell Programming Workshop

Positional ParametersPositional Parameters $1-$9 allows you to access 10 arguments$1-$9 allows you to access 10 arguments

How to access others?How to access others? Promote command-line arguments: shiftPromote command-line arguments: shift

Built-in command shift promotes each of the Built-in command shift promotes each of the command-line arguments.command-line arguments.

• The first argument ( which was $1) is discardedThe first argument ( which was $1) is discarded• The second argument ( which was $2) becomes $1The second argument ( which was $2) becomes $1• The third becomes the secondThe third becomes the second• And so onAnd so on

Makes additional arguments availableMakes additional arguments available Repeatedly using shift is a convenient way to Repeatedly using shift is a convenient way to

loop over all the command-line argumentsloop over all the command-line arguments

Page 24: Unix Shell Programming Workshop

Positional ParametersPositional Parameters

Example:Example:willow> more demo_shiftwillow> more demo_shift#!/bin/tcsh#!/bin/tcshecho $1 $2 $3echo $1 $2 $3shiftshiftecho $1 $2echo $1 $2shiftshiftecho $1echo $1willow> ./demo_shift 1 2 3willow> ./demo_shift 1 2 31 2 31 2 32 32 333

Page 25: Unix Shell Programming Workshop

willow> more demo_shiftwillow> more demo_shift

#!/bin/tcsh#!/bin/tcsh

echo $1 $2 $3echo $1 $2 $3

shiftshift

echo $1 $2echo $1 $2

shiftshift

echo $1echo $1

shiftshift

echo $?echo $?

shiftshift

echo $?echo $?

shiftshift

echo $?echo $?

willow> ./demo_shift 1 2 3 4willow> ./demo_shift 1 2 3 4

1 2 31 2 3

2 32 3

33

00

00

shift: No more words.shift: No more words.

Page 26: Unix Shell Programming Workshop

Special ParametersSpecial Parameters Useful valuesUseful values

Command-line argumentsCommand-line arguments Execution of shell commandsExecution of shell commands Can not change the value directly, like positional parametersCan not change the value directly, like positional parameters

Value of Command-line arguments: $* and $@Value of Command-line arguments: $* and $@ $* and $@represent all the command_line arguments ( not just $* and $@represent all the command_line arguments ( not just

the first nine)the first nine) ““$*” : treats the entire list of arguments as a single argument$*” : treats the entire list of arguments as a single argument ““$@” : produce a list of separate arguments (Only bash/ksh/sh)$@” : produce a list of separate arguments (Only bash/ksh/sh)

Page 27: Unix Shell Programming Workshop

TCSH SCRIPT WITH $*and $@

willow> more for_test#!/bin/tcsh

echo 'using $*'foreach arg ($*) echo "$arg"end

echo 'using $@'foreach arg ($@) echo "$arg"end

willow> ./for_test 1 2 3 using $*123using $@Illegal variable name.

BASH SCRIPT WITH $*and $@

willow> more for_test.bash#!/bin/bash

echo "using \$* "for arg in "$*"do echo "$arg"doneecho "using \$@ "for arg in "$@"do echo "$arg"donewillow> ./for_test.bash 1 2 3 using $* 1 2 3using $@ 123

Page 28: Unix Shell Programming Workshop

Special ParametersSpecial Parameters

The number of arguments: $#The number of arguments: $# Return a decimal numberReturn a decimal number Use the test to perform logical test on this Use the test to perform logical test on this

numbernumberwillow> more num_args

echo this script is called with $# arguments.

willow> chmod +x num_args

willow> ./num_args

this script is called with 0 arguments.

willow> ./num_args 1

this script is called with 1 arguments.

willow> ./num_args 2

this script is called with 1 arguments.

willow> ./num_args 0

this script is called with 1 arguments.

Page 29: Unix Shell Programming Workshop

Special ParametersSpecial Parameters Exit status: $?Exit status: $?

When a process stops executing for any reason, When a process stops executing for any reason, it returns an exit status to its parent process. it returns an exit status to its parent process.

By convention,By convention,• Nonzero represents a false value that the command Nonzero represents a false value that the command

failed.failed.• A zero value is true and means that the command was A zero value is true and means that the command was

successfulsuccessful You can specify the exit status that a shell script You can specify the exit status that a shell script

returns by using the exit built-in followed by a returns by using the exit built-in followed by a numbernumber

• Otherwise, the exit status of the script is the exit status Otherwise, the exit status of the script is the exit status of the last command the script ran. of the last command the script ran.

Page 30: Unix Shell Programming Workshop

willow> ls awillow> ls aa: No such file or directorya: No such file or directorywillow> echo $?willow> echo $?22willow> echo olemisswillow> echo olemissolemissolemisswillow> echo $?willow> echo $?00

willow> more exit_statuswillow> more exit_statusecho this program will have the exit code of 8.echo this program will have the exit code of 8.exit 8exit 8willow> ./exit_statuswillow> ./exit_statusthis program will have the exit code of 8.this program will have the exit code of 8.willow> echo $?willow> echo $?88willow> echo $?willow> echo $?00

Page 31: Unix Shell Programming Workshop

SummarySummary A shell is both a command interpreter and a programming A shell is both a command interpreter and a programming

language.language. Job controlJob control

Control-z/fg/bg/&Control-z/fg/bg/& VariablesVariables

Local and environment variablesLocal and environment variables Declare and initialize a variable ( no type)Declare and initialize a variable ( no type) Export unsetExport unset

Command line expansionCommand line expansion Parameter expansion/variable Parameter expansion/variable

expansion/command/substitution/pathname expansionexpansion/command/substitution/pathname expansion Quote ( ‘ ‘ “ “ \ ) Quote ( ‘ ‘ “ “ \ )

• “ “ “ “ all but parameter, variable expansion and \all but parameter, variable expansion and \• ‘ ‘ ‘ ‘ suppress all types of expansionsuppress all types of expansion• \ escaping the following special character\ escaping the following special character

Page 32: Unix Shell Programming Workshop

Basic Script ExampleBasic Script Examplewillow> more basic_script willow> more basic_script #!/bin/tcsh#!/bin/tcshecho 'Listing the files in long format appending due date/time' echo 'Listing the files in long format appending due date/time' echoechols -lrtahls -lrtah

echoechoecho 'Listing the files in long format appending due date/time' echo 'Listing the files in long format appending due date/time' echoechodf -k df -k

# Using diff to find two files differences and writing them to another file # Using diff to find two files differences and writing them to another file diff -c for_test.bash for_test >> file_differences &diff -c for_test.bash for_test >> file_differences &

echoechoecho 'sleeping mode for 4 seconds. Please wait!'echo 'sleeping mode for 4 seconds. Please wait!'echoechosleep 4sleep 4

echoechoecho 'GO REBELS'echo 'GO REBELS'echo 'To find out the differences of files for_test and for_test.bash, 'echo 'To find out the differences of files for_test and for_test.bash, 'echo 'Please open file_differences via using cat command as shown below:'echo 'Please open file_differences via using cat command as shown below:'echo 'cat file_differences’echo 'cat file_differences’

Page 33: Unix Shell Programming Workshop

Killing Killing BADBAD Processes Processes The “kill” command:The “kill” command:

kill [-<signal>] <pid>kill [-<signal>] <pid>Send Send <signal> <signal> to process to process <pid><pid>

The “killall” command:The “killall” command:killall [-<signal>] <command>killall [-<signal>] <command>Send Send <signal> <signal> to all processes that start with to all processes that start with <command><command>

Useful signals (Useful signals (kill –l kill –l for the complete list):for the complete list):TERMTERM the default, “the default, “termterminate”, kills things inate”, kills things

nicelynicelyKILLKILL will kill anything, but not nicelywill kill anything, but not nicelyHUPHUP ““hhangangupup”, used to reload configurations”, used to reload configurationsSTOPSTOP stops (suspends) a running processstops (suspends) a running process

Page 34: Unix Shell Programming Workshop

SummarySummary Shell parametersShell parameters

HOMEHOME PATHPATH PS1PS1 SHELLSHELL $0$0 $n$n $*$* $@$@ $#$# $$$$ $!$! $?$?

Page 35: Unix Shell Programming Workshop

SummarySummary Special CharactersSpecial Characters

NEWLINENEWLINE ;; ()() && || >> >>>> << <<<<

** ?? \\ ‘‘ ““ ` `` ` [][] $$ . . ## &&&& |||| !!