Unix Shell Scripting

Instructor

Mustafa Qasim

Certified Ethical Hacker (CEH)

Oracle Certified Expert Solaris Security Administrator

Oracle Certified Professional Solaris System Administrator

Oracle Certified Associate Solaris 10 Operating System

Oracle Certified Associate MySQL 5.5/5

Cisco Certified Network Associate (CCNA)

Catch Me

LinkedIn pk.linkedin.com/in/mustafaqasim

Google+ gplus.to/mustu

Twitter twitter.com/mustafaqasim

Introduction● Name● Job Description● Professional Experience● Expectations from Unix Shell Scripting course

Ideal Participants

Hardworking?

Ideal Participants

Hardworking?ŊÖ

Ideal Participants

LaaazzZZzzyyy....

Let's Start...

Unix● AT&T Employees at Bell Labs in 1969● Ken Thompson, Dennis Ritchie, Brian Kernighan, Douglas McIlroy,

Michael Lesk and Joe Ossanna● Unix Family OS

– University of California, Berkeley's BSD– Solaris– HP-UX– AIX– Sequent and as well as Darwin

GNU● GNU Not Unix● Richard Stallman, 1983● GNU General Public License (GPL)● GNU Compiler Collection (GCC), the GNU C library (glibc), and

GNU Core Utilities (coreutils),[1] but also the GNU Debugger (GDB), GNU Binary Utilities (binutils),[18] and the bash shell.

● GNU Hurd Kernel● Free Software Foundation (FSF)

GNU/Linux● In April 1991, Linus Torvalds, a 21-year-old

student at the University of Helsinki, Finland started working on some simple ideas for an operating system

● Kernel vs OS

Linus Torvalds

To Do

Make yourself familiar with Unix, Unix Derivatives, GNU, Linux and software license GPL and Free & Open Source Software (FOSS).

http://en.wikipedia.org/wiki/Unix

http://www.gnu.org/philosophy/philosophy.html

http://en.wikipedia.org/wiki/GNU

http://www.gnu.org/gnu/why-gnu-linux.html

http://en.wikipedia.org/wiki/Linux

To Do

Make yourself familiar with Unix, Unix Derivatives, GNU, Linux and software license GPL and Free & Open Source Software (FOSS).

http://en.wikipedia.org/wiki/Unix

http://www.gnu.org/philosophy/philosophy.html

http://en.wikipedia.org/wiki/GNU

http://www.gnu.org/gnu/why-gnu-linux.html

http://en.wikipedia.org/wiki/Linux

So, Where comes the Shell?

Shell?

Shell?

Compiler vs Interpreted

Complete Program > Compiler > Executable

Program Fragment > Interpreter > Instant Output

Compiler vs Interpreted (Contd.)● Compiled Languages

– C/C++– Pascal– FORTRAN

● Interpreted Languages– Unix Shell– Perl– Tcl/Tk

Unix Shell● First shell for Version 6 Unix was written by Ken Thompson

(Bell Labs) in 1971● It was only Interactive Shell

Bourne Shell● Created by Stephen Bourne at AT&T Bell Labs for V7 UNIX● Goals

– Interpreted Language– Scripting

● New Features– Control Flows, Loops, Variables

● Lacked Functions

Bourne Shell (Contd.)● Bourne Shell Derivatives

– Korn Shell (ksh)– Almquist shell (ash)– Bourne Again Shell (Bash)

C Shell (csh)● Bill Joy in 1978 for BSD● Create a scripting language similar to C● Prominent Feature

– Command History● Tenex Enhancements (tcsh)

– Command completion– Command line editing

Korn Shell (ksh)● By David Korn● Backward compatibility with Bourne Shell (sh)● Derivative of Bourne Shell (sh)

Bourne Again Shell (Bash)● GNU Project● Superset of sh and ksh

POSIX● Institute of Electrical and Electronics Engineering (IEEE)● International Organization for Standardization (ISO)● POSIX Compliance● Bash, gawk

Responsibilities of Shell

1. Reading input and parsing the command line

2. Evaluating special characters, such as wildcards and the history character

3. Setting up pipes, redirection, and background processing

4. Handling signals

5. Setting up programs for execution

To Do● Perform Login and Logout on a Unix shell● Perform and understand the following commands

– ls, dir, touch, mkdir, cd, rm, mv, cat, less, more, which, pwd, man, info

● Use vi editor to create and edit files● Understand Unix File System hierarchy especially the

following directories– /bin, /sbin, /home, /root, /var

Unix Startup● OS is booted and very first process is born

– Init, PID 1● OS authenticates the user and provides access to shell

– /bin/login● Data Streams are set

– Stdout, stderr, stdin● A process is forked to run Graphical Desktop

Parsing Commands● Lexical Analysis

– The process of breaking the line up into tokens is called lexical analysis.

● Tokens● Command?

– Built-in command => Execute Internally– Executable command => Fork a Child Shell

● Forking Child Shel

Types of Commands

1. Aliases

2. Keywords

3. Functions

4. Built-in commands

5. Executable programs

Shell Processes● PID● ps● Linux: pstree● Solaris: ptree

System Calls

Kernel

fork

exec

wait

exit }

fork System Call● Parent/Child● Inheritance of environment, open files, real and user

identifications, umask, current working directory, and signals.

fork System Call

wait System Call● Waits for Child's PID & Exit Code● Zombie Process● Orphan Zombie Process

exec System Call● exec call for executable● Executable becomes the child

exit System Call● sigchild signal to parent● Exit code 0 - 255

To Do● Perform and Understand the following commands

– top, ps, pgrep, pstree/ptree, kill● Learn to hunt a specific process and kill it● Learn to hunt a specific instance of a binary running multiple

times simultaneous

Environment and Inheritance● UID, GID● Ownership● Permission● Variables● Working Directory

User & Group IDs

$ id

uid=502(mustu) gid=502(mustu)

File Creation Mask● Umask

777 (Directory) 666 (File)

–022 (umask value) –022 (umask value)

------- ---------

755 644

drwxr-xr-x -rw-r--r--

Set Ownership & Permission● chown● chmod

Variables● Local Variables● Environment Variables

– $env

File Descriptor● 0 stdin● 1 stdout● 2 stderr

I/O Redirection

I/O Redirection

I/O Redirection

Pipe

Signals

To Do● Understand & Practice the following

– Bash Environmental variables– Check your current environment variables (hint: $env)– Check your user's UID & GID– Check a file's permission set for owner, group and others– Change a file/folder permission using both numerical and character format– Change a file/folder owner and group– Redirect stdout and stderr into a file– Pipe or Redirect output of programs

Let's write a Bash script ;-)

Writing Bash Script● Shebang

– #!/bin/bash● Comments

– #Getting input for value of age● Make script executable

– Chmod +x myscript.sh● Execute Script

– ./myscript.sh

Simple Bash Script

#!/bin/bash

# This is a comment

echo “Khamoshi Ka Boycot”

Use Variable

#!/bin/bash

name=”Telenor”

echo “$name Khamoshi Ka Boycot”

Pass Arguments

./myscript ali 15 Lahore

#!/bin/bash

echo “Mr. $1 is $2 years old and belongs to $3”

Get Input

#!/bin/bash

echo "What is your name?"

read uname

echo ” Welcome $uname”

Arithmetic Operators

== Equal to

!= Not equal to

> Greater then

>= Greater then, equal to

< Less then

<= Less then, equalt o

Logical Operators

&& and

|| or

! not

Conditional Statements

if Construct:

if command

then

block of statements

fi

Conditional Statements

if/else Construct:

if command

then

block of statements

else

block of statements

fi

Conditional Statementsif/else if/else Construct:

if command

then

block of statements

elif command

then

block of statements

else

block of statements

fi

Iterative Statements

for variable in word_list

do

block of statements

done

Schedule Script● Cron

– crontan -l #List Exiting Crons– crontab -e #Edit Cron Scheduling File

REMEMBER● Environment Variable Differ in Cron● Use the MAIL parameter to receive cron errors● Use the PATH parameter to set environment variables in cron● If your program needs to use neighbor files to run properly

then in cron script first change directory (cd) to your program's path then run the program.

Tip :)

Your best buddy

http://www.corntab.com/

Regular Expressions● Pattern of characters used to match the same characters in a

search● Enclosed by forward slash /telenor/

Regular Expressions

Hi Asad,

We've a upcoming training on Unix shell Programming later this week. As you've to deal with programming tasks frequently I would highly recommend you to participate in this training. It will improve your programming skills and help you perform daily tasks efficiently.

Thanks

Regular Expressions

:1,$s/programming/scripting/g

Regular Expression Metacharacters

:1,$s/\<[Pp]rogramming\>/scripting/g

1,$ First Line to Last Line

s Substitute

\<, \> Beginning and End of Word

[Pp] Match one of the word

<[Pp]rogramming\> Programming OR programming

g Globally

Regular Expression MetacharactersMeta Character

^

Function

Beginning of line anchor

Example

/^lahore/

What It Matched

Matches all lines beginning with lahore

Regular Expression MetacharactersMeta Character

$

Function

End of line anchor

Example

/lahore$/

What It Matched

Matches all lines ending with lahore

Regular Expression MetacharactersMeta Character

.

Function

Matches one character

Example

/la...e/

What It Matched

Matches all starting with la, followed by three characters, followed by e

Regular Expression MetacharactersMeta Character

*

Function

Matches zero or more of the preceding characters

Example

/ *lahore/

What It Matched

Matches lines with zero or more spaces, followed by the pattern lahore

Regular Expression MetacharactersMeta Character

[]

Function

Matches one in the set

Example

/[Ll]ahore/

What It Matched

Matches lines containing Lahore or lahore

Regular Expression MetacharactersMeta Character

[x-y]

Function

Matches one character within a range in the set

Example

/[A-Z]ahore/

What It Matched

Matches letters from A through Z followed by ahore

Regular Expression MetacharactersMeta Character

[^]

Function

Matches one character not in the set

Example

/[^A-Z]ahore/

What It Matched

Matches any character not in the range between A and Z

Regular Expression MetacharactersMeta Character

\

Function

Used to escape a metacharacter

Example

/lahore\./

What It Matched

Matches lines containing lahore, followed by a literal period; Normally the period matches one of any character

Regular Expression MetacharactersMeta Character

\<

Function

Beginning-of-word anchor

Example

/\<lahore/

What It Matched

Matches lines containing a word that begins with lahore (supported by vi and grep)

Regular Expression Metacharacters Additional

Meta Character

\>

Function

Ending-of-word anchor

Example

/lahore\>/

What It Matched

Matches lines containing a word that ends with lahore (supported by vi and grep)

Regular Expression Metacharacters Additional

Meta Character

\(..\)

Function

Tags match characters to be used later

Example

/\(love\)able \1er/

What It Matched

May use up to nine tags, starting with the first tag at the leftmost part of the pattern. For example, the pattern love is saved as tag 1, to be referenced later as \1. In this example, the search pattern consists of lovable followed by lover (supported by sed, vi, and grep)

Regular Expression Metacharacters Additional

Meta Character

\(..\)

Function

Tags match characters to be used later

Example

/\(love\)able \1er/

What It Matched

May use up to nine tags, starting with the first tag at the leftmost part of the pattern. For example, the pattern love is saved as tag 1, to be referenced later as \1. In this example, the search pattern consists of lovable followed by lover (supported by sed, vi, and grep)

Regular Expression Metacharacters Additional

Meta Character

x\{m\} or x\{m,\} or x\{m,n\}

Function

Repetition of character x, m times, at least m times, at least m and not more than n times

Example

o\{5,10\}

What It Matched

Matches if line contains between 5 and 10 consecutive occurrences of the letter o (supported by vi and grep)

Sed● Streamlined, non-interactive editor

How Sed Works

abc

Text File

abc

Sed Buffer

abc

Sed Output

Awk● UNIX programming language used for manipulating data and

generating reports

How Awk Works

Ahmad 33 LHRShakir 35 ISBQasim 28 KHI

Text File

Ahmad 33 LHR

Awk takes a line and put in an internal variable $0

Ahmad 33 LHR

Line is broken into fields separated by spaces and stored into internal numbered variables starting from $1

$0

$1 $2 $3

How Awk Works

Ahmad 33 LHRShakir 35 ISBQasim 28 KHI

data.txt

#awk '{print $1, $3}' data.txt

Ahmad LHRShakir ISBQasim KHI

To Do● Practice Regular Expressions with grep, sed and awk.