EECS 470 Lab 5 - Linux Shell Scripting
Transcript of EECS 470 Lab 5 - Linux Shell Scripting
EECS 470 Lab 5Linux Shell Scripting
Department of Electrical Engineering and Computer ScienceCollege of EngineeringUniversity of Michigan
Friday, 1st October, 2021
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 1 / 40
Overview
Administrivia
UNIXFilesUtilitiesConnecting Utilities
Bourne Again ShellVariablesFlow ControlFunctionsGlobbing
Scripting
Assignment
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 2 / 40
Administrivia
Administrivia
HomeworkI Homework 3 is due Monday, 4th October at midnight
ProjectsI Project 3 is due Saturday, 2nd October at midnightI If you haven’t, you need to get started now
We are available to answer questions on anything here. Office hours can befound in the course website.
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 3 / 40
UNIX
UNIX
What is UNIX?I Mainframe operating systemI Written by Dennis Ritchie, Ken Thompson and Rob Pike (among
others) at Bell LabsI The basis for many modern operating systems, e.g. Linux, BSD, Mac
OSX
History of UNIXI Written at Bell Labs in 1969I First version of BSD is installed in 1974I Last Bell Labs UNIX (Version 7) is published in 1979I The GNU Project is started by Richard StallmanI Linux Torvalds writes a monolithic kernel operating system in 1991
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 4 / 40
UNIX
UNIX Philosophy
“This is the Unix philosophy: Write programs that do one thing and do itwell. Write programs to work together. Write programs to handle textstreams, because that is a universal interface.”
– Douglas McIlroy
“Everything is a file descriptor or a process.”– Linus Torvalds
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 5 / 40
UNIX Files
Files
What is a file?I Anything referenced through a filesystemI Anything with a file descriptor
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 6 / 40
UNIX Files
Types of Files
I Regular – Anything not in one of the following categoriesI Directory – Can contain other files and directories (read up on inodes
sometime)I Symbolic Link – A pointer to another fileI Pipe – Covered in a few slidesI Socket – Covered in EECS 482/489
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 7 / 40
UNIX Files
Permissions
r w x - - - - - -User Group Other
Table: UNIX permissions are representedwith one bit for each permission in eachcategory, e.g. 700.
Three PermissionsI ReadI WriteI Execute
Three CategoriesI UserI GroupI Other
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 8 / 40
UNIX Utilities
Utilities
What is a utility?I A program used to process text streams/filesI Called from some command line/shell
Why do I care?I Utilities form the basis of “Linux skills”I Useful for automationI Necessary for today’s lab
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 9 / 40
UNIX Utilities
Navigation Utilities
pwdI Description: print working directory – where you areI Synopsis: pwd [OPTIONS]
lsI Description: list directory contentsI Synopsis: ls [OPTIONS]... [FILE] ...
cdI Description: change directoryI Synopsis: cd [OPTIONS] [PATH]
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 10 / 40
UNIX Utilities
File Utilities
cpI Description: copy filesI Synopsis: cp [OPTONS]... SOURCE DEST
mvI Description: move filesI Synopsis: mv [OPTONS]... SOURCE DEST
rmI Description: remove filesI Synopsis: rm [OPTONS]... FILE...
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 11 / 40
UNIX Utilities
diff
DescriptionI Shows the line-by-line differences between filesI Good for checking if your output is correct
SynopsysI diff [OPTIONS] FILES
ExamplesI diff -uy ../project3_correct/writeback.out writeback.outI vimdiff ../project3_correct/writeback.out writeback.out
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 12 / 40
UNIX Utilities
Graphical Diff
Alternatives to diffParsing output of diff is hard, so it might be useful to use some kind of“graphical diff” tool, like vimdiff, which opens up two files side-by-side inVim showing their differences. Try it and you’ll see how much easier toparse this.
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 13 / 40
UNIX Utilities
grep
DescriptionI Print lines matching a pattern
SynopsysI grep [OPTIONS] PATTERN [FILE]
ExamplesI grep ’@@@’ program.outI ps -axfuw | grep "$USER" | grep "vcs"
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 14 / 40
UNIX Utilities
Regular Expressions
I Really powerful/usefulI Complicated, and beyond the scope of this presentationI Read up on them
I at WikipediaI in the grep manual
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 15 / 40
UNIX Utilities
man
DescriptionI An interface to the on-line reference manuals (pager)
SynopsysI man PAGE
ExamplesI man grepI man diff
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 16 / 40
UNIX Utilities
Pager
DefinitionI A program which allows browsing of large text files by breaking them
into screen-sized chunks.
ExamplesI lessI man
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 17 / 40
UNIX Utilities
Other Utilities
These will be useful. . .I cutI touchI teeI xargs
I tailI columnI findI less
These are harder, but even more useful. . .
I sedI awkI patchI vi(m)
I fmtI tmux
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 18 / 40
UNIX Connecting Utilities
Program Features
Methods of CommunicationI Standard Text Streams
I stdinI stdoutI stderr
I Return value/code
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 19 / 40
UNIX Connecting Utilities
Return Codes
What is a return code?I The integer value a program returns (e.g. return(0);)I Conventionally, returning zero indicates success, non-zero failureI Specific values other than zero mean different things for different
programs
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 20 / 40
UNIX Connecting Utilities
Standard Text Streams
stdinI The default input to a programI From a keyboard by default
stdoutI The default output of a programI To a display by default
stderrI The default error output of a programI To a display by default, requires additional effort to save
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 21 / 40
UNIX Connecting Utilities
Connecting Utilities
Why do we want to connect utilities?I Combination jobs without intermediate filesI e.g. take the diff of two different grep operations (what you need to
do for today’s lab)
How can we connect utilities?I PipesI Redirection
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 22 / 40
UNIX Connecting Utilities
Pipe
What is a pipe?I Connects the stdout of one program to the stdin of anotherI Does not connect stderr
How do I use one?I Call a program on one side of the | and then call another on the other
sideI e.g. dmesg | less
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 23 / 40
UNIX Connecting Utilities
Pipe: xargs
ProblemWhat if we want to pipe to utility that uses arguments instead of input?
Solution: xargsxargs splits input into individual items and calls the program that is itsargument once for each input.
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 24 / 40
UNIX Connecting Utilities
Redirection
What is redirection?I Allows for modification of the standard text streams
I stdin is 0I stdout is 1I stderr is 2
I Several types:0< Use a file instead of the keyboard for stdini> Use a file instead of the terminal for the stream i
i>> Like i>, but append to a file instead of overwritingi>&j Put stream i into the same place as stream j
Advanced Bash Scripting Guide: I/O Redirection
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 25 / 40
UNIX Connecting Utilities
Redirection by Example
ExampleI ./vs-asm < test_progs/evens.s > program.mem
ExampleI make | tee 2>&1 build.out
ExampleI ./test 1>&2 2>&3
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 26 / 40
Bourne Again Shell
Shell
What is a shell?I Before we had graphical environments, we had text shellsI Basically, an interpreter for commands, executing programs and saving
informationI Possibly a Read-Execute-Print-Loop (REPL)
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 27 / 40
Bourne Again Shell
Bourne Again Shell
What is BASH?I Stands for the Bourne Again ShellI Created in 1989 by Brian FoxI Default shell in most Linux distributions and Mac OSX
Why BASH?I What I learned firstI Default in CAEN Redhat, finally
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 28 / 40
Bourne Again Shell
Warning
WarningEverything after this slide will be specific to BASH. Other shells behavesimilarly, but not identically. If you want to use something else (e.g. ZSH,TCSH, etc.), please find other resources.
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 29 / 40
Bourne Again Shell Variables
Variables
BASH VariablesI Store dataI Contain text, for the most partI No type system
SyntaxI Assignment/Declaration
I variable=valueI Referencing
I $variableI ${variable}
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 30 / 40
Bourne Again Shell Variables
Variable Scope
ScopeI Variables exist inside the shell they’re inI Unless exported
e.g. export EDITOR=vimI This is very important in scripts, particularly the shell startup scripts
(.bashrc, .bash_profile)
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 31 / 40
Bourne Again Shell Variables
Special Variables
$# The number of command line arguments$0 The name of the script/function called$1 The first argument to the script/function$? The return code of the last program run in this shell
$USER The current user$HOME The user’s home directory
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 32 / 40
Bourne Again Shell Flow Control
Flow Control
if/else
if [ var -eq "string" ]then
commandelif [ var -eq "string2" ]then
command2else
command3fi
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 33 / 40
Bourne Again Shell Flow Control
Flow Control
case
case "$var" inval)
command;;val2)
command;;( * )
default;;
esac
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 34 / 40
Bourne Again Shell Flow Control
Flow Control
for
for file in ./*; docommand $filecommand2
done
for (( a=1; a <= LIMIT; a++ )) docommandcommand2
done
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 35 / 40
Bourne Again Shell Flow Control
Conditionals
TestingI Testing happens in [ ]I String tests are different than arithmetic testsI Generally use -lt, -gt, -le, -ge, -eq, -neI Tests for files are special, e.g. [ -x simv ] makes sure that the
binary simv has execution permissions
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 36 / 40
Bourne Again Shell Functions
Functions
FunctionsI Packages of commandsI Arguments are referenced by positionI Useful for packaging up commonly reused bits
Syntax
function (){
commands}
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 37 / 40
Bourne Again Shell Globbing
File Globbing
What is globbing?I File name wildcarding in the shellI Expansion is done, and then passed to the command to be executedI e.g. test_progs/*.s
What should I know globbing?I Superior to parsing ls output in everywayI Can get more complicated, see this page of the BASH manual.
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 38 / 40
Scripting
Scripting
1. Write a series of shell commands into a text file2. On the first line of the file, specify an interpreter
e.g. #!/bin/bash3. Name it something appropriate
e.g. test.sh4. Add execute permissions
e.g. chmod +x test.sh
5. Run the scripte.g. $ ./test.sh
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 39 / 40
Assignment
Lab Assignment
I Assignment is posted to the course website as Lab 5 Assignment.I If you get stuck. . .
I Ask a neighbor, quietlyI Put yourself in the help queue
I When you finish the assignment, sign up in the help queue and markthat you would like to be checked off.
I If you are unable to finish today, the assignment needs to be checkedoff by a GSI in office hours before the next lab session.
(University of Michigan) Lab 5: Scripting Friday, 1st October, 2021 40 / 40