Post on 03-Apr-2018
7/29/2019 2013 Unix Wk2 Handout (1)
1/16
3/11/13
1
FACULTY OFENGINEERING &
INFORMATIONTECHNOLOGIES
Introduction to Unix
COMP2129
Bob Kummerfeld
Files and Shells
COMMONWEALTH OF AUSTRALIACopyright Regulations 1969
WARNINGThis material has been reproduced and communicated to you by or on behalf ofthe University of Sydneypursuant to Part VB of the Copyright Act 1968 (the Act).The material in this communication may be subject to copyright under the Act. Anyfurther copying orcommunication of this material by you may be the subject ofcopyright protection under the Act.
Do not remove this notice.
The file system
root
bin
usr home
pub
tmpdev
ascii
etc
passwdken hrmt
Ass1 Erat
Erat.c a.out
doit
Paths and full path names
root
bin
usr home
pub
tmpdev
ascii
etc
passwdken hrmt
Ass1 Erat
Erat.c. a.out
doit
/
Paths and full path names
root
bin
usr home
pub
tmpdev
ascii
etc
passwdken hrmt
Ass1 Erat
Erat.c. a.out
doit
/
/usr/pub/ascii
Full path name
Paths and full path names
root
bin
usr home
pub
tmpdev
ascii
etc
passwdken hrmt
Ass1 Erat
Erat c. a.out
doit
/
/usr/pub/ascii
Full path name
Relative path name
.
..
7/29/2019 2013 Unix Wk2 Handout (1)
2/16
3/11/13
2
Paths and full path names
root
bin
usr home
pub
tmpdev
ascii
etc
passwdken hrmt
Ass1 Erat
Erat c. a.out
doit
/
/home/hrmt
/usr/pub/ascii
Full path name
Relative path name
.
..
Home directory ~
~hrmt
Files
Several types of files- Ordinary files
- Contain data, eg text, program- Directories
- Actually a special file that contains a list of files- Devices
- The file is a name that refers to a device such as a disk drive ornetwork interface
- special files- The content of these files is generated when the file is read
- Eg the files in /proc on a linux system
File command
The file command tries to work out what type of file you have:
bash$ file Sunix
Sunix directory
bash$ cd Sunix
bash$ file notes
notes: English text
File command and C files
bash$ file fperror.c
fperror.c: C program text
bash$ gcc fperror.c
bash$ file a.outa.out: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked
(uses shared libs), not stripped
bash$ cp fperror.c fperrorbash$ gcc fperrorfperror: file not recognized: File format not recognized
collect2: ld returned 1 exit status
At login
Shell reads from/etc/profile
Then it reads your own.bash_profile
.profile
Is your environment different?How can you see these files, starting with .
Access and permissions
bash$ ls -l
drwxr-x--- 6 judy judy 4096 Dec 14 2012 Cs12000
drwxrwxr-x 12 judy judy 4096 Jul 10 2012 Cs12001
drwxrwxr-x 6 judy judy 4096 Mar 15 12:29 EPS
drwxr-x--- 4 judy judy 4096 Feb 11 1999 IF
-rw-r----- 1 judy judy 4882 Feb 12 2001 Kay.ps
drwxrwxr-x 9 judy judy 4096 Mar 13 22:01 SDM
drwxr-x--- 7 judy judy 4096 Dec 17 11:06 Uidp
Directory/file etcuser,group,other
7/29/2019 2013 Unix Wk2 Handout (1)
3/16
3/11/13
3
Meanings of permissions
Read- File:- Directory
Write- File:- Directory
Execute- File:- Directory
Meanings of permissions
Read- File: program can read it- Directory: can ls it
Write- File: program can alter it- Directory: can add/remove files in it
Execute- File: can run the program- Directory: can read files in it, if name given
Some important cases
Your home directoryls ld .
Protecting files from yourselfchmod u-w precious_file.c
Making files executablechmod u+x doit
General form of chmod symbolic commands
chmod [u g o] [+ - ] [r w x] filenames
What the shell does
gcc -ansi -wall -pedantic *.c
first word is the command name -gccspaces are delimitersexpansions of metacharacters*look for this along $PATHexecute program with that namepass rest of line as parameters
programmer of command (gcc) decides what isallowed
Paths
default search paths - $PATH echo $PATH colon separated list of paths
Initial path
PATH=
/local/usr/bin:/gnu/usr/bin:/usr/bin:/bin:/usr/ccs/bin:/usr/ucb:/usr/openwin/bin:.:
7/29/2019 2013 Unix Wk2 Handout (1)
4/16
3/11/13
4
Paths searched by shell
root
bin
usr home
pub
tmpdev
ascii
etc
passwdken html
Ass1 Erat
Erat.c a.out
doit
/local/usr/bin
/gnu/usr/bin
/usr/bin
/bin
/usr/ccs/bin
/usr/ucb
/usr/openwin/bin
.
bin
Special characters
Selected shell short cuts* [ ] 0-9 a-z
Examplesls *.c
gcc *.c
ls -ld [0-9]*
ls [a-z]*.c
rm i *
Redirection and pipes
stdin
stderr
stdout
a.out
bash$a.out
0
2
1
Redirection
stdin
stderr
stdouta.out
bash$ a.out
bash$ a.out < data1 >res1 2>errors
data1 res1
errors
Redirection - appending to a file
stdin
stderr
stdouta.out
bash$ a.out
bash$ a.out < data1 res1 2>errors
data1 res1
errors
Redirection andpipes
stdin
stdouta.out more
stdin
stdout
a.out |more
pipe symbol
7/29/2019 2013 Unix Wk2 Handout (1)
5/16
3/11/13
5
Initial shell on lab machines
bash$ env
LOGNAME=demo
LD_LIBRARY_PATH=/gnu/usr/lib:/usr/openwin/lib:/local/usr/lib:/usr/local/openssl/lib
WWW_HOME=http://www.ug.cs.usyd.edu.au/
TERM=vt100
HOSTTYPE=sparc
PATH=/local/usr/bin:/gnu/usr/bin:/usr/bin:/bin:/usr/ccs/bin:/usr/ucb:/usr/openwin/bin:.:/usr/bags/stubin
HOME=/usr/cs2/demo
SHELL=/gnu/usr/bin/bash
PS1=bash\$
HZ=100
http_proxy=http://www-cache.cs.usyd.edu.au:8000/
ftp_proxy=http://www-cache.cs.usyd.edu.au:8000/
MANPATH=/local/usr/man:/gnu/usr/man:/usr/share/man:/usr/openwin/man
gopher_proxy=http://www-cache.cs.usyd.edu.au:8000/
OSTYPE=SunOS5
NNTPSERVER=news.cs.usyd.edu.au
OPENWINHOME=/usr/openwin
SHLVL=1
EDITOR=/local/usr/bin/red
TZ=Australia/NSW
WWW_http_GATEWAY=http://www-cache.cs.usyd.edu.au:8000/
_=/usr/bin/env
Shell command to save typing
Create file called doit, containing#!/bin/bash!echo "Compile the masterpiece"!gcc -W -Wall -pedantic -ansi *.c!Make executable and run itbash$ chmod u+x doit!bash$ ./doit!Compile the masterpiece!bash$ !
Your own commands
Create a command that will show all the details of the 5 mostrecently changed files in the current directory
ls ltr | tail 5
Need execute permission Need it on path
Save typing on common command sequences
Create a file with commands to- compile erat.c- run it, taking data from 01_data, saving the results in 01_res- ditto 02_data and 02_res- ditto 03_data and 03_res- then compare actual and expected results in (01_expected,
02_expected, 03_expected)
doit
gcc -W -Wall -pedantic -ansi erat.c
a.out < 01_data > 01_res
a.out < 02_data > 02_res
a.out < 03_data > 03_res
echo comparing first test results ..
diff 01_res 01_expected
echo comparing second test results ..
diff 02_res 02_expected
echo comparing third test results ..
diff 03_res 03_expected
doit
Then make doit executableIf . not in $PATH, need ./doit, else doitWhat about 50 tests?
7/29/2019 2013 Unix Wk2 Handout (1)
6/16
3/11/13
6
doit again
gcc -W -Wall -pedantic -ansi erat.c
echo Test on 01_data - normal > result
a.out < 01_data >> result
printf Test on 02_data - normal\n >> result
a.out < 02_data >> result
echo Test on 03_data - boundary >> result
a.out < 03_data >> result
doit so far
Saves typing Means you will be more likely to test systematically you know there has to be a better way to handle this, especially
with 50..100.. tests
Another tiny task
Afile has a list of people's last names several per line. We want toknow how many people with different names are in the file.
bash$ cat name_list
Wang Ng Smith Lee
Ng Howard Lee
bash$
bash$ fmt -1 name_list > names_split
bash$ sort -u names_split > names_sort
bash$
bash$ fmt -1 name_list > names_split
bash$ sort -u names_split > names_sort
bash$ wc names_sort
the programmer might check as they go.
bash$ fmt -1 name_list > names_split
bash$ cat names_split
Wang
Ng
Smith
Lee
Ng
Howard
Lee
bash$ wc names_split
7 7 32 names_split
bash$ sort -u names_split > names_sort
bash$ wc names_sort
5 5 25 names_sort
bash$ cat names_sort
Howard
Lee
Ng
Smith
Wang
7/29/2019 2013 Unix Wk2 Handout (1)
7/16
3/11/13
7
How do you get to know about fmt and all the othercommands?
Online manual
Practice to automate your knowledge
Summary
Shell- scripts a scripting language- Speed of creating solutions- Interpreted, slower- default search paths - $PATH
- colon separated list of paths
File System- Files- Directories- Paths
xkcd.org comment on kilobyte Shell commands and scripts
The Unix Shell
a shellis unix command interpreter the shell reads commands typed in a command window or from a
file, interprets them and runs the appropriate programs
simple shell commands such as:gcc myprog.c
will run the program (gcc) you specified
more complex commands may involve running several programsand connecting one to the next throughpipes
the shell also as simple control structures that dont involverunning a program but instead control how other programs are run
eg: if and for statement
7/29/2019 2013 Unix Wk2 Handout (1)
8/16
3/11/13
8
The Unix Shell: key concepts
I/O redirection and pipes scripts variables control structures
these shell features are supported by many simple commands thatcan be combined in scripts to provide powerful services
I/O redirection
when programs are run they have three standard files opened:- standard input (0)- standard output (1)- standard error (2)
these are normally all connected to the terminal window can be redirected using > or < or |!gcc myprog.c >output!
Pipes
pipes let you connect the standard output of one program to thestandard input of another program
myprog | hisprog! this has the same effect as
myprog >tempfile!! !hisprog
7/29/2019 2013 Unix Wk2 Handout (1)
9/16
3/11/13
9
Script Arguments
shell script arguments are available in specialvariables $1, $2 etc
for example, the script called compile:#!/bin/sh!
! !gcc $1.c -o $1!you invoke this like:
compile myprog!a list of all script arguments is available in theshell variable $*!
Control Structures
there are many different shell commandinterpreters available
-we use bashMost shells offer the control stuctures:
-if-for-case-while
these are implemented by the shell programitself, not by separate commands
if statement
The if statement is used to test a condition andcontrol flow of execution in a script
if command!!then!! !command2!! !command3!!fi!
if statement
example:if myprog
then
echo myprog ok
else
echo myprog failed
fi
then clause executed ifmyprogexits with result 0
else clause executed ifmyprog exitswith non-zero result
test command
there is a command called test that will testsome condition and exit with true or false (0 or 1)exit status
if test $1 == fred !! !then!! ! !echo first arg is fred!! !fi!test can test many conditions, especially relatingto files - see the manual entry (man test)
calling your own program test is a bad idea!
for statement
the for statement is used to iterate over a list,usually a list of files:
doit script:
for name in $*!! !do!! ! !gcc $name.c -o $name!! !done!this will compile each file name given when thescript is invoked, eg doit myprog hisprog
7/29/2019 2013 Unix Wk2 Handout (1)
10/16
3/11/13
10
while
the while statement executes a block of a shellscript while some condition is true, eg:
while test $result != 10!!do!! !!!done!
case
case is a selection statement (like switch in C)case $type in!
! !1) !myprog;;!! !2) !hisprog;;!! !pattern) otherprog;;!! !esac!
Useful Commands for handling text files
see the manual entries for details
sort!sorts lines of text in a file, veryflexible choice of key field to sorton, can remove duplicates,sends sorted lines to output
cut! cuts fields out lines of text from afile and sends the result to output
tr! transliterates or removescharacters from a fileeg remove \r from a file
comm!compares files and prints linesthat appear in only one or boththe files
Example
#!/bin/sh
tr A-Z a-z < $1 |
tr -cs a-z \n |
sort -u |comm -13 /usr/dict/words
Example
#!/bin/sh
tr A-Z a-z < $1 |
tr -cs a-z \n |
sort -u |
comm -13 /usr/dict/words
Read from first argument to script
Example
#!/bin/sh
tr A-Z a-z < $1 |
tr -cs a-z \n |
sort -u |
comm -13 /usr/dict/words
convert upper case to lower case
7/29/2019 2013 Unix Wk2 Handout (1)
11/16
3/11/13
11
Example
#!/bin/sh
tr A-Z a-z < $1 |
tr -cs a-z \n |
sort -u |
comm -13 /usr/dict/words
convert upper case to lower case
convert non-alphabetic to a newline
Example
#!/bin/sh
tr A-Z a-z < $1 |
tr -cs a-z \n |
sort -u |
comm -13 /usr/dict/words
convert upper case to lower case
convert non-alphabetic to a newline
sort and remove duplicates
Example
#!/bin/sh
tr A-Z a-z < $1 |
tr -cs a-z \n |
sort -u |comm -13 /usr/dict/words
convert upper case to lower case
convert non-alphabetic to a newline
sort and remove duplicates
print lines that appear in input but notin /usr/dict/words
Example
#!/bin/sh
tr A-Z a-z < $1 |
tr -cs a-z \n |
sort -u |comm -13 /usr/dict/words
What does it do?!
Shell scripts instead of programs
The combination of the shell scripting language (variables, pipes,control structures etc) and many general purpose text processingcommands can often be a more effective way of solving a problemthan writing a program in a conventional language (Java, C etc)
More useful shell features
changing your promptthe prompt is printed by the shell to indicate that it is waiting for you to
type a command
you can change the prompt to contain useful information such as currentmachine or directory. This is done using the PS1 shell variable
bash2.5$ PS1=\h \W
congo1 bob%
7/29/2019 2013 Unix Wk2 Handout (1)
12/16
3/11/13
12
command substitutionan argument enclosed in backquotes indicates that a command is to run
and the output used as the actual argument(s)
eg myprog `cat argfile`
will run the cat command, take the output (ie the contents of argfile) anduse it as the arguments for myprog
subshells
sometimes commands are run in another copy of the shell a subshell the environment is copied from the parent shell, the subshell may change
it but it reverts to the original when the subshell exits
tar cf MyDir | (cd Somewhere; tar xf -)
subshells
sometimes commands are run in another copy of the shell a subshell the environment is copied from the parent shell, the subshell may change
it but it reverts to the original when the subshell exits
tar cf MyDir | (cd Somewhere; tar xf -)
done in a subshell
also useful to collect output from several commands:(echo My Data; cat file1) >output!
this also works with for and whilefor X in $*!do!!echo $X!!cat $X!done > newfile!
take the collected output from the for loop andredirect to newfile
arithmeticthe expr command in standard shell will evaluate its arguments as an
expression
myprog `expr $count + 1`!this will run expr to calculate the value of the shell variable count, add
1 to it and send the answer to the output where it will be used as theargument to myprog
in the Bash shell you can use the command let for assignmentstatements of variables
! !let count = count +1!
the test command will evaluate boolean expressions. It will alsolet you test file characteristics
if test d name!! !then!! ! !echo name is a directory!! !fi!
-d name is a directory-s file exists and is non-zero size-r file exists and is readable-w file exists and is writableetc
7/29/2019 2013 Unix Wk2 Handout (1)
13/16
3/11/13
13
reading text from within a shell scriptread x!
will read a line from standard input and store it in x
read name!if [ $name == Fred ]!then!!echo name was Fred!fi!
input from within a script
also called a here document
This word can be anything, it isused to terminate the document
grep $1
7/29/2019 2013 Unix Wk2 Handout (1)
14/16
3/11/13
14
Comparing files
diff will show the differences between files sdiff is a side by side diff that shows both texts next to each other with
the differences indicated.
top
the top command will show running processes dynamically the displayis updated every second or so to show the details of processes
very flexible options, such as sorting on various process attributes
Strange file names
if you inadvertently (or on purpose) end up with a file with a strange nameit can be hard to work with it
eg a file called -x will often be confused with an argument -x (x can be almostany char)
so nano R will not try and open a file called
-R. To make this work use: nano ./-R
Common problems
forgetting to set execute permission forgetting that the current directory is not in $PATH using test as the name of a prog or script
End of Section
Example: generating web pages
web pages can be static files that contain a description of the pagewritten in the HTML markup language
web pages are often generatedby a program when you requestthe page
web page generation is carried out by programs that take currentinformation from many sources and write out the HTML for thepage
7/29/2019 2013 Unix Wk2 Handout (1)
15/16
3/11/13
15
How the Web works(the 2 minute version)
a user types a URL into a web browserprogram
the browser parses the URL and determinesthe machine to contact for the page
the browser makes a network connection to aweb serverrunning on the remote machine andrequests the web page specified by the URL
the web server copies the page from a file tothe browser
the browser displays the page
Web Pages
web pages are usually written in HTML (HyperText MarkupLanguage) eg:
Heading
This is bold text.
but can be images, sounds, plain text etc
Creating your own web pages
web pages are stored in files in (SIT servers):$HOME/lib/html
a file ending in .html is written in HTML a file ending in .txt will be displayed as plain text to allow the web server to access your files they need to be world
readable along with the directories
Creating your own web pages
setting permissions:chmod 711 $HOME
chmod 711 $HOME/lib
chmod 711 $HOME/lib/html
chmod 744 $HOME/lib/html/myfile.txt
CGI Scripts
generated web pages are often created by cgi scripts that run onthe web server machine in response to a browser asking for a URL
cgi scripts are executable programs or scripts in files ending in.cgi
could be written in C, java, perl, pythonorcould be written in shell!
Making a web pagewith a CGI script
Web Browser Web Server
url
web page
CGIscript
7/29/2019 2013 Unix Wk2 Handout (1)
16/16
3/11/13
16
CGI shell script
#!/bin/shecho "Content-type: text/plain"echoecho Testing my CGI script
tells the browser the type ofweb page
the blank line is essential
content of the page
CGI script
#!/bin/shecho "Content-type: text/plain"echocat example.txt
page content can come from anywhere
Count the visitors
#!/bin/shecho "Content-type: text/plain"echocat example.txtecho "Hello, today is " `date`
count=`cat counter`count=`expr $count + 1`echo $count >counterecho "There have been " $count " visits to this page"
use the outputof thecommandat that point
Whos logged on?
echo "There are "who | cut -c1-10|sort -u|wc -lecho "people logged on"echo "Friends logged in:"for name in `cat friends`do
who | grep $namedone | cut -c1-10 |sort -u
Output of the group ofcommands within the forloop is sent through the pipe
Entire web sites are generated from data held in databases Languages such as Perl, PHP or Python are commonly used COMP2129 class2go web site is generated by python scripts Shell can be used for simple web page generation
Summary