1
Advanced UNIX Tutorial
Shelling, Otheus
For University of Innsbruck, March 2007
• NOTE: NOT THE FINAL VERSION ! !
Topics Covered• Why Unix?• Brief History• Architecture• Inter-process Communication
• Filesystem• Regular Expressions• Program Execution
• PATH to enlightenment• Dynamic Linking
• The Shells• BASH Programming• Shell Startup• Short-cuts
• Cool Commands
root:/ $
Why UNIX? And Why Not?
• High Performance• Fine-grain control over hardware
• Fine-grain control over OS
• WYSIWYG .. Not WYSIAYG
• Windows has “caught up” in performance
• Windows is easier to administrate
• Windows systems don’t require fine-grain control
2
Flavors, Versions, DistributionsFlavors
UNIX comes in various Flavors• System V, BSD, Linux
DistributionsA particular Flavor will be packaged and marketed through various Distributions
• FreeBSD, RedHat, GentooVersions
A particular Distribution will be released with a specific Version number
• RedHat EL4 US, Solaris 10 (2.10) Other version designations
The kernel and various software tools and applications each have their own version numbering
• X11R6, Linux 2.6, Perl 5.6
UNIX Architecture – System LevelDevices -
managed by drivers
Drivers -interact through an API with the Kernel
Kernel -manages core resources• Processes• Memory (physical and virtual)
• Interprocesscommunication (IPC)
presents to users: • system calls• device files
UNIX Architecture – User LevelUser:
interacts with an application through a device
Application:commands
ls, qsub,run-time environment
bash, perl, python, Java, PHP, mozilla
subsystemX Windows, SGE
Shared libraries:glibc, gtk
Each application is in its own process and has its own private data. The code can be shared across instances.Each process runs under the permissions of the user that started it – “effective user id”The application and its libraries make system calls and operates on files.
3
Inter-Process Communication
Inter-Process communication (IPC)• Semaphores• Shared Memory• Filesystem• Pipes• Signals
*otheus* ~/eg$ ls -1
example
example2
exists
myexample
ozzie
*otheus* ~/eg$ ls -1 >files
*otheus* ~/eg$ sort -k 1.2,1.4 files
files
example
example2
exists
myexample
ozzie
*otheus* ~/eg$
Inter-Process communication (IPC)• Semaphores• Shared Memory• Filesystem• Pipes
– Unnamed– Named / FIFOs– Network Sockets
• RPC• Signals
*otheus* ~/eg$ ls -1 exampleexample2
existsfilesmyexample
ozzie*otheus* ~/eg$ ls -1 | sort -k 1.2,1.4 filesexampleexample2
existsmyexampleozzie
*otheus* ~/eg$
4
Inter-Process communication (IPC)• Semaphores• Shared Memory• Filesystem• Pipes
– Unnamed– Named / FIFOs– Network Sockets
• RPC• Signals
*otheus* ~/eg$ mkfifo newfifo *otheus* ~/eg$ *otheus* ~/eg$ ls -l newfifoprw-r--r-- 1 otheus dps 0 Mar 28 19:52 newfifo
*otheus* ~/eg$ { while true; > do echo $RANDOM >>/home/otheus/eg/newfifo ; > sleep 5; done } &[1] 15533
*otheus* ~/eg$ ls -l newfifo prw-r--r-- 1 otheus dps 0 Mar 28 19:52 newfifo
*otheus* ~/eg$ date ; cat newfifo ; dateWed Mar 28 19:52:49 CEST 200732102Wed Mar 28 19:52:49 CEST 2007
*otheus* ~/eg$ date ; cat newfifo ; dateWed Mar 28 19:52:51 CEST 200726331Wed Mar 28 19:52:54 CEST 2007
*otheus* ~/eg$
Inter-Process communication (IPC)• Semaphores• Shared Memory• Filesystem• Pipes
– Unnamed– Named / FIFOs– Network Sockets
2-way pipes over a network
• RPC• Signals
*otheus* ~/eg$ sudo netstat -anp |> grep -E "(181):(3306|80|22) " | > head -3
tcp 0 0 138.232.66.181:22 0.0.0.0:* LISTEN 4176/sshd
tcp 0 48 138.232.66.181:22 192.168.64.52:2131 ESTABLISHED 9343/5
tcp 0 0 138.232.66.181:3306 138.232.66.164:49325 TIME_WAIT -
Inter-Process communication (IPC)• Semaphores• Shared Memory• Filesystem• Pipes • RPC
– RPC process– Portmapperprocess
– YP/NIS & NFS
• Signals
*otheus* ~/eg$ ps -e -o uid,pid,cmd |> grep rpc
29 3338 rpc.statd0 3599 rpc.idmapd0 4093 rpc.yppasswdd -e chsh -e chfn0 4274 rpc.rquotad0 4312 [rpciod]0 4323 rpc.mountd
201 17112 grep rpc
*otheus* ~/eg$ rpcinfo -p | uniq -f 4
program vers proto port100000 2 tcp 111 portmapper100024 1 udp 32768 status100007 2 udp 856 ypbind100004 2 udp 845 ypserv100009 1 udp 878 yppasswdd100011 1 udp 640 rquotad100003 2 udp 2049 nfs100021 1 udp 32774 nlockmgr100005 1 udp 686 mountd
5
Inter-Process communication (IPC)• Semaphores• Shared Memory
• Filesystem• Pipes • RPC• Signals
– Only within a process “family”
– Processes “0”, “-1”
– SIGHUP– SIGTERM– SIGKILL– SIGSEGV
*otheus* ~/eg$ kill -l | head -81) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL5) SIGTRAP 6) SIGABRT 7) SIGBUS 8) SIGFPE9) SIGKILL 10) SIGUSR1 11) SIGSEGV 12) SIGUSR213) SIGPIPE 14) SIGALRM 15) SIGTERM 17) SIGCHLD18) SIGCONT 19) SIGSTOP 20) SIGTSTP 21) SIGTTIN22) SIGTTOU 23) SIGURG 24) SIGXCPU 25) SIGXFSZ26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 29) SIGIO30) SIGPWR 31) SIGSYS 34) SIGRTMIN 35) SIGRTMI
*otheus* ~/eg$ echo $$ 2432*otheus* ~/eg$ /bin/ash
*\u* \w$ echo $$18201*\u* \w$ kill -HUP 18201Hangup
*otheus* ~/eg$ echo $$ 2432
*otheus* ~/eg$ trapped &[1] 18664*otheus* ~/eg$ kill -1 18664 ; sleep 1Caught signal!*otheus* ~/eg$ kill 18664 ; sleep 1 [1]+ Terminated trapped*otheus* ~/eg$
UNIX Filesystem
UNIX Filesystem – Tree Structure• Resembles a tree
– File-types– Mounting drives– Breaking structure
• Promiscuous• Permissions
6
UNIX Filesystem – Tree Structure• Resembles a tree
– File-types– Mounting drives– Breaking structure
• Promiscuous• Permissions
UNIX Filesystem – Tree Structure• Resembles a tree
– File-types– Mounting drives– Breaking structure• Soft-links• Loopback mounts• “Bind” mounts
• Promiscuous• Permissions
UNIX Filesystem – Tree Structure• Resembles a tree
– File-types– Mounting drives– Breaking structure• Soft-links• Loopback mounts• “Bind” mounts
• Promiscuous• Permissions
7
UNIX Filesystem – Tree Structure• Resembles a tree
– File-types– Mounting drives– Breaking structure• Soft-links• Loopback mounts• “Bind” mounts
• Promiscuous• Permissions
UNIX Filesystem – Very Promiscuous• Resembles a tree• Promiscuous:
Nearly everything in UNIX can be accessed through the filesystemDevicesProcesses (Linux & Solaris) via /proc filesystemKernel stats and configuration (Linux: /proc)
Exception: Network interfaces
• Permissions
UNIX Filesystem – Very Promiscuous• Resembles a tree• Promiscuous
– Devices– Processes– Kernel stats and configuration
• Permissions
MAKEDEV loop5 ram14 skip tty2XOR loop6 ram15 stderr tty3arpd loop7 ram2 stdin tty4cdrom lp0 ram3 stdout tty5cdrom1 lp1 ram4 systty tty6console lp2 ram5 tap0 tty7core lp3 ram6 tap1 tty8cpu mapper ram7 tap10 tty9cpu0 mcelog ram8 tap11 urandomcpu1 md0 ram9 tap12 usersockcpu2 mem ramdisk tap13 vcscpu3 mice random tap14 vcs1dnrtmsg mosal rawctl tap15 vcs2dvd net root tap2 vcs3fd nflog route tap3 vcs4floppy null route6 tap4 vcs5full parport0 rtc tap5 vcs6fwmonitor parport1 scd0 tap6 vcsahda parport2 sda tap7 vcsa1hw_random parport3 sdb tap8 vcsa2initctl port sdc tap9 vcsa3input ppp sdd tcpdiag vcsa4ip6_fw ptmx sde ts_ua0 vcsa5js0 pts sde1 ts_ua1 vcsa6kmsg ram sde2 ts_ua2 vipkllog ram0 sde3 ts_ua3 xfrmloop0 ram1 sdf ts_ua4 zeroloop1 ram10 sdf1 ts_ua5loop2 ram11 sdf2 ts_ua6loop3 ram12 sdf3 tty0loop4 ram13 shm tty1
8
UNIX Filesystem – Very Promiscuous• Resembles a tree• Promiscuous
– Devices– Processes– Kernel stats and configuration
• Permissions
*otheus* /proc$ ls -Cd [1-2]*
1 14642 17888 21007 21942 258
10 14644 17889 21008 21943 2619
10377 14657 17890 21009 2251 2620
10395 15 17892 21010 2269 2647
11 155 17894 21017 2270 2648
11639 17473 19570 21019 2312 2649
12 17476 2 21020 2313 2650
13 17478 20998 21021 240 2658
13956 17511 20999 21022 241 27265
13964 17871 21000 2121 2431 286
14 17884 21002 21646 2432
1433 17887 21003 21941 257
*otheus* /proc$ cat 1/cmdline; echo
init [4]
*otheus* /proc$ ls -F 1
attr/ environ maps stat wchan
auxv exe@ mem statm
cmdline fd/ mounts status
cwd@ loginuid root@ task/
*otheus* /proc$
UNIX Filesystem – Very Promiscuous• Resembles a tree• Promiscuous
– Devices– Processes – Kernel stats and configuration
• Permissions
*otheus* /proc$ ls -CdF [a-z]*acpi/ iomem net/buddyinfo ioports partitionsbus/ irq/ pcicmdline kallsyms scsi/cpuinfo kcore self@crypto key-users slabinfodevices keys statdiskstats kmsg swapsdma loadavg sys/driver/ locks sysrq-triggerexecdomains mdstat sysvipc/fb meminfo topspin@filesystems misc tty/fs/ modules uptimeide/ mounts@ versioninfiniband/ mpt/ vmstatinterrupts mtrr*otheus* /proc$ cat uptime704398.43 593929.58*otheus* /proc$ uptime20:56:58 up 8 days, 3:39, 7 users,
load average: 1.72, 1.75, 1.67*otheus* /proc$
UNIX Filesystem – Security• Resembles a tree• Promiscuous• Permissions
– 3 Classes:• User• Group• Other (World)
– 4 bits:• Read• Write
• Execute• Special
9
UNIX Filesystem – Security• Resembles a tree• Promiscuous• Permissions
– 3 Classes:• User• Group• Other (World)
– 4 bits:• Read• Write• Execute• Special
UNIX Filesystem – Security• Resembles a tree• Promiscuous• Permissions
– 4 bits:• Read• Write• Execute• Special
– Directories– Set-uid
– Set-gid
UNIX Filesystem – Security• Resembles a tree• Promiscuous• Permissions
– 4 bits:• Read• Write• Execute• Special
– Directories
– Set-uid– Set-gid
13
The Shells
The Shells• Run applications and “commands”– Interactively– “Batch mode”– In parallel
• Interact with Files
• Manage environment used by applications
The Shells• Run applications and “commands”
• Interact with Files– “Send” files to commands.
– “Send” command’s output to files.
• Manage environment used by applications
14
The Shells• Run applications and “commands”
• Interact with Files
• Manage environment used by applications– Environment variables
– File creation mask
The Shells• Flavors
– Bourne (sh)• Korn (ksh)• Bourne-Again (bash)
• Almquist (ash)• “Z” shell (zsh)
– “C” Shell (csh)• tcsh
– “Exotic”• fish• zoidberg
– GUI• Konqueror
– Text-GUI• Midnight Commander (mc)
The ShellsMan’s Best Friend• help (BASH only)• man
more or lessman –kman (Solaris)man (GNU/BSD)
• info
15
The ShellsMan’s Best Friend• help (BASH only)• man• info
Hyperlink viewer– CTRL-H– --vi-keys
File: info.info, Node: Top, Next: Getting St\arted, Up: (dir)
Info: An Introduction*********************
The GNU Project distributes most of its on-lin\e manuals in the "Infoformat", which you read using an "Info reader"\. You are probably usingan Info reader to read this now.
If you are new to the Info reader and want \to learn how to use it,type the command `h' now. It brings you to a \programmed instruction--zz-Info: (info.info.gz)Top, 35 lines --Top---Basic Commands in Info Windows******************************
C-x 0 Quit this help.C-x C-c Quit Info altogether.h Invoke the Info tutorial.
Selecting other nodes:----------------------n Move to the "next" node of this no\de.p Move to the "previous" node of thi\s node.u Move "up" from this node.m Pick menu item specified by name.-----Info: *Info Help*, 763 lines --Top--------No cross references in this node.
The Shells – Using Aliases• What is it?• Creating an Alias• Automatic aliasing (bash)
hash
type / which
unalias
alias
The Shells – Using Aliases• What is it?• Creating an Alias• Automatic aliasing (bash)
hash
type / which
unalias
alias
16
Advanced Shell Programming
Advanced Shell Programming with BASH• Working Directory• Variable Creation & Expansion
• Job Control• File Manipulation• File Specification
• Negative Logic• Execution Control• Functions
exec
fg / bg
eval
export
set / echo
cd
pwd
Advanced Shell Programming with BASH• Working Directory• Variable Creation & Expansion
• Job Control• File Manipulation• File Specification
• Negative Logic• Execution Control• Functions
exec
fg / bg
eval
export
set / echo
cd
pwd
17
Advanced Shell Programming with BASH• Working Directory• Variable Creation & Expansion
• Job Control• File Manipulation• File Specification
• Negative Logic• Execution Control• Functions
ls
kill
jobs
exec
fg / bg
eval
export
Advanced Shell Programming with BASH• Working Directory• Variable Creation & Expansion
• Job Control• File Redirection and Manipulation– Listing Files– Viewing Files
• File Specification
• Negative Logic• Execution Control• Functions
rm /cp / mv
grep
head / tail
more / less
cat
ls
exec
Advanced Shell Programming with BASH• Working Directory• Variable Creation & Expansion
• Job Control• File Manipulation
– standard streams– Copying, moving, renaming, new dir
• File Specification
• Negative Logic• Execution Control• Functions
newgrp
id / groups
chmod / chown
mkdir / rmdir
rm /cp /mv
grep
head / tail
18
Advanced Shell Programming with BASH• Working Directory• Variable Creation & Expansion
• Job Control• File Manipulation
– Changing owner– Changing permissions
• File Specification
• Negative Logic• Execution Control• Functions
umask
newgrp
id / groups
chmod / chown
mkdir / rmdir
Advanced Shell Programming with BASH• Working Directory• Variable Creation & Expansion
• Job Control• File Redirection• File Specification– File globs
• Negative Logic• Execution Control• Functions
exec
fg / bg
xargs
find
umask
newgrp
id / groups
Advanced Shell Programming with BASH• Working Directory• Variable Creation & Expansion
• Job Control• File Redirection• File Specification– find and xargs
• Negative Logic• Execution Control• Functions
exec
fg / bg
xargs
find
umask
newgrp
id / groups
19
Advanced Shell Programming with BASH• Working Directory• Variable Creation & Expansion
• Job Control• File Redirection• File Specification
• Negative Logic• Execution Control• Functions
for VAR in .. do .. done
case W in .. esac
if .. fi
true / false
test
eval
Advanced Shell Programming with BASH• Working Directory• Variable Creation & Expansion
• Job Control• File Redirection• File Specification
• Negative Logic• Execution Control
– if/elif/else/fi
• Functions
while.. do.. done
for VAR in .. do .. done
case W in .. esac
if .. fi
true / false
Advanced Shell Programming with BASH• Working Directory• Variable Creation & Expansion
• Job Control• File Redirection• File Specification
• Negative Logic• Execution Control
– case WORD in ... esac
• Functions
while.. do.. done
for VAR in .. do .. done
case W in .. esac
if .. fi
true / false
20
Advanced Shell Programming with BASH• Working Directory• Variable Creation & Expansion
• Job Control• File Redirection• File Specification
• Negative Logic• Execution Control
– case WORD in ... esac
• Functions
while.. do.. done
for VAR in .. do .. done
case W in .. esac
if .. fi
true / false
Advanced Shell Programming with BASH• Working Directory• Variable Creation & Expansion
• Job Control• File Redirection• File Specification
• Negative Logic• Execution Control
– for VAR in ... done
• Functions
while.. do.. done
for VAR in .. do .. done
case W in .. esac
if .. fi
true / false
Advanced Shell Programming with BASH• Working Directory• Variable Creation & Expansion
• Job Control• File Redirection• File Specification
• Negative Logic• Execution Control
– while ... do ... done
• Functions
while.. do.. done
for VAR in .. do .. done
case W in .. esac
if .. fi
true / false
21
Advanced Shell Programming with BASH• Working Directory• Variable Creation & Expansion
• Job Control• File Redirection• File Specification
• Negative Logic• Execution Control
– break, next, exit– ||, &&– type, which, whence
• Functions
|| and &&
exit
next
break
while.. do.. done
Advanced Shell Programming with BASH• Working Directory• Variable Creation & Expansion
• Job Control• File Redirection• File Specification
• Negative Logic• Execution Control
– break, next, exit– ||, &&– type, which, whence
• Functions
whence
which
type
Advanced Shell Programming with BASH• Working Directory• Variable Creation & Expansion
• Job Control• File Redirection• File Specification
• Negative Logic• Execution Control• Functions
– Declaring, returning
declare
command
return
22
Advanced Shell Programming with BASH• Working Directory• Variable Creation & Expansion
• Job Control• File Redirection• File Specification
• Negative Logic• Execution Control• Functions
– Using– Overriding builtins & commands
declare
builtin
command
Advanced Shell Programming with BASH• Working Directory• Variable Creation & Expansion
• Job Control• File Redirection• File Specification
• Negative Logic• Execution Control• Functions
– Using– Overriding builtins & commands
type
builtin
command
Shell Startup
Top Related