The POSIX shell as a programming language
Transcript of The POSIX shell as a programming language
![Page 1: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/1.jpg)
The POSIX shell as a programming language
Michael Greenberg (Pomona College)
OBT 2017 — Paris, France
![Page 2: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/2.jpg)
i love shell
![Page 3: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/3.jpg)
shell is everywhere• vital for managing systems
• maintenance
• deployment
• universal tool for sysadmins
• extremely powerful
![Page 4: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/4.jpg)
POSIX shell
• Open Group Spec/IEEE Standard 1003.1
• Intimately connected to POSIX
• Many implementations!
![Page 5: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/5.jpg)
# figure out the absolute path to the script being run a bit # non-obvious, the ${0%/*} pulls the path out of $0, cd's into the # specified directory, then uses $PWD to figure out where that # directory lives - and all this in a subshell, so we don't affect # $PWD STEAMROOT="$(cd "${0%/*}" && echo $PWD)"
# Scary! rm -rf "$STEAMROOT/"*
https://github.com/ValveSoftware/steam-for-linux/issues/3671
curl -k https://<master hostname>:8140/packages/current/install.bash | bash
https://puppetlabs.com/blog/simplified-agent-installation-puppet-enterprise-3.2
![Page 6: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/6.jpg)
![Page 7: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/7.jpg)
i love reasoning
![Page 8: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/8.jpg)
hasn’t shell been ‘fixed’ already?
• scsh and shill?
• not POSIX shells!
• tclsh
• no formal attention, to my knowledge
• and a bit out of date at this point
![Page 9: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/9.jpg)
ShellCheck
• Linter for shell
• Catches bug in Steam script…
• …but not a trivial refactoring
![Page 10: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/10.jpg)
NoFAQ
• Machine learning to correct console commands
• No semantics insights
• No guarantees
• More about commands than about the shell
D’Antoni and Vaughn 2016
![Page 11: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/11.jpg)
ABash
• Static analysis for number of arguments
• Semantic understanding
• Great start!
Mazurak and Zdancewic 2007
![Page 12: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/12.jpg)
shell is unique• unique evaluation model
• expansion, not evaluation, of args by default
• deploy and manage concurrency
• uniquely interactive programming model
• try before you buy
![Page 13: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/13.jpg)
conventional evaluation
e1 eval v1 e2 eval v2 δ(⊗,v1,v2) = v3
e1 ⊗ e2 eval v3
![Page 14: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/14.jpg)
expansion by defaulte1 expand s1 e2 expand s2
unparse(δ(⊗,parse(s1),parse(s2))) = s3
e1 ⊗ e2 eval v3
e eval v unparse(v) = s`e` expand s
![Page 15: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/15.jpg)
c ::= v=a … a … | c r
| c1|c2|c3|…|cn | c & | ( c )
| c1 && c2 | c1 || c2
| ! c | c1 ; c2 | if c1 c2 c3
| switch a … { case a…) c } …
| while c1 c2 | for x in a … c
| defun v c
![Page 16: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/16.jpg)
semanticstokenize
parse
expand
redirect
execute
wait
read
fixed behavior at compile time}
![Page 17: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/17.jpg)
expansiontokenize
parse
expand
redirect
execute
wait
read
echo ~ /Users/mgree
echo ${PWD} /Users/mgree/talks/obt
basename `pwd` obt
echo $((1+1)) 2
IFS=“” cat `echo some file` [shows contents of ‘some file’]
echo * abstract.txt posix.key some file
echo you can “” me you can me
![Page 18: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/18.jpg)
backquotingtokenize
parse
expand
redirect
execute
wait
read
echo ~ /Users/mgree
echo ${PWD} /Users/mgree/talks/obt
basename `pwd` obt
echo $((1+1)) 2
IFS=“” cat `echo some file` [shows contents of ‘some file’]
echo * abstract.txt posix.key some file
echo you can “” me you can me
![Page 19: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/19.jpg)
backquotingtokenize
parse
expand
redirect
execute
wait
read
$ basename `pwd`
$ basename /Users/mgree/talks/obt
obt
![Page 20: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/20.jpg)
…
essentials of the semanticstokenize
parse
expand
redirect
execute
wait
read
parse
expand
executeexecute
expand
![Page 21: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/21.jpg)
essentials of the semantics
PATH /usr/bin:…
$ x=${y:=1} ; echo $((x+=`echo 2`))$ x=${1} ; echo $((x+=`echo 2`))
PATH /usr/bin:…
x 1
$ echo $((x+=`echo 2`))$ echo $((x+=2))$ echo 3
PATH /usr/bin:…
x 3
3 env
legend: expansion evaluation
![Page 22: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/22.jpg)
what do I want to do?
![Page 23: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/23.jpg)
SibylFSRidge et al.
SOSP’15
ptrace
semantics for shell
analyses RehearsalShambaugh et al. PLDI’16
tools
ForestFisher et al.
ICFP’11
program logic
Gardner Ntzik OOPSLA 2015
![Page 24: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/24.jpg)
support the programming model
• have script echo commands until script is just right
• maybe running some commands
• set -x prints commands run…but it still runs the commands!
• can we do better?
![Page 25: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/25.jpg)
other tools• compile to other languages as a form of “gradual
scripting”
• “cruft” inserter
• hardens a shell script against, e.g., signals
• uses weakest preconditions to guarantee good exit status of all commands
![Page 26: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/26.jpg)
types!• commands take a regular expression over args
as input, produces certain patterns of system calls
• summarize sets of commands/system calls/outputs
• e.g., this script will delete all files in ~/.foo/ except for ~/.foo/cache
• analyze curl-based installers!
![Page 27: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/27.jpg)
design$ lsfilenamespacesfilename with spaces $ x=“filename with spaces” $ rm $xrm: with: No such file or directory $ lsfilename with spaces $ rm “$x”$ ls$
![Page 28: The POSIX shell as a programming language](https://reader035.fdocuments.us/reader035/viewer/2022062515/62b2594b9fab0b4aaf3cff2f/html5/thumbnails/28.jpg)
what else?
• theoretical ideas/angles i’m missing?
• suppose we’ve got a great model… what else should we do with it?thanks to: Arjun Guha for early chats Calvin Aylward and Austin Blatt