Rubish: A Quixotic Shell

Howard Yeh

Quix.ot.ic

• \kwik-’sä-tik\

• foolishly impractical especially in the pursuit of ideals ; especially : marked by rash lofty romantic ideas or extravagantly chivalrous action.

Why Do You Care?

• Closes the gap between script and shell

• Object-Oriented Meta-shell (frobbable)

• Extensible

– Namespace

– Local extensions (with singleton objects)

– Abstraction as codified memory

• Ruby you know and love

Symbiosis With Ruby

• Meta access

• Data type

• Plausibly concise syntax

• Object#instance_eval(&block)

• Rubish has no metasyntax

• Rubish uses no monkey patches

– Not strictly true, but…

Introduction to Rubish

abandon bash all ye who enters…

Overview

• Executable

– Command, Pipe

– Sed, Awk

– Batch

• Job Control

– Concurrency; Exception Mechanism

• Context

– Dynamically scoped IO; Workspace

Command

Rubish Bash

foo foo

foo :abc foo -abc

foo “a b c” foo a b c

foo.q “a b c” foo “a b c”

foo “a”, “b”, “c”

foo ["a","b","c"]

foo [["a","b"],"c","d"]

Handling weird filenamesrsh> touch("a b","c d").qrsh> wc(ls.map).q

Pipe

• Build pipe with factory

rsh> p { cmd1; cmd2; cmd3 }

• Build pipe from array of commands

rsh> @cmds = [cmd1,cmd2,cmd3]

rsh> p(@cmds)

IO Redirection

• Methods defined on the Executable class

– Executable#{i,o,err}

– Common API to all subclasses

– So far only supports stdin, stdout, stderr

• With File

• With Ruby IO object

• With a block that receives a pipe

– Good for building further abstractions

IO to File

rsh> cat.o("output")

a

b

c

^D

rsh> wc("output").first.to_i

3

IO with Block

rsh> @c = cat.i {|pipe| pipe.puts 1,2,3 }

rsh> @c

1

2

3

# ask the command to write to a pipe instead

rsh> @c.o { |pipe| ... }

IO Abstractions

• Enumerating methods

– Common API to all Executable subclasses

– Executable#{each,map}

– Executable#{head(n=1),tail(n=1),first,last}

• Implemented with the Rubish IO architecture.

– OOP for the win!

Executable#each!

def each!self.o do |pipe|pipe.each_line do |line|

line.chomp!yield(line)

endendjob = self.exec!return job

end

Processing with Ruby

# array of listed file names

ls.map

# last file as string

ls.last # == ls.tail(1).first

# extension name of last 10 files

ls.tail(10) {|f| File.extname(f) }

# ditto with pipe

p { …}.map {|line| … }

Awk & Sed

Sed “One-Liner”

• Courtesy J. Zawinsky (Unix Hater Handbook)# find *.el files that didn't have corresponding *.elc files

# only two processes per file.

sh> find . -name ’*.el’ -print \

| sed ’s/^/FOO=/’|\

sed ’s/$/; if * ! -f \ ${FOO}c ]; then \

echo \ $FOO ; fi/’ | sh

• Rubish Sedrsh> find(". -name '*.el'").sed { p if !File.exist?(line + "c") }

rsh> find(". -name '*.el'").sed { p if !File.exist?(line + "c") }.map {|f| … }

Awk & Sed

• Unix Powertools are for Powerfools

• Hard to predict complexity.

• Hard to get right.

• Hard to remember.

• Hard to extend.

• Hard to generalize.

– “weird chars” problem

Rubish Sed & Awk

• Doesn’t aim for full generality

– Since we are embedded in Ruby anyway

• Captures common usage patterns

– Common-Lisp loopesque helpers

• Be explicit

– Sed does not print by default

• Fits well

– Both are subclasses of Executable

grep with context (Sed)

# grep -A1 -B1

sed -n -e '/regexp/{=;x;1!p;g;$!N;p;D;}' -e h

grep with context (Rubish)

# rubish sedsed {if line =~ /regexp/p "===="# special case if the first line matchesp prev if respond_to?(:prev)pp peek(1)

endhold(:prev,1,line) # dynamically creates a method “prev”

}

Streamer

• Awk and Sed are subclasses of Streamer

• Streamer#{peek(n=1),skip(n=1)}

– No more sedistic register shufflings

• Streamer#{max,min,count,collect,…}

– Common usage patterns

• Streamer#{quit,stop}

• Streamer#{each,map,head,tail,…}

– Inherited methods from Executable

grep with context (Rubish)

rsh> cat.i {|p| p.puts((1..11).to_a)}.sed { ... }====12====91011====1011

grep with context (Rubish)

• Easy to generalize# simulates ‘grep regexp -Aa -Bb’sed {if line =~ regexp

p "===="# special case if the first line matchesp prev if respond_to?(:prev)pp peek(a)

endhold(:prev,b,line)

}

Aggregator

• Streamer#{max,min,count,collect,…}

– Capture common usage patterns

– Inspired by Common Lisp’s Loop Facility

– Method signature: helper(name,value,key)

• Aggregation partitioned by key

– nil is the special global key under which everything is aggregated.

Aggregator

# longest and shortest filename lengths

rsh> ls.awk {

max(:mx,line)

min(:mn,line)

}.end {[mx,mn]}

[25, 3]

Aggregator

rsh> ls.awk { f=a[0]; collect(:fn,f,File.extname(f))}.end { fn}{""=> ["a",...,"util"],

".gz"=>["ruby-termios-0.9.5.tar.gz"],".yml"=>["VERSION.yml"],".sh"=>["test.sh"],".output"=>["awk.output"],".5"=>["ruby-termios-0.9.5"],".textile"=>["README.textile"],".rb"=>["address.rb", "foo.rb", "my.rb"],".bar"=>["foo.bar"]nil=>["a", ...,"VERSION.yml"]}

# fn(“.rb”) contains the array of *.rb

Addressed Patterns

.sed(3) { … } # triggered for line 3

.sed(3,9) { … } # lines 3 to 9 inclusive

.sed(3,:eof) { … } # lines 3 to end of file

.sed(/a/) { … } # triggered for matching lines

.sed(/a/,/b/) # triggered for lines between

# ditto for awk

.awk(/a/,/b/)

Rubish Concurrency

Concurrency

• Coarse Grained

– For independent tasks

• No interaction between tasks

– No deadlock

– No shared memory (by abstinence)

– But is safety by policy (read: no safety)

• Ruby Green Thread

– You could always fork… I guess?

Background Jobs

• Executable#{exec!,each!,map!}– #exec! returns an instance of Job

– #each! invokes the iterator in a background thread

– #map!(acc) << into an (hopefully thread safe) accumulator.

• Job#{wait,stop}– #wait returns the result of a job after it completes

– #stop signal a job to terminate, then wait

• For Command,Pipe,Sed,Awk, and more.

Background Jobs

# returns immediatelyrsh> @job = slowcat(3).exec! rsh> jobs # == [@job]# slowcat(3) takes 3 seconds to completersh> waitall # after 3 seconds => @job.wait

rsh> @acc = [] # should use a thread-safe accrsh> ls.map!(@acc)rsh> ls.map!(@acc)

Exception Handling

• Job#wait would raise on abnormal completion

– exitstatus != 0

• Exception avoids error checking cruft

– It’s the 21st century!

• A little suprising

– grep "notfound *"

– wc a-directory

Rubish in Context

Context

• Encapsulates IOs

– Dynamically scoped

• Encapsulates Bindings

– Lexically scoped

– Namespace management

– Local extensibility

• A context defines the meaning of a closure with free bindings.

Context-Sensitive Block

• Object#instance_eval(&block)

obj.instance_eval {

binding1

binding2

}

• Local extensibility

obj.extend(module).instance_eval { … }

Context IO

with {

cmd1.exec

cmd2.exec

with { cmd3 }.o("output3").exec

}.o("output1-2").exec

Context Extension

# ad hoc, local extensions

with(derive({def foo; ...; end})) {

... # outer foo

with(derive({def foo; ...; end})) {

... #inner foo

}

... # outer foo

}

Context Extension with Modules

# prebaked (modules from ‘load’)

with(derive(mod1,mod2)) {

…

}

Batch

• A batch job is a contextualized block executed in a thread.

• Schematically :

@job = Thread.new { context.eval { … }}

@job.wait

• Similar to subshell, but in the same process

• A Batch is also an Executable!

Batch

# first extend context# then carry out a block within a batch threadbatch(derive(...)) { ... }

# all Executable methods apply@job = batch(derive(...)) { ... }.exec!batch(derive(...)) { ... }.mapbatch(derive(...)) { ... }.awk…

Structured Concurrency

# Concurrent Jobs arranged in a tree:batch {exec! cmd1, cmd2batch {exec! cmd3, cmd4batch { exec! cmd5 }

}.exec # we’ll wait till this batch completes…

}.exec!

Conclusion

• OOP is great!

– Inheritance makes code confusing

– Polymorphism is powerful

– Excellent namespace management

• Design by Symbiosis

• Singleton objects for local extensibility

• Object#instance_eval(&block)

• Metaprogramming

– Make things frobbable

– I don’t miss lisp that much…

Thank You

• http://github.com/hayeah/rubish/tree/master

– Need lots more work.

– Generalize Rubish for remote scripting.

• Looking for interesting projects

– I like weird languages.

• hayeah@gmail.com