Next Generation Software Systems and Programming Languages Research Matthias Felleisen.

Next Generation Software Systems and Programming Languages Research

Matthias Felleisen

Our Experience

• PLT has built a bunch of software, including PDE, Web server, IMAP clients, and many more.

• These products share a number of characteristics with other modern software products.

• What does our experience suggest for PL research?

My Thesis, My Goal

• Software systems have become extensible collections of components.

• What does this mean for programming languages: linguistics, pragmatics, virtual machines/compilers, and run-time environments?

Outline

• Two Examples: DrScheme, Web Server

• Language Support for COP• Virtual Machine Support • Contracts for Components

• Summary, Advice

Disclaimer

• PLT’s experience

• My thesis and trajectory

• these are not universal concerns

Examples: Extensible Systems

Two Examples

• DrScheme: a modern, pedagogic programming environment

• The PLT Web Server: an experiment in serving dynamic Web content

DrScheme

• What does it take to build a PDE from scratch?

• How about extensions?

DrScheme scope-sensitive syntax checker

interactive evaluator

algebraic stepper

static debugger

syntax-sensitive editor

Thank You to a Fantastic Team

Cormac, SRC

Shriram Krishnamurthi, Brown

Matthew Flatt, Utah

John Clements, NEU

Robby Findler, UChicago

Paul Steckler, NEU

Philippe Meunier, NEU

Paul Graunke, NEU

DrScheme

Editor

Read-Eval-Print-Loop

model

GUI

view

mutual references

DrScheme: Plugging in Tools

Editor

Read-Eval-Print-LoopTools -- plug-ins

Syntax Checker

Algebraic Stepper

DrScheme: Plugging in Student Programs

Editor

Read-Eval-Print-LoopTools -- plug-ins

Syntax Checker

Algebraic StepperStudent Programs

DrScheme: The Big Picture

Editor

Read-Eval-Print-Loop

a single execution environment: the VM

DrScheme

• powerful plug-ins that interact with editor and read-eval-print loop

• student programs become dynamically plugged in components

• easy communication among all components

A Web Server From Components

• What does it take to build a Web server?

• What about dynamic content?


A server is basically a read-eval-print loop. Upon a TCP connect:

read a request: GET /uniquelylara/paintings/index.html HTTP/1.0 … headers …

interpret the request, the path, and the headers: dispatch on file path, check for defaults, …

print the file: send the requested file back via a TCP write


pattern matching TCP connections URL parsing HTML

Dispatcher

REPL aka Web Server

These components are good old libraries.



Dis: remote

REPL aka Web Server

Dis: local Dis: dynamic

code duplication



Dispatch

REPL aka Web Server

Dispatch Dispatchlocal

remote

dynam

these components are parameterized and instantiated

Web Server: Dynamic Content

• dynamic content generation is becoming more and more important

• dynamic content generation is inefficient and slow

• dynamic content generation is difficult to manage

Web Server: Plug-in Scripts

Dispatcher

Server Loop

Scripts

Web Server: Scripts as Components

Dispatcher

Server Loop

dynamically linked

exchange of valuesnot bit streams


Dispatcher

Server Loop

store script as first-class value


Dispatcher

Server Loop

link in once


Dispatcher

Server Loop

link in once, link in twice


Dispatcher

Server Loop

make the internals of the server available: stack, environment, store


• Our Web server’s performance for dynamic content generation is four times better than that of Apache/Perl and 20 to 40% than mod-Perl.

• It is the only Web server whose scripts are automatically safe for back buttons and cloning: control state.

Other Examples

• Linux:– kernel modules– applications

• Web browsers:– plug-ins– applets

• Multimedia players:– drivers– formats

• MS Office– component: COM

• Java Beans

• .NET

From Component Systems to Programming Languages

From Component Systems to PLs

• component-oriented programming:– different than OOP– different than modules

• VM (virtual machines) for extensible systems of components– more than a virtual machine

• contracts for components

Component Software, Szyperski

• A component is …

– a unit of independent deployment – a unit of third-party composition – a blueprint without state

Szyperski, page 30

COP and Languages: Linguistic Support

• parameterization over– imported components– exchange of values, including critical pieces

• linking– graph-based (mutual references)– hierarchical – dynamic (first-class values in core)

• multi-lingual components

COP and Languages: Virtual Machines

• a VM for many languages

• a VM for dynamic extensions

• a VM for “non-cooperative” extensions

COP and Languages: Quality Assurance

• components exchange values, including objects and functions

• components must protect these values and must promise certain properties– types– (partial) functional specifications– concurrency specifications– QoS guarantees

Linguistic Support for COP

with Matthew Flatt, Robby Findler, Shriram Krishnamurthiand a little bit of help from Dan Friedman

Existing Language Support for Components

Which language constructs should we use to support programming with components?


Proposal: Use first-class procedures and arrange them in patterns.

Pro: hierarchical linking, dynamic invoking, by-reference passing work

Cons: the granularity is too far off. Maintaining “linking patterns” by hand is error prone.


Proposal: Use classes and objects as components.

Pro: larger granularity, contains many functions, protection, dynamic loading

Cons: The super-class is hard-wired, i.e., no parameterization. It is impossible to wire-in the same class many times. Inheritance poses programming and contract checking problems (Szyperski).


Proposal: Use Modula-style modules, Java-style packages, or C#-style assemblies.

Pro: proper granularity

Cons: modules are too static (compile-time). Importing is hard-wired. They are not first-class values.


Proposal: Use ML-style functors. They find good uses in SML and OCAML.

Pro: They are parameterized over their importing modules, by definition.

Cons: they are not values. It is not possible to link them in a graphical manner. Like plain modules, they are static and not dynamically linkable.

PLT’s Components: Units

• Units are parameterized over– imported components– references to values

• Units are linked in – graph-based– hierarchical – dynamic manner

Units are first-class values:

Units: A Language Constructs for COP

imports

definitions

exports

Units: A Language Constructs for COP

type t , val g : int -> t

fun f (an-int, a-t) = …

val f : int t -> string

example:

Units: Graphical Linking

type t


f : int t -> string

type t


f : int t -> string

type t


f : int t -> string

Units: Hierarchical Linking

type t


f : int t -> string

type t


f : int t -> string

type t


f : int t -> string

Units: Dynamic Linking

type t


f : int t -> string

type t


f : int t -> string

type t


f : int t -> string

type t

fun f (an-int, a-t) = … f : int t -> string

Units: Flatt & Felleisen

• PLT’s units support COP well.

• … linking together components is flexible, with rich control

• … widely used in Version 103

• But there are open problems

Units: A Program Design Problem

• Given: a component with – a datatype T with N variants– a collection F of M functions on T

• Wanted: extend T with variants and F with additional functions without access to the source of the component

Krishnamurthi, Felleisen, Friedman

Units: A Program Design Problem

• Why does it matter?– abstract reasons– examples

• How do we do it? – where units work – where we need more

Units Extensibility - Why?

• maintenance: fix once, fix everywhere

• extensibility: add functionality once, add it everywhere

Unit Extensibility: Example 1

• DrScheme: Krishnamurthi et alli the language hierarchy and the set of tools

• datatype is a grammar

• tools: syntax check, type check, stepper, … and more to come


• Geometry server: Hudak, Darpa productivity study

• datatype describes arrangements of geometric shapes

• tools: containment, drawing, moving and many others


• San Francisco project: Bohrer, IBM

• datatype describes customer, manager, account hierarchies

• tools: back office maintenance tools

Unit Extensibility: The Picture

f : … t … -> … g : … -> … t … h : … t … -> … t …

datatype t

The Core Component


f : … t … -> … g : … -> … t … h : … t … -> … t …

datatype t

The Variant Extension

f, g, h : extended


f : … t … -> … g : … -> … t … h : … t … -> … t …

datatype t

The Functionality Extension

f, g, h : extended

j : … t … -> … t ..

k : … t … -> … t ..


f : … t … -> … g : … -> … t … h : … t … -> … t …

datatype t

The Full Component

f, g, h : extended

j : … t … -> … t ..

k : … t … -> … t ..

self-reference (has-a)

self call

Unit Extensibility: The Experience

• Units work well for extensibility

• … just re-link for each extension

• … each layer around a given component is just an adapter unit (Lieberherr & Lorenz ‘01)


• Units alone are not enough

• … need lasses to specify datatypes

• … and mixins for extensions


f : … t … -> … g : … -> … t … h : … t … -> … t …

datatype t

The Full Component

f, g, h : extended

j : … t … -> … t ..

k : … t … -> … t ..

Interfaces don’t knowsuperclass -- mixin

Findler & Flatt


• Linking with units is complex

• Units do not support multiple languages

• PLT also introduced modules.

Component-Oriented Programming

• modules for multi-lingual programs

• units for complex linking

• mixins for specifying and working with data hierarchies plus implementation inheritance

Components in Languages: Research Problems

• still complex: can we simplify this world?

• practical experience:– units and modules “feel” inside out

• type systems: what will a flexible type system for units and modules look like? – mixins (Krishnamurthi)– sharing by specification (Flatt)

Virtual Machines for COP: Resource Administration

with Matthew Flatt, Robby Findler, Shriram Krishnamurthi

Resource Administration: The Situation

• a dynamic plug-in component is probably third-party code with unknown properties

• several plug-ins may run in parallel

• plug-ins compete for finite resources:– files, database servers– network connections– GUI items

Resource Administration: Example 1

• a Web server responds to many requests simultaneously; each may spawn a Web program:– sharing (communicating) with server– sharing among “scripts”

• a script consumes– time– memory– (references to) server stacks– ports, files, databases, …


• DrScheme runs student programs as dynamic extensions so that tools can directly communicate with executing programs

• a program “consumes”– time– memory– GUI windows, … – ports, files, databases, …


• Netscape runs Java applets as dynamic extensions so that …

• a program “consumes”– time– memory– GUI windows, … – ports, files, databases, …

Resource Administration: PLs as OSes

• an extensible component system must play OS for its plug-ins

• the underlying VM must provide mechanisms for resource control

• the underlying programming language must “dress” these mechanisms as language constructs

Resource Administration: PLT’s Solution

• dynamic components as threads

• threads are a resource

• all resources are managed in a custodian

Resource Administration: Custodians

current custodian:

thread 1

thread 2

thread 3

network port 1

network port 2

custodianthread 4 file port 3

GUI 1

custodian shutdown

Resource Administration: Experience

• custodians work well in DrScheme and the Web server

• “special” custodians for GUI because we need to manage GUI events (event spaces)

Resource Administration: Problems

• Custodians don’t deal with memory. (Flatt)

• Custodian actions are fixed. Can we specify them?

• Other OS services?

• Update code dynamically? (Krishnamurthi)

Resource Administration

• In the past, advanced languages provided memory administration (garbage collection). Java has introduced GC into the mainstream.

• In the future, advanced languages will provide other resource administration tools. .NET will provide meta-data with assemblies.

Enforcing Invariants in COP

with Robby Findler, Mario Latendresse

Contract for Components: The Problem

The System A Plugin

invariants

Contract for Components: The Problem

The System A Plugin

invariants

The System consists of many components.

If it blows up, who did it? Economics!

Contracts for Components: The Idea

• need behavioral contracts in addition to type signatures for external wires of components

• must monitor these assertions at run-time (or partially verify them)

• assign blame to faulty component

Contracts for Components:Eiffel

• design by contract a la– Eiffel, iContract, jContractor, HandShake,

JVMAssert, …

• get it right, scale to other constructs

• then scale to COP

Contracts for Components: Example

double square_root(double x) { … } @pre: x >= 0 @post: fn result -> result >= 0

double square_root(double x) { … } @pre: x >= 0 @post: fn x -> fn result -> x *x >= result >= 0

partial behavior:

total behavior:

Contracts for Components: Meaning and Blame

• test pre-conditions for method call, blame caller

• test post-condition for method return, blame callee

• deal with invariants for variables similarly

Contracts for Components: Classes and Objects

• classes (or mixins) are in an inheritance hierarchy

• method contracts must be related

• test implications for pre/post conditions at method call & return, blame class extension

Contracts for Components: Functions

• testing predicates on functions is impossible

• use proxy functions that perform tests for first-order types, blame current co-variant party for bad arguments and contra-variant party for bad result

• switch blame positions with applications

Contracts for Components: Experience

• contracts are as important as types

• contracts complement types:– properties of arguments (subset of types)– relating two arguments– relating two functions in a component

Contracts for Components

• OOP and contracts

• FP and contracts

• COP and contracts: – types– linking: graphs, dynamic, hierarchical

Summary and Advice

Software Systems

• build large systems from components– in-house– third party– different sizes

• plug-in components statically

• plug-in components at run-time

Software Systems

• systems have complex interactions

• systems act as quasi-OS’es for dynamic plug-ins

• such systems consist of components in many languages

Programming Languages

• provide VMs that support high-level values and value sharing

• provide VMs that support resource administration for non-collectible resources

• provide VMs that accommodate many languages


• provide constructs for constructing and linking components of all sizes and all shapes

• provide constructs for dynamic extensions, support mini-OSes for COP


• provide interfaces with contracts

• check contracts in a well-founded manner

• assign blame properly

PLT’s Solutions

• modules, units, & mixins are a good solution, but not the last word

• custodians are a good start toward resource administration, but not the last word

• and this slide isn’t the last word

Advice for Academics: PL Research

• build systems, not (just) compilers:• useful and team-sized, e.g., DrScheme • useful and pair-sized, e.g., Web server• useful individual scripts• get “customers”

• test academic languages with all kinds of system; use feedback for PL design

Advice for Industry

• the JVM was a start; it does not accommodate other languages easily, it does not administrate resources

• the .NET world is made for other languages; it still lacks resource administration

• these machines are not the last word, they are just a start for COP

now go forth and do some more type research

Next Generation Software Systems and Programming Languages Research Matthias Felleisen.

Documents

Transcript of Next Generation Software Systems and Programming Languages Research Matthias Felleisen.