Generative Software Development. Overview and Examples

Generative Software Development

overview and examples

Eelco [email protected]

Universiteit Utrecht, Faculty of Science,Department of Information and Computing Sciences,

Center for Software Technology

October 27, 2005

[email protected]


Generative software development is a system-familyapproach, which focuses on automating the creation ofsystem-family members: a given system can beautomatically generated from a specification written inone or more domain-specific languages.

K. Czarnecki. Overview of Generative Software Development. In J.-P. Bantre

et al. (Eds.): Unconventional Programming Paradigms (UPP) 2004, Mont

Saint-Michel, France, LNCS 3566, pp. 313-328, 2005

Outline

• Software system families

• Commonality and variability• Domain engineering and application engineering• Feature modeling

• Domain-specific languages

• Examples

• Generative programming

• Problem space• Solution space• Mapping

• Realizing generative software development

• Overview of techniques• Example: assimilating embedded DSLs

• Risks

• Summary

Software System Families

Current research and practical experience suggest thatachieving significant progress with respect to softwarereuse requires a paradigm shift towards modeling anddeveloping software system families rather thanindividual systems.

Czarnecki 2005

Examples of Software System Families

• Compilers

• program representation, parsing, code generation, ...

• Web applications

• processing http requests, session management

• E-commerce systems

• products, user accounts, payment

• Office applications

• text processing

• Web browsers

• html rendering

• TV software

System Family Engineering

System-family engineering (product-line engineering)seeks to exploit the commonalities among systems froma given problem domain while managing the variabilitiesamong them in a systematic way.

Czarnecki 2005

Commonality and Variability

• E-commerce systems

• Commonalities: products, user accounts, payment, ...• Variabilities: types of products, shipment methods, ...

• Web applications

• Commonalities: processing http requests, session management• Variabilities: application workflow, page layout, ...

• Web browsers

• Commonalities: render HTML• Variabilities: plugins for email, chatting, ...

Commonality and Variability

• Compilers

• Commonalities: program representation, parsing, codegeneration, ...

• Variabilities: syntax and semantics of language

• Office applications

• Commonalities: text processing• Variabilities: spellchecking, grammarchecking, ...

• TV software

• Commonalities: ?• Variabilities: ?

System Family Engineering Process

system family engineering=

domain engineering+

application engineering

Domain engineering

Domain analysisDomain design

Domain implementation

Application engineering

System requirementsSystem derivation System tailoring

reusable assets

new requirements

Domain Engineering = Development for Reuse

Development of reusable assets

• Reusable components, generic architectures, documentation

Software engineering process for a class of systems

• Domain analysis

• scope of family• identify commonalities and variabilities

• Domain design

• common architecture• plan creation of systems from reusable assets

• Domain implementation

• implementation of reusable assets

Feature Modeling

A method and notation to elicit and represent common andvariable features of the systems in a system family

• And: all subfeatures must be selected

• Alternative: only one subfeature can be selected

• Or: one or more can be selected

• Mandatory: features that are required

• Optional: features that are optional

Additional information

• constraints

• binding times

• default attribute values, default features

• stakeholders, priorities, etc.

Feature Modeling: Example

e-shop

payment taxCalculation shipping

creditCard debitCard electronicCheque addressResolution

country postalCode streetAddress

Feature Modeling: Alternative Notations

• Feature diagrams can be expressed as context-free grammars

• Feature selection is sentence generated by the grammar

e-shop : payment taxCalculation [shipping]

payment : paymentMethod+

paymentMethod : creditCard | debitCard | electronicCheque

taxCalculation : addressResolution

addressResolution : country | postalCode | streetAddres

• Expressivity of feature diagrams is restricted

• Features and other constraints can be expressed by means ofpropositional formulae

See also D. Batory. Feature Models, Grammars, and Propositional Formulas. In

H. Obbink and K. Pohl (Eds.) SPLC 2005. LNCS 3714, pp. 7–20, 2005

Application Engineering = Development with Reuse

Build concrete applications using reusable assets

• Requirements = delta of generic system requirements

• System derivation: manual or automated construction ofsystem from reusable assets

Feedback to domain engineering

• Refine and adapt requirements

• Tailor reusable assets to application

Features vs Software Modules

Concrete features

• map to individual components

Crosscutting features

• can be implemented using aspect-oriented techniques

• Examples: logging, synchronization

Abstract features

• Also known as: non-functional requirements

• map to configuration of components and/or aspects

Grouping features

• Also known as: variation points

• map to common interface and plug-compatible components



Czarnecki 2005

General Purpose Languages

Expressivity of general purpose languages is restricted to

• while(cond) { x = foo(y, z); }

• variable binding: storing values

• function calls: computing values

• control-flow: combining and choosing computations

This is a rather powerful and generic approach

• any computation can be expressed (Turing completeness)

• modularization facilities allow further abstraction

But it does not match application domains very well

Advantages of Domain-Specific Languages

• Domain-specific abstractions

• pre-defined abstractions for concepts in application domain

• Domain-specific concrete syntax

• natural notation for a domain

• Domain-specific error checking

• find and report errors based on domain knowledge and notation

• Domain-specific optimizations

• opportunities for optimizations based on domain knowledge

• Domain-specific tool support

• tools can be aware of domain

Domain-Specific Languages: Examples

Successful domain-specific languages exist for many domains

• Numerical computation

• Database queries

• Text search and analysis

• Symbolic computation

• Document transformation

• Workflow

• ...

Domain-Specific Languages: Arithmetic

x = c * y + z

• Notation and semantics based on long tradition inmathematics

• General purpose language notation would require

x = plus(times(c, y), z)x = z.plus(y.times(c))x = c.times(y).plus(z)

• Compiler knows a lot about arithmetic

• optimizations (partial evaluation, strength reduction, ...)• automatic coercions (int to real)• exceptions

Domain-Specific Languages:

Matrix Computation

A = c * B + C

• Now the elements of the assignment are matrices

• The expression ‘means’

for i = lb to ub doA[i] = c * B[i] + C[i]

• General-purpose languages (C, C++, Java, ...) don’tunderstand matrices

• ‘Mathematical’ languages such as Matlab and Octave do

• Matrix notation allows

• concise programs• compiler may apply optimizations

Domain-Specific Languages: Matrix Computation

A = c * B + C

• Now the elements of the assignment are matrices

• The expression ‘means’

for i = lb to ub doA[i] = c * B[i] + C[i]

• General-purpose languages (C, C++, Java, ...) don’tunderstand matrices

• ‘Mathematical’ languages such as Matlab and Octave do

• Matrix notation allows

• concise programs• compiler may apply optimizations

Domain-Specific Languages: Database Queries

SELECT NAMEFROM ORDERS, INCLUDES, SUPPLIESWHERE CUST = ’Chaudron’

AND ORDERS.O# = INCLUDUES.O#AND INCLUDES.ITEM = SUPPLIES.ITEM

• SQL: notation for querying relational databases

• Abstracts from physical data structures of database

• Based on small set of relational algebra operators

• Allows efficient implementation of queries

• Queries can be checked against database schema

Domain-Specific Languages: Regular Expressions

^([a-z]+)/([A-Z][^/]+)/(.*)

• A regular expression describes or matches a set of strings,according to certain syntax rules

• Used by many text editors and utilities to search andmanipulate bodies of text based on certain patterns.

• Many programming languages support regular expressions forstring manipulation.

• Notation abstracts from underlying finite automata

Domain-Specific Languages: Context-free Grammars

<postal-address> ::= <name-part> <street-address> <zip-part>

<personal-part> ::= <first-name> | <initial> "."

<name-part> ::= <personal-part> <last-name> [<jr-part>] <EOL>

| <personal-part> <name-part>

<street-address> ::= [<apt>] <house-num> <street-name> <EOL>

<zip-part> ::= <town-name> "," <state-code> <ZIP-code> <EOL>

• Describes syntax of a formal language

• Used to derive parsers that automatically recognize texts

• Writing parsers manually is complex and error-prone

Domain-Specific Languages: Rewrite Systems

|[ if 0 then s1 else s2 ]| -> |[ s2 ]|

|[ i + j ]| -> |[ k ]| where <add>(i,j) => k

• Rewrite rule replaces a matching term with another term

• Exhaustive application of rules reduces term to normal form

• Applications: symbolic computation, program transformation,

Domain-Specific Languages: Path Queries

/bookstore/book[price>35]/title

• XPATH path queries address parts of an XML document

• used in document transformation with XSLT

Domain-Specific Languages: Workflow

Domain-Specific Languages: Observations

• Domain-specific languages are everywhere

• DSL are sucessful when they cover a well-defined domain andare based on solid foundations

• DSLs typically do not cover entire systems, but ratherwell-defined parts/aspects of systems



Czarnecki 2005

Generative Programming: Mapping Problems to Solutions

Problem space Solution space

mapping

Problem Space

• Set of domain specific abstractions

• Used to specify desired system family member

Solution Space

• Implementation oriented abstractions

• Can be instantiated to implement domain abstractions

Problem Space and Solution Space: Example

Problem Space

• Regular expressions

Solution Space

• Finite automata

• Expressed as transition table in code

Mapping

• Lexical analyzer generator (flex)

Mapping: Configuration View

Problem space

• domain-specific concepts and their features

• constraints: illegal feature combinations, default settings,default dependencies

Specification

• selection of features for a specific system

Solution space

• implementation components, composed to create system

Mapping

• feature configuration mapped to component configuration

• mapping defined by construction rules and optimizations

Separation of spaces allows freedom of implementation

Configuration Example: Apache’s httpd.conf

Software composition

LoadModule file_cache_module modules/mod_file_cache.so

LoadModule mem_cache_module modules/mod_mem_cache.so

LoadModule cgi_module modules/mod_cgi.so

Configuration settings

<Directory /home/*/public_html>

AllowOverride FileInfo AuthConfig Limit

Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec

<Limit GET POST OPTIONS>

Order allow,deny

Allow from all

</Limit>

<LimitExcept GET POST OPTIONS>

Order deny,allow

Deny from all

</LimitExcept>

</Directory>

Binding-time: configuration is processed at start-up time

Mapping: Transformational View

Problem space

• domain concepts captured by domain-specific language

Specification

• DSL program

Solution space

• implementation language

• components/libraries (‘run-time system’)

Mapping

• transformation from DSL to implementation language

Configuration vs Transformation

• Two views on mappings provide different perspectives

• Feature configuration defines a DSL with components asimplementation language

Combining Mappings

chaining

multiple problem spaces multiple solution spaces

alternative problem spaces alternative solution spaces

Combining Mappings

text search

regexp

SDF

embed

DFA

automaton

SLR(0) table

pgen

pp table

sdf2pp

signature

sdf2sig

languagedefinition

code

codegen

sglr

parse

Realizing Generative Software Development

Requires techniques/technologies for creating

• Domain-specific languages

• Mappings

• Solution components

Realizing Domain-Specific Languages

Domain-specific languages come in many shapes and forms

• Textual languages

• Embedded in a programming language (MetaBorg)

• Graphical languages

• Wizards, GUIs

Realizing Mappings

Product line configurators

• map configuration file + components to system

Generators, implemented using

• Template and frame processors

• Transformation systems

• Multi-staged programming

• Program specialization

• Built-in meta-programming facility

Interpreters

• run-time interpretation of DSL program

Realizing Solution Space

• Functions or classes

• Generic components

• Component models (JavaBeans, ActiveX, Corba)

• Aspect-oriented programming (AspectJ)

Combining Techniques

Generative software development can be realized using variouscombinations of techniques

Example scenario:

• Textual embedded language

• ‘Assimilation’ by program transformation

• Library for basic functionality of target programs

Example scenario:

• Graphical model language

• Graph transformations to map to aspect-oriented programs

• Aspects ‘inject’ code from model into base application

60�

��

��

��

��

��

��

Relation To Other Fields

System Families, Product Lines

Configuration KnowledgeProblem Space Solution Space

Components

Architectures

Generic programming

Aspect-oriented programming

Feature modeling, feature interactions

Domain-specific languages

Generators/Transformers

Aspect-oriented modeling

Assimilation of Embedded Languages

Embedding domain-specific languages

• Extend language with syntax for application domain

Examples

• Java + Swing Userinterface Language: Expressions reflect

hierarchy of Swing GUI objects

• Java + RegExps: Compile-time syntactic checks of regular

expressions, string rewrite rules

• Java + Java: Java programs that generate Java programs

Assimilation

• Meld embedded DSL code with host code

• Often requires local-to-global transformations




Examples






Assimilation



Concrete syntax for objects

I syntactic abstractions for libraries (APIs)

I transformations to assimilate (and optimize)




Examples






Assimilation



local to global transformation

I generation of event handler class

I caching of regexp compiles

I adding class variable declarations

Example: Embedding Regular Expressions

re := str | alpha | re1|re2 | re1re2 | re* | re+e := /re/ e | /re;f / xt := regexp regexps

/re/e tests whether regular expression re matches value of e

function isEmail(x : string) : intbeginreturn / alpha+ "@" (alpha+ ".")+ alpha+ / x;

end

/re;f/e tests if regular expression re matches a prefix of e, andthe suffix satisfies predicate f

Assimilating Regular Expression Matching

function match(x : string) : int

/ ("a" | "b") "c" / x;

Desugar =

ReMch <+ ReStr <+ ReAlt <+ ReSeq <+ ReKle <+ once-AddDef



/ ("a" | "b") "c" / x;

Desugar =


ReMch :

|[ /re/ e ]| ->

|[ begin

var x : string; x := e;

return /re; isEmpty/ x;

end ]|

where new => x

Rewrite rule

I L : t1 -> t2 where s

I replace fragment matching t1 with t2



begin


end

ReSeq :

|[ /re1 re2 ; f/ x ]| -> |[ /re1 ; g/ x ]|

where new => g

; add-def(||[

function g(a : string) : int /re2; f/ a;

]|)

add-def(|def) =

rules( AddDef :+ |[ def* ]| -> |[ def def* ]| )



begin


end

ReSeq :

|[ /re1 re2 ; f/ x ]| -> |[ /re1 ; g/ x ]|

where new => g

; add-def(||[


]|)

add-def(|def) =


AddDef :

|[ def* ]| -> |[ function k(a : string) : int /"c"; isEmpty/ a; def* ]|

Dynamic rules

I rules( L : t1 -> t2 )

I defines new rewrite rule

I inherits variable bindings from definition context



begin

var z : string; z := x; return / ("a" | "b") ; k / z;

end

ReSeq :

|[ /re1 re2 ; f/ x ]| -> |[ /re1 ; g/ x ]|

where new => g

; add-def(||[


]|)

add-def(|def) =


AddDef :




begin

var z : string; z := x;

return (isPref("a",z) & k(getSuf("a",z)))

| (isPref("b",z) & k(getSuf("b",z)))

end

ReStr :




begin

var z : string; z := x;



end

Desugar =


AddDef :



function k(a : string) : int

begin

return /"c"; isEmpty/ a;

end


begin

var z : string; z := x



end

Desugar =


AddDef :




begin

return /"c"; isEmpty/ a;

end


begin




end

ReStr :




begin

return isPref("c", a) & isEmpty(getSuf("c", a));

end


begin




end

ReStr :


Risks of Domain-Specific Languages

• Design and implementation effort

• implementing complete toolset for language is major effort

• Documentation

• tutorials, reference manual, implementation documentation

• Training

• need programmers that are skilled in using the language

• Maintenance

• of the language• of DSL programs

• How does it fit in the existing work flow

• tool chain

• DSLs need abstraction/modularity features too

• DSL programs become large; reuse of parts needed

Conclusion

Summary

• Generative software development is encapsulation of domainknowledge in generators and target components

• Domain-specific languages are the interface to this knowledge

• DSL programs can be seen as a configuration for the ‘solutionspace’ components

Challenges

• Finding good formalizations and encapsulations of newapplication domains

• A platform for cheap construction of DSLs and generators

• Integration of generators in workflow/toolchain

References

• K. Czarnecki. Overview of Generative Software Development.In J.-P. Bantre et al. (Eds.): Unconventional ProgrammingParadigms (UPP) 2004, Mont Saint-Michel, France, LNCS 3566,pp. 313-328, 2005

• D. Batory. Feature Models, Grammars, and PropositionalFormulas. In H. Obbink and K. Pohl (Eds.) SPLC 2005. LNCS3714, pp. 7–20, 2005

• M. Bravenboer and E. Visser. Concrete Syntax for Objects.Domain-Specific Language Embedding and Assimilationwithout Restrictions. In D. C. Schmidt, editor, Proceedings of the19th ACM SIGPLAN Conference on Object-Oriented Programing,Systems, Languages, and Applications (OOPSLA’04), pages365–383, Vancouver, Canada, October 2004. ACM Press.

• E. Visser. Transformations for Abstractions. In J. Krinke and G.Antoniol, editors, Fifth IEEE International Workshop on SourceCode Analysis and Manipulation (SCAM’05), Budapest, Hungary,October 2005. IEEE Computer Society Press. (Keynote paper)

Further Studying

At Master Program Software Technology; Universiteit Utrecht

• Seminar on Software Generation and Configuration

• Master course on Program Transformation

Generative Software Development. Overview and Examples

Technology

Transcript of Generative Software Development. Overview and Examples