Post on 05-Dec-2014
description
Generative Software Development
overview and examples
Eelco Visservisser@acm.org
Universiteit Utrecht, Faculty of Science,Department of Information and Computing Sciences,
Center for Software Technology
October 27, 2005
Generative Software Development
Generative software development is a system-familyapproach, which focuses on automating the creation ofsystem-family members: a given system can beautomatically generated from a specification written inone or more domain-specific languages.
K. Czarnecki. Overview of Generative Software Development. In J.-P. Bantre
et al. (Eds.): Unconventional Programming Paradigms (UPP) 2004, Mont
Saint-Michel, France, LNCS 3566, pp. 313-328, 2005
Outline
• Software system families
• Commonality and variability• Domain engineering and application engineering• Feature modeling
• Domain-specific languages
• Examples
• Generative programming
• Problem space• Solution space• Mapping
• Realizing generative software development
• Overview of techniques• Example: assimilating embedded DSLs
• Risks
• Summary
Software System Families
Current research and practical experience suggest thatachieving significant progress with respect to softwarereuse requires a paradigm shift towards modeling anddeveloping software system families rather thanindividual systems.
Czarnecki 2005
Examples of Software System Families
• Compilers
• program representation, parsing, code generation, ...
• Web applications
• processing http requests, session management
• E-commerce systems
• products, user accounts, payment
• Office applications
• text processing
• Web browsers
• html rendering
• TV software
System Family Engineering
System-family engineering (product-line engineering)seeks to exploit the commonalities among systems froma given problem domain while managing the variabilitiesamong them in a systematic way.
Czarnecki 2005
Commonality and Variability
• E-commerce systems
• Commonalities: products, user accounts, payment, ...• Variabilities: types of products, shipment methods, ...
• Web applications
• Commonalities: processing http requests, session management• Variabilities: application workflow, page layout, ...
• Web browsers
• Commonalities: render HTML• Variabilities: plugins for email, chatting, ...
Commonality and Variability
• Compilers
• Commonalities: program representation, parsing, codegeneration, ...
• Variabilities: syntax and semantics of language
• Office applications
• Commonalities: text processing• Variabilities: spellchecking, grammarchecking, ...
• TV software
• Commonalities: ?• Variabilities: ?
System Family Engineering Process
system family engineering=
domain engineering+
application engineering
Domain engineering
Domain analysisDomain design
Domain implementation
Application engineering
System requirementsSystem derivation System tailoring
reusable assets
new requirements
Domain Engineering = Development for Reuse
Development of reusable assets
• Reusable components, generic architectures, documentation
Software engineering process for a class of systems
• Domain analysis
• scope of family• identify commonalities and variabilities
• Domain design
• common architecture• plan creation of systems from reusable assets
• Domain implementation
• implementation of reusable assets
Feature Modeling
A method and notation to elicit and represent common andvariable features of the systems in a system family
• And: all subfeatures must be selected
• Alternative: only one subfeature can be selected
• Or: one or more can be selected
• Mandatory: features that are required
• Optional: features that are optional
Additional information
• constraints
• binding times
• default attribute values, default features
• stakeholders, priorities, etc.
Feature Modeling: Example
e-shop
payment taxCalculation shipping
creditCard debitCard electronicCheque addressResolution
country postalCode streetAddress
Feature Modeling: Alternative Notations
• Feature diagrams can be expressed as context-free grammars
• Feature selection is sentence generated by the grammar
e-shop : payment taxCalculation [shipping]
payment : paymentMethod+
paymentMethod : creditCard | debitCard | electronicCheque
taxCalculation : addressResolution
addressResolution : country | postalCode | streetAddres
• Expressivity of feature diagrams is restricted
• Features and other constraints can be expressed by means ofpropositional formulae
See also D. Batory. Feature Models, Grammars, and Propositional Formulas. In
H. Obbink and K. Pohl (Eds.) SPLC 2005. LNCS 3714, pp. 7–20, 2005
Application Engineering = Development with Reuse
Build concrete applications using reusable assets
• Requirements = delta of generic system requirements
• System derivation: manual or automated construction ofsystem from reusable assets
Feedback to domain engineering
• Refine and adapt requirements
• Tailor reusable assets to application
Features vs Software Modules
Concrete features
• map to individual components
Crosscutting features
• can be implemented using aspect-oriented techniques
• Examples: logging, synchronization
Abstract features
• Also known as: non-functional requirements
• map to configuration of components and/or aspects
Grouping features
• Also known as: variation points
• map to common interface and plug-compatible components
Generative Software Development
Generative software development is a system-familyapproach, which focuses on automating the creation ofsystem-family members: a given system can beautomatically generated from a specification written inone or more domain-specific languages.
Czarnecki 2005
General Purpose Languages
Expressivity of general purpose languages is restricted to
• while(cond) { x = foo(y, z); }
• variable binding: storing values
• function calls: computing values
• control-flow: combining and choosing computations
This is a rather powerful and generic approach
• any computation can be expressed (Turing completeness)
• modularization facilities allow further abstraction
But it does not match application domains very well
Advantages of Domain-Specific Languages
• Domain-specific abstractions
• pre-defined abstractions for concepts in application domain
• Domain-specific concrete syntax
• natural notation for a domain
• Domain-specific error checking
• find and report errors based on domain knowledge and notation
• Domain-specific optimizations
• opportunities for optimizations based on domain knowledge
• Domain-specific tool support
• tools can be aware of domain
Domain-Specific Languages: Examples
Successful domain-specific languages exist for many domains
• Numerical computation
• Database queries
• Text search and analysis
• Symbolic computation
• Document transformation
• Workflow
• ...
Domain-Specific Languages: Arithmetic
x = c * y + z
• Notation and semantics based on long tradition inmathematics
• General purpose language notation would require
x = plus(times(c, y), z)x = z.plus(y.times(c))x = c.times(y).plus(z)
• Compiler knows a lot about arithmetic
• optimizations (partial evaluation, strength reduction, ...)• automatic coercions (int to real)• exceptions
Domain-Specific Languages:
Matrix Computation
A = c * B + C
• Now the elements of the assignment are matrices
• The expression ‘means’
for i = lb to ub doA[i] = c * B[i] + C[i]
• General-purpose languages (C, C++, Java, ...) don’tunderstand matrices
• ‘Mathematical’ languages such as Matlab and Octave do
• Matrix notation allows
• concise programs• compiler may apply optimizations
Domain-Specific Languages: Matrix Computation
A = c * B + C
• Now the elements of the assignment are matrices
• The expression ‘means’
for i = lb to ub doA[i] = c * B[i] + C[i]
• General-purpose languages (C, C++, Java, ...) don’tunderstand matrices
• ‘Mathematical’ languages such as Matlab and Octave do
• Matrix notation allows
• concise programs• compiler may apply optimizations
Domain-Specific Languages: Matrix Computation
A = c * B + C
• Now the elements of the assignment are matrices
• The expression ‘means’
for i = lb to ub doA[i] = c * B[i] + C[i]
• General-purpose languages (C, C++, Java, ...) don’tunderstand matrices
• ‘Mathematical’ languages such as Matlab and Octave do
• Matrix notation allows
• concise programs• compiler may apply optimizations
Domain-Specific Languages: Database Queries
SELECT NAMEFROM ORDERS, INCLUDES, SUPPLIESWHERE CUST = ’Chaudron’
AND ORDERS.O# = INCLUDUES.O#AND INCLUDES.ITEM = SUPPLIES.ITEM
• SQL: notation for querying relational databases
• Abstracts from physical data structures of database
• Based on small set of relational algebra operators
• Allows efficient implementation of queries
• Queries can be checked against database schema
Domain-Specific Languages: Regular Expressions
^([a-z]+)/([A-Z][^/]+)/(.*)
• A regular expression describes or matches a set of strings,according to certain syntax rules
• Used by many text editors and utilities to search andmanipulate bodies of text based on certain patterns.
• Many programming languages support regular expressions forstring manipulation.
• Notation abstracts from underlying finite automata
Domain-Specific Languages: Context-free Grammars
<postal-address> ::= <name-part> <street-address> <zip-part>
<personal-part> ::= <first-name> | <initial> "."
<name-part> ::= <personal-part> <last-name> [<jr-part>] <EOL>
| <personal-part> <name-part>
<street-address> ::= [<apt>] <house-num> <street-name> <EOL>
<zip-part> ::= <town-name> "," <state-code> <ZIP-code> <EOL>
• Describes syntax of a formal language
• Used to derive parsers that automatically recognize texts
• Writing parsers manually is complex and error-prone
Domain-Specific Languages: Rewrite Systems
|[ if 0 then s1 else s2 ]| -> |[ s2 ]|
|[ i + j ]| -> |[ k ]| where <add>(i,j) => k
• Rewrite rule replaces a matching term with another term
• Exhaustive application of rules reduces term to normal form
• Applications: symbolic computation, program transformation,
Domain-Specific Languages: Path Queries
/bookstore/book[price>35]/title
• XPATH path queries address parts of an XML document
• used in document transformation with XSLT
Domain-Specific Languages: Workflow
Domain-Specific Languages: Observations
• Domain-specific languages are everywhere
• DSL are sucessful when they cover a well-defined domain andare based on solid foundations
• DSLs typically do not cover entire systems, but ratherwell-defined parts/aspects of systems
Generative Software Development
Generative software development is a system-familyapproach, which focuses on automating the creation ofsystem-family members: a given system can beautomatically generated from a specification written inone or more domain-specific languages.
Czarnecki 2005
Generative Programming: Mapping Problems to Solutions
Problem space Solution space
mapping
Problem Space
• Set of domain specific abstractions
• Used to specify desired system family member
Solution Space
• Implementation oriented abstractions
• Can be instantiated to implement domain abstractions
Problem Space and Solution Space: Example
Problem Space
• Regular expressions
Solution Space
• Finite automata
• Expressed as transition table in code
Mapping
• Lexical analyzer generator (flex)
Mapping: Configuration View
Problem space
• domain-specific concepts and their features
• constraints: illegal feature combinations, default settings,default dependencies
Specification
• selection of features for a specific system
Solution space
• implementation components, composed to create system
Mapping
• feature configuration mapped to component configuration
• mapping defined by construction rules and optimizations
Separation of spaces allows freedom of implementation
Configuration Example: Apache’s httpd.conf
Software composition
LoadModule file_cache_module modules/mod_file_cache.so
LoadModule mem_cache_module modules/mod_mem_cache.so
LoadModule cgi_module modules/mod_cgi.so
Configuration settings
<Directory /home/*/public_html>
AllowOverride FileInfo AuthConfig Limit
Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec
<Limit GET POST OPTIONS>
Order allow,deny
Allow from all
</Limit>
<LimitExcept GET POST OPTIONS>
Order deny,allow
Deny from all
</LimitExcept>
</Directory>
Binding-time: configuration is processed at start-up time
Mapping: Transformational View
Problem space
• domain concepts captured by domain-specific language
Specification
• DSL program
Solution space
• implementation language
• components/libraries (‘run-time system’)
Mapping
• transformation from DSL to implementation language
Configuration vs Transformation
• Two views on mappings provide different perspectives
• Feature configuration defines a DSL with components asimplementation language
Combining Mappings
chaining
multiple problem spaces multiple solution spaces
alternative problem spaces alternative solution spaces
Combining Mappings
text search
regexp
SDF
embed
DFA
automaton
SLR(0) table
pgen
pp table
sdf2pp
signature
sdf2sig
languagedefinition
code
codegen
sglr
parse
Realizing Generative Software Development
Requires techniques/technologies for creating
• Domain-specific languages
• Mappings
• Solution components
Realizing Domain-Specific Languages
Domain-specific languages come in many shapes and forms
• Textual languages
• Embedded in a programming language (MetaBorg)
• Graphical languages
• Wizards, GUIs
Realizing Mappings
Product line configurators
• map configuration file + components to system
Generators, implemented using
• Template and frame processors
• Transformation systems
• Multi-staged programming
• Program specialization
• Built-in meta-programming facility
Interpreters
• run-time interpretation of DSL program
Realizing Solution Space
• Functions or classes
• Generic components
• Component models (JavaBeans, ActiveX, Corba)
• Aspect-oriented programming (AspectJ)
Combining Techniques
Generative software development can be realized using variouscombinations of techniques
Example scenario:
• Textual embedded language
• ‘Assimilation’ by program transformation
• Library for basic functionality of target programs
Example scenario:
• Graphical model language
• Graph transformations to map to aspect-oriented programs
• Aspects ‘inject’ code from model into base application
60�
����
�����
��
�� ����
���
���� �����
Relation To Other Fields
System Families, Product Lines
Configuration KnowledgeProblem Space Solution Space
Components
Architectures
Generic programming
Aspect-oriented programming
Feature modeling, feature interactions
Domain-specific languages
Generators/Transformers
Aspect-oriented modeling
Assimilation of Embedded Languages
Embedding domain-specific languages
• Extend language with syntax for application domain
Examples
• Java + Swing Userinterface Language: Expressions reflect
hierarchy of Swing GUI objects
• Java + RegExps: Compile-time syntactic checks of regular
expressions, string rewrite rules
• Java + Java: Java programs that generate Java programs
Assimilation
• Meld embedded DSL code with host code
• Often requires local-to-global transformations
Assimilation of Embedded Languages
Embedding domain-specific languages
• Extend language with syntax for application domain
Examples
• Java + Swing Userinterface Language: Expressions reflect
hierarchy of Swing GUI objects
• Java + RegExps: Compile-time syntactic checks of regular
expressions, string rewrite rules
• Java + Java: Java programs that generate Java programs
Assimilation
• Meld embedded DSL code with host code
• Often requires local-to-global transformations
Concrete syntax for objects
I syntactic abstractions for libraries (APIs)
I transformations to assimilate (and optimize)
Assimilation of Embedded Languages
Embedding domain-specific languages
• Extend language with syntax for application domain
Examples
• Java + Swing Userinterface Language: Expressions reflect
hierarchy of Swing GUI objects
• Java + RegExps: Compile-time syntactic checks of regular
expressions, string rewrite rules
• Java + Java: Java programs that generate Java programs
Assimilation
• Meld embedded DSL code with host code
• Often requires local-to-global transformations
local to global transformation
I generation of event handler class
I caching of regexp compiles
I adding class variable declarations
Example: Embedding Regular Expressions
re := str | alpha | re1|re2 | re1re2 | re* | re+e := /re/ e | /re;f / xt := regexp regexps
/re/e tests whether regular expression re matches value of e
function isEmail(x : string) : intbeginreturn / alpha+ "@" (alpha+ ".")+ alpha+ / x;
end
/re;f/e tests if regular expression re matches a prefix of e, andthe suffix satisfies predicate f
Example: Embedding Regular Expressions
re := str | alpha | re1|re2 | re1re2 | re* | re+e := /re/ e | /re;f / xt := regexp regexps
/re/e tests whether regular expression re matches value of e
function isEmail(x : string) : intbeginreturn / alpha+ "@" (alpha+ ".")+ alpha+ / x;
end
/re;f/e tests if regular expression re matches a prefix of e, andthe suffix satisfies predicate f
Example: Embedding Regular Expressions
re := str | alpha | re1|re2 | re1re2 | re* | re+e := /re/ e | /re;f / xt := regexp regexps
/re/e tests whether regular expression re matches value of e
function isEmail(x : string) : intbeginreturn / alpha+ "@" (alpha+ ".")+ alpha+ / x;
end
/re;f/e tests if regular expression re matches a prefix of e, andthe suffix satisfies predicate f
Assimilating Regular Expression Matching
function match(x : string) : int
/ ("a" | "b") "c" / x;
Desugar =
ReMch <+ ReStr <+ ReAlt <+ ReSeq <+ ReKle <+ once-AddDef
Assimilating Regular Expression Matching
function match(x : string) : int
/ ("a" | "b") "c" / x;
Desugar =
ReMch <+ ReStr <+ ReAlt <+ ReSeq <+ ReKle <+ once-AddDef
ReMch :
|[ /re/ e ]| ->
|[ begin
var x : string; x := e;
return /re; isEmpty/ x;
end ]|
where new => x
Rewrite rule
I L : t1 -> t2 where s
I replace fragment matching t1 with t2
Assimilating Regular Expression Matching
function match(x : string) : int
begin
var z : string; z := x; return / ("a" | "b") "c"; isEmpty / z;
end
Desugar =
ReMch <+ ReStr <+ ReAlt <+ ReSeq <+ ReKle <+ once-AddDef
ReMch :
|[ /re/ e ]| ->
|[ begin
var x : string; x := e;
return /re; isEmpty/ x;
end ]|
where new => x
Assimilating Regular Expression Matching
function match(x : string) : int
begin
var z : string; z := x; return / ("a" | "b") "c"; isEmpty / z;
end
ReSeq :
|[ /re1 re2 ; f/ x ]| -> |[ /re1 ; g/ x ]|
where new => g
; add-def(||[
function g(a : string) : int /re2; f/ a;
]|)
add-def(|def) =
rules( AddDef :+ |[ def* ]| -> |[ def def* ]| )
Assimilating Regular Expression Matching
function match(x : string) : int
begin
var z : string; z := x; return / ("a" | "b") "c"; isEmpty / z;
end
ReSeq :
|[ /re1 re2 ; f/ x ]| -> |[ /re1 ; g/ x ]|
where new => g
; add-def(||[
function g(a : string) : int /re2; f/ a;
]|)
add-def(|def) =
rules( AddDef :+ |[ def* ]| -> |[ def def* ]| )
AddDef :
|[ def* ]| -> |[ function k(a : string) : int /"c"; isEmpty/ a; def* ]|
Dynamic rules
I rules( L : t1 -> t2 )
I defines new rewrite rule
I inherits variable bindings from definition context
Assimilating Regular Expression Matching
function match(x : string) : int
begin
var z : string; z := x; return / ("a" | "b") ; k / z;
end
ReSeq :
|[ /re1 re2 ; f/ x ]| -> |[ /re1 ; g/ x ]|
where new => g
; add-def(||[
function g(a : string) : int /re2; f/ a;
]|)
add-def(|def) =
rules( AddDef :+ |[ def* ]| -> |[ def def* ]| )
AddDef :
|[ def* ]| -> |[ function k(a : string) : int /"c"; isEmpty/ a; def* ]|
Assimilating Regular Expression Matching
function match(x : string) : int
begin
var z : string; z := x; return / ("a" | "b") ; k / z;
end
ReAlt :
|[ / (re1 | re2) ; f / x ]| -> |[ (/re1;f/x) | (/re2;f/x) ]|
Assimilating Regular Expression Matching
function match(x : string) : int
begin
var z : string; z := x; return (/ "a" ; k / z) | (/ "b" ; k / z);
end
ReAlt :
|[ / (re1 | re2) ; f / x ]| -> |[ (/re1;f/x) | (/re2;f/x) ]|
Assimilating Regular Expression Matching
function match(x : string) : int
begin
var z : string; z := x; return (/ "a" ; k / z) | (/ "b" ; k / z);
end
ReStr :
|[ /str ; f/ x ]| -> |[ isPref(str, x) & f(getSuf(str, x)) ]|
Assimilating Regular Expression Matching
function match(x : string) : int
begin
var z : string; z := x;
return (isPref("a",z) & k(getSuf("a",z)))
| (isPref("b",z) & k(getSuf("b",z)))
end
ReStr :
|[ /str ; f/ x ]| -> |[ isPref(str, x) & f(getSuf(str, x)) ]|
Assimilating Regular Expression Matching
function match(x : string) : int
begin
var z : string; z := x;
return (isPref("a",z) & k(getSuf("a",z)))
| (isPref("b",z) & k(getSuf("b",z)))
end
Desugar =
ReMch <+ ReStr <+ ReAlt <+ ReSeq <+ ReKle <+ once-AddDef
AddDef :
|[ def* ]| -> |[ function k(a : string) : int /"c"; isEmpty/ a; def* ]|
Assimilating Regular Expression Matching
function k(a : string) : int
begin
return /"c"; isEmpty/ a;
end
function match(x : string) : int
begin
var z : string; z := x
return (isPref("a",z) & k(getSuf("a",z)))
| (isPref("b",z) & k(getSuf("b",z)))
end
Desugar =
ReMch <+ ReStr <+ ReAlt <+ ReSeq <+ ReKle <+ once-AddDef
AddDef :
|[ def* ]| -> |[ function k(a : string) : int /"c"; isEmpty/ a; def* ]|
Assimilating Regular Expression Matching
function k(a : string) : int
begin
return /"c"; isEmpty/ a;
end
function match(x : string) : int
begin
var z : string; z := x
return (isPref("a",z) & k(getSuf("a",z)))
| (isPref("b",z) & k(getSuf("b",z)))
end
ReStr :
|[ /str ; f/ x ]| -> |[ isPref(str, x) & f(getSuf(str, x)) ]|
Assimilating Regular Expression Matching
function k(a : string) : int
begin
return isPref("c", a) & isEmpty(getSuf("c", a));
end
function match(x : string) : int
begin
var z : string; z := x
return (isPref("a",z) & k(getSuf("a",z)))
| (isPref("b",z) & k(getSuf("b",z)))
end
ReStr :
|[ /str ; f/ x ]| -> |[ isPref(str, x) & f(getSuf(str, x)) ]|
Risks of Domain-Specific Languages
• Design and implementation effort
• implementing complete toolset for language is major effort
• Documentation
• tutorials, reference manual, implementation documentation
• Training
• need programmers that are skilled in using the language
• Maintenance
• of the language• of DSL programs
• How does it fit in the existing work flow
• tool chain
• DSLs need abstraction/modularity features too
• DSL programs become large; reuse of parts needed
Conclusion
Summary
• Generative software development is encapsulation of domainknowledge in generators and target components
• Domain-specific languages are the interface to this knowledge
• DSL programs can be seen as a configuration for the ‘solutionspace’ components
Challenges
• Finding good formalizations and encapsulations of newapplication domains
• A platform for cheap construction of DSLs and generators
• Integration of generators in workflow/toolchain
References
• K. Czarnecki. Overview of Generative Software Development.In J.-P. Bantre et al. (Eds.): Unconventional ProgrammingParadigms (UPP) 2004, Mont Saint-Michel, France, LNCS 3566,pp. 313-328, 2005
• D. Batory. Feature Models, Grammars, and PropositionalFormulas. In H. Obbink and K. Pohl (Eds.) SPLC 2005. LNCS3714, pp. 7–20, 2005
• M. Bravenboer and E. Visser. Concrete Syntax for Objects.Domain-Specific Language Embedding and Assimilationwithout Restrictions. In D. C. Schmidt, editor, Proceedings of the19th ACM SIGPLAN Conference on Object-Oriented Programing,Systems, Languages, and Applications (OOPSLA’04), pages365–383, Vancouver, Canada, October 2004. ACM Press.
• E. Visser. Transformations for Abstractions. In J. Krinke and G.Antoniol, editors, Fifth IEEE International Workshop on SourceCode Analysis and Manipulation (SCAM’05), Budapest, Hungary,October 2005. IEEE Computer Society Press. (Keynote paper)
Further Studying
At Master Program Software Technology; Universiteit Utrecht
• Seminar on Software Generation and Configuration
• Master course on Program Transformation