Domain-Specific Languagesin Automated Testing Tools

Mickey PhoenixSenior Software Engineer

Solutions IQSeattle, Washington

mphoenix@solutionsiq.com

Rand HusoSenior Software Engineer

Solutions IQSeattle, Washington

rhuso@solutionsiq.com

Thursday March 6, 20088:30 - 10:00 am

Presenters:

What is a Domain-Specific Language?

• Concise and Expressive• Specific • Consistent in Granularity

Concise and Expressive• Written in the user's language

• Relative to the level the user is thinking at

• Thorough coverage at that level

Specific

• Domain-specific; often, project-specific

• Portability and re-usability are not goals

• Customized for each client, project, and domain

Consistent Granularity

• Assists logical decomposition• Avoids getting lost in the details• Matches the way people give instructions to each

other– “How do you brush your teeth?”

. . . Oooops.

Customers think at multiple levels at different times, so a single Domain-Specific Language

will not suffice!

But…

Why do we want multiple levels of DSL?

• Test at multiple levels

• Avoid hiding too much information inside DSL commands

• Compose higher-level DSL commands from lower-level DSL commands

– not from the native commands of the testing tool

• Keep the core testing tool simple at alllevels

Selling Domain-Specific Languages

• Up-front costs to the customer

• Additional effort by the developer

How do we get Customer and Developer Buy-In?

Why Customers Love DSLs

• Customers can participate fully

• Customers can write tests in their own language (no lossy translations)

• Customers can validate tests just by reading them -- “executable documentation”

Customers Are Full Participants• Customers pair with developers to write

tests• Customers understand developer-written

tests just by reading them

Tests in Customer Language

Translations are lossy:

We want as few translations aspossible between the customer's"language of thought" and the test.

“The vodka is good, but the meat is spoiled.”

“The spirit is willing, but the flesh is weak.”

Tests As Executable Documentation

• The goal of Agile testing is "executable documentation“

• A test written in a DSL is like self-documenting code

– It requires minimal comments, because the test is written in a language the users can read

Why Developers Love DSLs

• Easier and faster to write at appropriate granularity

• Abstraction reduces test maintenance cost

• Keeps the core testing tool simple

Appropriate GranularityMakes Tests:

• Faster to Write – fewer, bigger statements

• Easier to Read – comprehension by “chunking”

• Easier to Verify – don’t get lost in the details

• Easier to Maintain – all of the above

Think OOD

AbstractionReduces Maintenance Cost

• Tests must change with changes to the spec

• Abstracting out common actions keeps changes to a single point

– “once and only once”

Core Testing Tool Stays Simple

• Every client and project will have some unique needs

• Easy per-client and per-project extensibility keeps changes local to those who need them

Extensibility minimizes feature creepin the core tool

What Do We Expect of an Automated Testing Tool?

• Presupposed: a testing tool which has a written "script" – as opposed to a keystroke/click/etc. recorder

• Provides extension mechanisms – allows us to define new statements in the DSL(s) that can

then be used in the test scripts

• Provides hierarchical abstraction mechanisms – the ability to define test script "subroutines" which

can be invoked by name

Extension Mechanisms

• Used by the developer– Need not be accessible to the customer

• Allow multiple implementation languages where practical (plugins, etc.)

• Support several granularities of DSL

• Can be customized and overridden at the client and project levels independently

Hierarchical Abstraction

WALK:Step forward with right foot

Step forward with left foot

STEP FORWARD:Shift weight to opposing foot

Bend knee

Swing foot forward

Set foot down in advance of body’scenter of mass

DSL Examples

• A low-level test written without DSL• The same low-level test written in a DSL• The definition of the low-level DSL• A mid-level test written in a low-level DSL• The same mid-level test written in a mid-level

DSL

We will review:

Low-Level Test (no DSL)

falsedocument.getElementById("field_one_mandatory_flag").style.visibility

assertJavascriptValue

truedocument.getElementById("field_one").style.editable

assertJavascriptValue

truedocument.getElementById("field_one").style.visibility

assertJavascriptValue

Assert that field_one is “enabled" (visible and editable, but not mandatory)

comment

Low-Level Test (with DSL)

field_oneassertEnabled

Custom DSL Definition(Low-Level)

Selenium.doAssertEnabled = new function(fieldName){ var field = document.getElementById(fieldName); var mandatoryFlag = document.getElementById(

fieldName + "_mandatory_flag"); if (!field.style.visibility){

throw new Error("Field is not visible."); } if (!field.style.editable) {

throw new Error("Field is not editable."); } if (mandatoryFlag.style.visibility){

throw new Error("Field is mandatory."); }}

[no body text for thisentry]

assertTextPresent

link=This is a test entry subject

clickAndWait

link=This is a test entry subject

assertElementPresent

This is a test entry subjectassertTextPresent

Link=List EntriesclickAndWait

Mid-Level Test (Low-Level DSL)

This is a test entry subjecttype

link=Create New EntryclickAndWait

button=LoginclickAndWait

passwordtestPasswdtype

usernamebsmithtype

button=Create EntryclickAndWait

Mid-Level Test (Mid-Level DSL)

!include LoginStandardUser

!include CreateEntry !include VerifyEmptyEntry

entrySubjectThis is a test entry subject

store

Example Extensibility Framework• STIQ – StoryTestIQ

– Free open-source project at SourceForge (http://storytestiq.sourceforge.net/)

• Extensibility mechanisms evolved along with the tool, in response to need

• Framework-level and client-level extensibility contributed the most to power

• Project-level and hierarchicalextensibility contributed the most to readability

Framework-Level Extensibility

• Java code changes on the server (e.g. SQL support)

– Requires re-compilation of the local (STIQ) project 

– Written in Java (may not be as familiar to developers in a .NET shop)

Client-Specific Extensibility

• client-extensions.js Allows corporate-level testing support people to add features that are useful for all projects

– Requires re-compilation of the local (STIQ) project  

– Written in Javascript (familiar to all web developers)

Project-Specific Extensibility

• project-extensions.jsAllows individual projects to have added features (in the browser) that will be useful only for their project– Requires reloading the browser page, but

does not require re-compiling– Written in Javascript (familiar to all web

developers)

Hierarchical Extensibility

• !include syntax

allows hierarchical decomposition of tests, including tests written in low-level DSL terms defined in: – client-extensions.js source– project-extensions.js source

Conclusion

• Providing DSL support in your automated testing tool has many advantages – to you– to the project developers and testers– to the user

• ...?

• Profit!

Questions?

Small-Group Discussions

Discussion Questions

• One implementation language or many implementation languages? – You can write Java code on the server, or

Javascript code or SQL on the client, to augment the tests (in STIQ). Should a testing tool provide customization support in multiple implementation languages?

Discussion Questions

• Are some implementation languages more suited to certain levels of granularity than others? – Will some levels of DSL be easier to

implement/customize in low-level vs. high-level languages?  Which languages are most suitable to implement which levels of DSL, and why?

Discussion Questions

• Should a customizable testing tool define its own, custom implementation language, or should it use a standard language like Java or Javascript?

Discussion Questions

• Which agents should be able to customize?  Testers?  Developers? Only the team tasked with supporting the testing tool?

Discussion Questions

• How do DSLs affect customer transparency? – A single-line DSL statement makes the test more

readable, but also obscures the actions taking place within that single-line command, which may potentially be quite complex. Doesn't this just bring us back to the same old problem of the customer having to "take it on faith" that the test asserts the behavior they desire?

Discussion Time

Working Group Presentations

Authors:

Michele CoxRand HusoMichael Phoenix