PowerPoint Presentation

Twitter #NIEMNTE1

Agile Standards Development:Lessons Learned from STAR and Open Source Development

David CarverPrincipal Software Architectdcarver@intalio.com

Twitter #NIEMNTE1

Principal Software Engineer at Intalio.

Open Source Projects Eclipse Web Tools Incubator (Project Lead)

Eclipse XSL Tools and Psycho Path Xpath 2 processor

Eclipse Architecture Council and EclipseCon Program Committee

eBay Open Source Turmeric SOA (Project Lead)

Twitter kingargyle

Blog: http://intellectualcramps.wordpress.com

Linked In, Google+

IRC chat.freenode.net, #turmeric-dev (Turmeric SOA)

STAR, XML Data Architect for 5 years.

Who is Dave?

Twitter #NIEMNTE1

Changing Requirements Requirements change Monthly Weekly Daily Users of the Data Specifications need thestandards sooner Data Specifications typically released every 2years or longer Users requirements have changed. Government regulations have changed.

Data Standards Development Challenges

Twitter #NIEMNTE1

Manual Processes Schema are created manually one timebefore release. Manual Verification of user requirements Manual Code Merges. No Source CodeControl besides Zipping the repository. Documentation Maintained outside the schema Not kept in sync with the schema Available in limited formats or proprietaryFormats. Testing/Validation is an after thought.

Data Standards Development Challenges

Twitter #NIEMNTE1

Twitter #NIEMNTE1

Waterfall Development Process

Nouve

Twitter #NIEMNTE1

Learning from Open Source Projects

Twitter #NIEMNTE1

Eclipse Foundation (www.eclipse.org) Hosts an Annual Release Train

62 Projects are all released simultaneously, developed on their own timelines and own processes.

All are guaranteed to work together (i.e. no conflicts).

Milestone builds every 6 weeks

Continuous Integration

Eclipse has over 200 open source projects.

Google Android Platform (source.android.com) Updated every 6 months

Internal development releases to partners.

Open Sourced code after internal development is done.

eBay Open Source (www.ebayopensource.org) Home to Turmeric SOA and VJET

Turmeric SOA follows Scrum development methodology. 2 week sprints, nightly snapshots published, continuous integration builds.

Twitter #NIEMNTE1

Twitter #NIEMNTE1

Twitter #NIEMNTE1

30+ projects

Projects are all released together, but have their own development times frames.

Twitter #NIEMNTE1

IEPD should Include Iterations and Constant Communication

Twitter #NIEMNTE1

Twitter #NIEMNTE1

30 Members

Founded in 2000

Has 150 + messages, covering 45 business processes.

Adopted an Agile approach in 2005.

Has an annual public release every year.

Has 4 draft releases to members only.

Maintained until 2010 3 different standards. STAR 4, STAR 5 and DTS.

STAR 5 is based on OAGIs 9.x

Follows UNCEFACT Core Components Model and Naming and Design Rules.

About STAR

Twitter #NIEMNTE1

3

4

5

1

2

Publication and Release

Test, Build and Validate

Iterations

Initial Requirements

Draft 1

Draft 2

Draft 3

Draft 1

Major Non Compatible Development

Internal Build Label

Agile Development Process, Based on short iterations including tests, development, builds, and validation steps.

BaselineRepository

Twitter #NIEMNTE1

Publishing: 1 hour.

All manual steps automated

Plus new steps:

Unit Tests Regression Testing (Run all Unit Tests)

2300+ Unit Tests

Merging Business Rules

Generating Guidelines

PDF

HTML

XSL FO

RTF

Zipping Guidelines.

Twitter #NIEMNTE1

Publishing: 1 -2 hours.

All manual steps automated

Assembling Schema

Unit Tests Regression Testing (Run all Unit Tests) 6500+ tests

Generating Guidelines. PDF, HTML, XSL FO, RTF

Zipping Guidelines

This includes over 50 guidelines, covering 150+ use cases.

Process scaled to maintaining two different Versions of the repository. STAR 4 and STAR 5.

Full Process is repeatable. Runs every night.

Twitter #NIEMNTE1

Unit Testing Test Driven Development - XML Schema: Elements, ComplexTypes, SimpleTypes, Code Lists, Naming and Design Rules Continuous Integration Source Code Control All Development checked in daily- such as Jenkins, Hudson, Luntbuild, CruiseControl XSDs, Automated Builds, Sample XML, Unit Tests

Required Agile Practices

Twitter #NIEMNTE1

Iterative Development Iterations 2 Weeks in Length. Requirements, Unit Tests, Schema Development Publication ready build at end of iteration. Automated Builds Automate everything! Builds, Guidelines, Testing. Involve the Stakeholders Those requesting the requirements need to be involved throughout the process, including Developers working on Implementations. Open and Honest communication is a must.

Required Agile Practices

Twitter #NIEMNTE1

More accurate and timely representation of the requirements

Reduced overall development time.

Improved compliance to the specifications and NDR as tests are included to meet compliance as part of development.

Verification happens more often

Less Manual/Human intervention, reducing overall process mistakes.

Greater interoperability and backwards compatibility.

Unit Tests catch errors early instead of months later.

Benefits

Twitter #NIEMNTE1

Don't try to get the PERFECT Model. Get one that is Good Enough.

Unit Test..Unit Test...Unit Test

Catches Errors Early Provides a Regression Test Suite Run as often as possible Catches changes that break compatibility Automate Everything Possible and have Source Code Control Conduct Nightly, Milestone, and Release Builds Agile Development is not a Silver Bullet. There is no one RIGHT way to do it. Adopters LOVE getting their requirements sooner rather than later.

Lessons Learned from STAR's implementation.