No Respect

1National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

No Respect

Danny, Be the BallDanny, Be the Ball

NA-MICNational Alliance for Medical Image Computing http://na-mic.org

Dart2 Quality FrameworkSoftware Quality: Past, Present, FutureDan BlezekJim MillerBill Lorensen



A brief history of quality

Very First VTK Dashboard• Update information• Builds – Irix, Solaris, WinNT• Regression tests

• WinNT “Catastropic failure”



Last VTK Dashboard• 8 Platforms• 650 Nightly Tests•70% coverage (Still) • Nightly Purify



A Bad Day



The Continuous Build



My first VTK checkin

Continuous UpdateContinuous Update Continuous BuildContinuous Build



The Big Bat of Quality



Bill “Yogi” Lorensen



Lessons learned

• Test on different platforms

• Test nightly

• Make it easy to add a test

• Track changes daily

• Keep historical information


Two roads diverged in a wood, and I--I took the one less traveled by,And that has made all the difference.Robert Frost

Closed Source



Dart (v1)

• Tests• Reports• Dashboards• Captures state of the system• Distills data into information

– Convert build log errors/warnings– Summarize test execution– Rollup Coverage statistics



Someone Someone broke the broke the build!build!



Someone broke the build!



Dart’s Power

• Distributed testing

• If I don’t have a platform, you do

• Distill data from many tools

• Distributed Extreme Programming

• Know the state of the system

• Instant feedback on changes



Quality Statistics

Original VTK dashboard• 8 platforms / 650 tests• 13.6 G over 4 yearsCurrent VTK dashboard• 29 Nightly platforms / 500 tests• 1-2 G / weekInsight dashboard• 60 Nightly builds / 970 tests• 1-2 G / week



Dart2 Design Goals

• One Server, multiple Projects• Simple, flexible setup and

management• Configurable presentation• Persist data on dashboard over time• Aggregate Dashboards• Authenticated submission, if desired• Resource management tools



Implementation

• Java– Many, many available packages– Cross-platform

• Everything in one package– No extra OS packages required– Distribute as Jar and/or platform exe

• Should be easily extensible– Even to non-Java programmers



Packages



Components/Concepts

• Client, Submission• Test Hierarchy• Results• RDBMS• XML-RPC Server• Task Manager• Scheduler• Tracks• HTTP / Template Engine



Client, Submission

• Client: a unique platform– Need to define criteria– Currently Site / BuildName

• Submission– One TimeStamped set of Test data– Particular to a sub-Project

• slicer.itk• slicer.vtk



Test Hierarchy

• Test is a logic group of Results

• Has a Pass/Fail/NotRun status

• May contain other Tests

• Has Hierarchial naming convention– itk.common.PrintSelf

• SubTest information rolled up



Results

• Data produced by a Test• Examples:

– Log of standard out– Image– ExecutionTime

• Typed– text/string, text/url, text/xml, text/text– numeric/integer, numeric/double– image/png,image/jpeg



RDBMS

• Core of Dart2– Bundled with Derby embedded RDBMS– Any JDBC compliant DB works

• Stores “small” data– Images, large blocks of text in files

• Jaxor– Object – Relational Bridge package– No fancy SQL required– Creates objects from rows in DB



XML-RPC Server

• Accepts Submissions

• Administrative functions

• HTTP transport– Easy submission through firewalls

• Digester used to process XML– Executes code when tags found



Task Manager

• Tasks are units of work for the server– Project and Server Tasks– Scheduled, Event driven

• When a Submission arrives, a Task is queued

• QueueManager executes Tasks

• Plug-ins allow Project specific Tasks– Simply implement the Task Interface



Scheduler

• Quartz Enterprise Scheduler

• Executes Tasks– Uses enhanced “cron” syntax

• Uses– Regular DB maintained– Purge unnecessary data– Archive aging data



Tracks

• Groups of Submissions– Dashboard consists of intersecting Tracks

• Temporal Tracks– Time based, i.e. 12am start, 24hr duration

• “Most Recent” Track– Last 5 Continuous builds

• Project specific



HTTP / Template Engine

• Jetty is HTTP/Servlet server• FreeMarker

– Data prepared in Servelet– Template processed– Returned to client via HTTP

• Flexible• Easy to add new pages• No XSLT!



Dart2 Current Status

• Alpha version ready

• Test server– http://www.na-mic.org:8081/Insight/Dashboard/– Populated with Build & Test from public.kitware.com

• Subversion Code Repository– svn co http://svn.na-mic.org:8000/svn/Dart– Web SVN: http://www.na-mic.org:8000/websvn/



Acknowledgements

• Andy Cedilnik

• Bill Hoffman

• Will Schroeder

• Ken Martin

• Amitha Perera

• Fred Wheeler


Dart2 Quality FrameworkSoftware Quality: Past, Present, FutureDan BlezekJim MillerBill Lorensen



Desirable Qualities

• Frequent testing– Identify defects as soon as they are introduced – Hard to find cause if not done frequently

• Minimally invasive to daily activities• Automated testing• Automated report generation/summaries

– Must be concise yet informative

• Track results over time


NAMIC Software Process

Dan BlezekJim MillerBill Lorensen



Motivation

• Many algorithms, many platforms– VTK, ITK, Slicer, LONI– Linux, Windows, Mac OSX, Solaris, SGI

• Many users

• Many datasets

• Many sources of problems!



Motivation

Negative example

• MIT codes ITK algorithm for LONI pipeline

• UCLA developer changes LONI

• GE changes ITK

• Time for release, everything’s broken



Motivation

Ensuring high quality software

• System’s state must be known– If UCLA knew about MIT code, they

would have been more careful w/changes

• All the code works, all the time– As often as is feasible, compile and test

the code



Extreme Programming



NAMIC Process

• Light weight• Based on Extreme Programming

– High intensity cycle• Design• Test• Implement

– Supported with web-enabled tools– Automated testing integrated with the

software development



Software Process

• Design Process• Coding Standards• Testing• Bug Tracker• Communication

– Mailing lists, Discussion forum, Wiki– Tcons

• Documentation• Releases



Design Process

• Take the time to design a good API

• Plan for future use

• Plan for future extension

• Two routes– Code something, check it in

• Others will tear it down & make it better

– Put together a strawman• Solicit ideas, implement



Coding Standards

• Follow the package’s rules

• ITK has certain coding standards– Style guidelines– Naming conventions– Macros



Testing

• If it isn’t tested, it’s broken.

• Tests– Ensure your code works– Documents expected results– Others free to change



Bug Tracker

• Bugs assigned / taken by developers

• Tracks progress to releases

• Captures feature requests

• Communication mechanism



Documentation

• Doxygen– Automatic API documentation– Algorithm references– Implementation details

• Books / Manuals– Insight Book



Communication

• Email lists

• Discussion forum

• Wiki

• Tcon



Extreme Programming

• Compression of standard analyze, design, implement, test cycle into a continuous process



Daily Testing Is The Key

• Testing anchors the development process (Dart)

• Developers monitor the testing dashboard constantly

• Problems are identified and fixed immediately

• Developers receive e-mail if they“Break the Build”



Daily rhythmDesign, implement algorithm• write regression test• check it in

Dart takes over• Code is updated, built, tested• Guilty parties are blamed

Core 2• Build breakers are notified• Work on cleaning up dashboard

Repeat



Dart

• Testing• Reports• Dashboards• Central site for state of the system

– Updates– Builds– Test– Coverage



Someone Someone broke the broke the build!build!



Conclusion

• Have fun

• Process extends your impact– Many can use your code– Many can improve your code

• Communicate, Communicate, Communicate


NAMIC Software Process

Dan BlezekJim MillerBill Lorensen

No Respect

Documents

Transcript of No Respect