No Respect

Post on 05-Jan-2016

66 views 4 download

description

Danny, Be the Ball. No Respect. Dart2 Quality Framework. Dan Blezek Jim Miller Bill Lorensen. Software Quality: Past, Present, Future. A brief history of quality. Very First VTK Dashboard Update information Builds – Irix, Solaris, WinNT Regression tests WinNT “Catastropic failure”. - PowerPoint PPT Presentation

Transcript of No Respect

1National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

No Respect

Danny, Be the BallDanny, Be the Ball

NA-MICNational Alliance for Medical Image Computing http://na-mic.org

Dart2 Quality FrameworkSoftware Quality: Past, Present, FutureDan BlezekJim MillerBill Lorensen

3National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

A brief history of quality

Very First VTK Dashboard• Update information• Builds – Irix, Solaris, WinNT• Regression tests

• WinNT “Catastropic failure”

4National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Last VTK Dashboard• 8 Platforms• 650 Nightly Tests•70% coverage (Still) • Nightly Purify

5National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

A Bad Day

6National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

The Continuous Build

7National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

My first VTK checkin

Continuous UpdateContinuous Update Continuous BuildContinuous Build

8National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

The Big Bat of Quality

9National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Bill “Yogi” Lorensen

10National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Lessons learned

• Test on different platforms

• Test nightly

• Make it easy to add a test

• Track changes daily

• Keep historical information

NA-MICNational Alliance for Medical Image Computing http://na-mic.org

Two roads diverged in a wood, and I--I took the one less traveled by,And that has made all the difference.Robert Frost

Closed Source

12National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Dart (v1)

• Tests• Reports• Dashboards• Captures state of the system• Distills data into information

– Convert build log errors/warnings– Summarize test execution– Rollup Coverage statistics

13National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

14National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

15National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Someone Someone broke the broke the build!build!

16National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Someone broke the build!

17National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

18National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Dart’s Power

• Distributed testing

• If I don’t have a platform, you do

• Distill data from many tools

• Distributed Extreme Programming

• Know the state of the system

• Instant feedback on changes

19National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Quality Statistics

Original VTK dashboard• 8 platforms / 650 tests• 13.6 G over 4 yearsCurrent VTK dashboard• 29 Nightly platforms / 500 tests• 1-2 G / weekInsight dashboard• 60 Nightly builds / 970 tests• 1-2 G / week

20National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Dart2 Design Goals

• One Server, multiple Projects• Simple, flexible setup and

management• Configurable presentation• Persist data on dashboard over time• Aggregate Dashboards• Authenticated submission, if desired• Resource management tools

21National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Implementation

• Java– Many, many available packages– Cross-platform

• Everything in one package– No extra OS packages required– Distribute as Jar and/or platform exe

• Should be easily extensible– Even to non-Java programmers

22National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Packages

23National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Components/Concepts

• Client, Submission• Test Hierarchy• Results• RDBMS• XML-RPC Server• Task Manager• Scheduler• Tracks• HTTP / Template Engine

24National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Client, Submission

• Client: a unique platform– Need to define criteria– Currently Site / BuildName

• Submission– One TimeStamped set of Test data– Particular to a sub-Project

• slicer.itk• slicer.vtk

25National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Test Hierarchy

• Test is a logic group of Results

• Has a Pass/Fail/NotRun status

• May contain other Tests

• Has Hierarchial naming convention– itk.common.PrintSelf

• SubTest information rolled up

26National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Results

• Data produced by a Test• Examples:

– Log of standard out– Image– ExecutionTime

• Typed– text/string, text/url, text/xml, text/text– numeric/integer, numeric/double– image/png,image/jpeg

27National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

RDBMS

• Core of Dart2– Bundled with Derby embedded RDBMS– Any JDBC compliant DB works

• Stores “small” data– Images, large blocks of text in files

• Jaxor– Object – Relational Bridge package– No fancy SQL required– Creates objects from rows in DB

28National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

XML-RPC Server

• Accepts Submissions

• Administrative functions

• HTTP transport– Easy submission through firewalls

• Digester used to process XML– Executes code when tags found

29National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Task Manager

• Tasks are units of work for the server– Project and Server Tasks– Scheduled, Event driven

• When a Submission arrives, a Task is queued

• QueueManager executes Tasks

• Plug-ins allow Project specific Tasks– Simply implement the Task Interface

30National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Scheduler

• Quartz Enterprise Scheduler

• Executes Tasks– Uses enhanced “cron” syntax

• Uses– Regular DB maintained– Purge unnecessary data– Archive aging data

31National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Tracks

• Groups of Submissions– Dashboard consists of intersecting Tracks

• Temporal Tracks– Time based, i.e. 12am start, 24hr duration

• “Most Recent” Track– Last 5 Continuous builds

• Project specific

32National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

HTTP / Template Engine

• Jetty is HTTP/Servlet server• FreeMarker

– Data prepared in Servelet– Template processed– Returned to client via HTTP

• Flexible• Easy to add new pages• No XSLT!

33National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Dart2 Current Status

• Alpha version ready

• Test server– http://www.na-mic.org:8081/Insight/Dashboard/– Populated with Build & Test from public.kitware.com

• Subversion Code Repository– svn co http://svn.na-mic.org:8000/svn/Dart– Web SVN: http://www.na-mic.org:8000/websvn/

34National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Acknowledgements

• Andy Cedilnik

• Bill Hoffman

• Will Schroeder

• Ken Martin

• Amitha Perera

• Fred Wheeler

NA-MICNational Alliance for Medical Image Computing http://na-mic.org

Dart2 Quality FrameworkSoftware Quality: Past, Present, FutureDan BlezekJim MillerBill Lorensen

36National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Desirable Qualities

• Frequent testing– Identify defects as soon as they are introduced – Hard to find cause if not done frequently

• Minimally invasive to daily activities• Automated testing• Automated report generation/summaries

– Must be concise yet informative

• Track results over time

NA-MICNational Alliance for Medical Image Computing http://na-mic.org

NAMIC Software Process

Dan BlezekJim MillerBill Lorensen

38National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Motivation

• Many algorithms, many platforms– VTK, ITK, Slicer, LONI– Linux, Windows, Mac OSX, Solaris, SGI

• Many users

• Many datasets

• Many sources of problems!

39National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Motivation

Negative example

• MIT codes ITK algorithm for LONI pipeline

• UCLA developer changes LONI

• GE changes ITK

• Time for release, everything’s broken

40National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Motivation

Ensuring high quality software

• System’s state must be known– If UCLA knew about MIT code, they

would have been more careful w/changes

• All the code works, all the time– As often as is feasible, compile and test

the code

41National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Extreme Programming

42National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

NAMIC Process

• Light weight• Based on Extreme Programming

– High intensity cycle• Design• Test• Implement

– Supported with web-enabled tools– Automated testing integrated with the

software development

43National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Software Process

• Design Process• Coding Standards• Testing• Bug Tracker• Communication

– Mailing lists, Discussion forum, Wiki– Tcons

• Documentation• Releases

44National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Design Process

• Take the time to design a good API

• Plan for future use

• Plan for future extension

• Two routes– Code something, check it in

• Others will tear it down & make it better

– Put together a strawman• Solicit ideas, implement

45National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Coding Standards

• Follow the package’s rules

• ITK has certain coding standards– Style guidelines– Naming conventions– Macros

46National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Testing

• If it isn’t tested, it’s broken.

• Tests– Ensure your code works– Documents expected results– Others free to change

47National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Bug Tracker

• Bugs assigned / taken by developers

• Tracks progress to releases

• Captures feature requests

• Communication mechanism

48National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Documentation

• Doxygen– Automatic API documentation– Algorithm references– Implementation details

• Books / Manuals– Insight Book

49National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Communication

• Email lists

• Discussion forum

• Wiki

• Tcon

50National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Extreme Programming

• Compression of standard analyze, design, implement, test cycle into a continuous process

51National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Daily Testing Is The Key

• Testing anchors the development process (Dart)

• Developers monitor the testing dashboard constantly

• Problems are identified and fixed immediately

• Developers receive e-mail if they“Break the Build”

52National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Daily rhythmDesign, implement algorithm• write regression test• check it in

Dart takes over• Code is updated, built, tested• Guilty parties are blamed

Core 2• Build breakers are notified• Work on cleaning up dashboard

Repeat

53National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Dart

• Testing• Reports• Dashboards• Central site for state of the system

– Updates– Builds– Test– Coverage

54National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

55National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

56National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Someone Someone broke the broke the build!build!

57National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

58National Alliance for Medical Image Computing http://na-mic.org

Salt Lake City AHMFebruary, 2005

Conclusion

• Have fun

• Process extends your impact– Many can use your code– Many can improve your code

• Communicate, Communicate, Communicate

NA-MICNational Alliance for Medical Image Computing http://na-mic.org

NAMIC Software Process

Dan BlezekJim MillerBill Lorensen