Trilinos Lifecycle Model 2.0

Trilinos Lifecycle Model 2.0

Roscoe A. BartlettTrilinos Software Engineering Technologies and Integration Lead

Computer Science and Mathematics Div

Trilinos User Group Meeting, November 3, 2011

2 Managed by UT-Battellefor the U.S. Department of Energy Presentation_name

What is Trilinos?• Trilinos is a collection of unrelated unspecified software that share a common build,

test, and distribution system– Software quality, usability, etc. is purely the responsibility of individual package developers– Customers must evaluate each and every package on their own to determine quality, suitability, etc.– Customers must try to use individual packages together and negotiate with various package

development teams to resolve incompatibilities

Extremes • Trilinos is a collection of well-designed, reliable, interoperable, composable,

consistent, sustainable software components– Requires more coordination between Trilinos package developers– Requires the development of some standards and approaches to address them– Requires greater software quality practices and overhead

The stated goal for Trilinos is clearly the second, but we are really only doing a great job of the first right now.


Trilinos Lifecycle Model 2.0 Document

Table of Contents


Current State of Trilinos Software Engineering• Separation of Primary Stable (PS), Secondary Stable (SS), and Experimental (EX)

code for testing purposes

• Increasingly more extensive automated nightly regression testing and portability testing (on a growing list of platforms) of PS and SS code posting results to CDash

• Synchronous (pre-push) CI testing of PS code using the Python tool checkin-test.py

• Asynchronous (post-push) CI testing of PS and SS code using package-based CTest driver posting results to CDash

• Strong compiler warnings enabled with flags with g++ such -ansi -pedantic –Wall -Wshadow -Woverloaded-virtual

• Strong policies on the maintenance of 100% clean PS and SS builds and tests for all Nightly and CI builds

• Regular quarterly releases of Trilinos


PS and SS Code: Defined• Sub-categorizations of “stable” code:

– “Primary Stable” code is “Stable” code that only depends on:• C, and C++ compilers• Fortran 77 compiler (optional)• BLAS and LAPACK• MPI

– “Secondary Stable” code• Has additional dependencies such as:

– SWIG/Python (i.e. PyTrilinos)– Fortran 2003+ (i.e. ForTrilinos)– External direct sparse solvers like UMFPACK, SuperLU, etc. (i.e. Amesos adapters)

• Or, could be considered “Primary Stable” Code but is excluded from pre-checkin testing– Didasko– NewPackage– ...

• “Stable” code in one package can only depend on “Stable” code in other packages.

• “Stable” code should by default only build “Primary Stable” code.• Enabling “Secondary Stable” code should require extra configure-time options.


Weekly Coverage Testing

Trilinos Testing Infrastructure and Needed Improvements

Nightly Regression TestsSecondary Stable (SS)

CATEGORIES [BASIC NIGHTLY](more platforms, more TPLs)

Asynchronous CI TestsSecondary Stable (SS)

CATEGORIES [BASIC NIGHTLY](post-push CTest/Cdash, Linux/GCC)

Synchronous CI TestsPrimary Stable (PS)

CATEGORIES BASIC(pre-push

checkin-test.py)

Performance TestsSecondary Stable (SS)

CATEGORIES PERFORMANCE(CTest/CDash)

Scalability Tests?Secondary Stable (SS)

CATEGORIES SCALING???(CTest/CDash)

Weekly Memory (Valgrind) Testing

Cor

rect

ness

Tes

ting

• Primary Stable (PS), Secondary Stable (SS), and Experimental (EX) Code is about what gets tested.

• PS, SS, and EX does NOT imply maturity level!


Self-Sustaining Software: Defined• Open-source: The software has a sufficiently loose open-source license allowing the

source code to be arbitrarily modified and used and reused in a variety of contexts (including unrestricted usage in commercial codes).

• Core domain distillation document: The software is accompanied with a short focused high-level document describing the purpose of the software and its core domain model (see LIME Theory Document).

• Exceptionally well testing: The current functionality of the software and its behavior is rigorously defined and protected with strong automated unit and verification tests.

• Clean structure and code: The internal code structure and interfaces are clean and consistent.

• Properties apply recursively to upstream software: All of the external upstream software that are depended on are also themselves self-sustaining software.

• All properties are preserved under maintenance: All maintenance of the software maintains all of these properties of self-sustaining software (by applying Agile/Emergent Design and Continuous Refactoring and other good Lean/Agile software development practices).


Overview of New Maturity Levels


Goals for an updated Trilinos Life-cycle Model

• Allow pure research and applied research with a realistic path to productionization

• Provide smooth low-effort transitions from research to production in phases

• Provide maximum confidence with low cost (for research results and then for real users)

• Allow the use of Lean/Agile practices and processes along the way


Overview of the New Maturity Levels

1. Purely Experimental (PE) Code

2. Research Stable (RS) Code

3. Production Growth Stable (PGS) Code

4. Production Maintenance Stable (PMS) Code


PS and SS Tested Codevs.

RS, PGS, and PMS Maturity

PS

SS

EX

PE

Testing Categories:PS = Primary StableSS = Secondary StableEX = Experimental

Maturity Levels:PE = Purely Experimental RS = Research StablePGS = Production Growth Stable

RSPGSPMS


1: Purely Experimental Code• Generally not developed in a Lean/Agile consistent way from the very beginning,

does not provide sufficient unit (or otherwise) testing to prove correctness,

• Could actually be declared to be Secondary Stable code with respect to CI and nightly Trilinos testing but in general would be considered to be untested Experimental code in Trilinos

• Likely has a messy design and code base,

• No one should use such code for anything important (not even for research results but in the current CSE environment, publication of results using such software would likely still be allowed),

• Generally should not go out in general releases of Trilinos (but could go out in releases and is allowed by this lifecycle model), and

• Does not provide a direct foundation for creating production-quality code.


2: Research Stable Code• Developed in a Lean/Agile consistent way,

• Strong unit and verification tests (i.e. proof of correctness) written as the code/algorithms are being developed (should have almost 100% line coverage at the very least),

• Could be treated as Primary Stable or Secondary Stable code with respect to testing,

• Has a very clean design and code base maintained through Agile practices of emergent design and constant refactoring,

• Generally does not have higher quality documentation, user input checking and feedback, space/time performance, portability, or acceptance testing,

• Would tend to provide for some regulated backward compatibility but may not,

• Is appropriate to be used only by “expert” users,

• Is appropriate to be used only in “friendly” customer codes,

• Is appropriate to be part of a general release, and

• Provides a strong foundation for creating production-quality software.


3: Production Growth Stable Code

• Includes all the good qualities of Research Stable code,

• Provides increasingly improved checking of user input errors and better error reporting, has increasingly better formal documentation (Doxygen, technical reports, etc.) as well as better examples, tutorial material, etc.,

• Maintains clean structure through refactoring of the code and user interfaces to make more consistent, easier to maintain etc.,

• Maintains increasingly better regulated backward compatibility with few (if any) truly incompatible changes with new releases, as increasing better portability,

• Has increasing better space/time performance characteristics, and

• Has expanding usage in more customer codes.


4: Production Maintenance Stable Code

• Includes all the good qualities of Production Growth Stable code,

• Primary development includes mostly just bug fixes and performance tweaks,

• Maintains rigorous backward compatibility with typically no deprecated features or any breaks in backward compatibility, and

• Could be maintained by parts of the user community if necessary (i.e. as “self-sustaining software”).


Typical (non-Lean/Agile) Lifecycle

Research ProductionGrowth

ProductionMaintenance

Unit and Verification Testing



Acceptance Testing



Code and Design Clarity



Documentation and Tutorials



User Input Checking and Feedback



Backward compatibility



Portability



Space/Time Performance


Lean/Agile Consistent Lifecycle

ResearchStable

ProductionGrowthStable


Stable

Unit and Verification Testing

ResearchStable



Stable

Acceptance Testing

ResearchStable



Stable

Code and Design Clarity

ResearchStable



Stable

Documentation and Tutorials

ResearchStable



Stable

User Input Checking and Feedback

ResearchStable



Stable

Backward compatibility

ResearchStable



Stable

Portability

ResearchStable



Stable

Space/Time Performance


End of Life?

Long-term maintenance and end of life issues for Self-Sustaining Software:

• User community can help to maintain it

• If Trilinos project is disbanded, users can take parts of Trilinos they are using and maintain it long term

• Can stop being built and tested if not being currently used

• However, if needed again, software can be resurrected, and continue to be maintained

NOTE: Git actually greatly helps in reducing risk and sustaining long lifetime issues.


Regulated Backward Compatibility


Need for Backward Compatibility

Xyce J+1(released against

Trilinos X)

VTK M+1(released against

Trilinos X+1)

Multiple releases of Trilinos presents a possible problem with complex applications

Solution: => Provide sufficient backward compatibility of Trilinos X through Trilinos SIERRA Y+1

SIERRA Y+1(released against

Trilinos SIERRA Y+1)

TrilinosSIERRA

Y+1?


The Cost of Maintaining Backward Compatibility

• Static interfaces and design degrades as new unanticipated functionality is added– Leads to messy software that is hard to understand and dangerous to modify– Hacks to get around design problems make the software more fragile

• Backward compatibility must be tested– New tests must be written and maintained by manual labor– Backward compatibility tests must be built and run– Example: Change in [TEUCHOS_]TEST_FOR_EXCEPTION(…) macros.

Bottom Line => Maintaining backward compatibility increases “technical debt”


The Paradox of Backward Compatibility and Agile Development• Agile software development:

– Design changes as functionality is added and modified over multiple releases• Emergent Design: Design changes as the domain is better understood and

functionality is added.• Continuous Refactoring: Keep “conceptual integrity” with clean design and clean code.

– Requires close customer interaction and feedback to help drive software development

• However:– Many customers don’t want to use software while it is constantly changing

• The paradox:– Agile developed software must change as it is developed and must have real feedback

from customer use– Customers driving Agile developed software don’t want to be constantly chasing changes.

The solution => Regulated Backward Compatibility!


12.5 (Dev)

Regulated Backward Compatibility

11.0 12.0

11.2 11.4 11.6

• Trilinos Version Numbering X.Y.Z:• X: Defines backward compatibility set of releases• Y: Major release (off the master branch) number in backward compatible set• Z: Minor releases off the release branch X.Y• Y and Z: Even numbers = release, odd numbers = dev

• Makes logic with Trilinos_version.h easier• Special Exception: Let X+1.0 be the development version leading to X+1.0 release

• Allows X+1.0 to be the first major release as clear message to users• Backward comparability between releases

• Example: Trilinos11.6 is backward compatible with 11.0 through 11.4• Example: Trilinos 12.X is not compatible with Trilinos 11.Y

12.2 12.4

[Future] Test backward compatibility of Dev with current release every night!

12.0(Dev)

Drop backward compatibility of 12.0 with 11.Y! How to manage this?


Three Apps that Depend on Trilinos

App3

App2

Trilinos

App1


Releases with Three Apps and Trilinos

Trilinos

App1

App2

App3

Trilinos X

App1 K+1(Trilinos X+1)

Trilinos X+1

App2 M+1(App1 K+1,Trilinos X+2)

App1 K(Trilinos X)

App2 M(App1 K,Trilinos X)

Trilinos X+2

App3 J+1(App2 M+1,App1 K+1,Trilinos X+3)

Trilinos X+3

App3 J(App2 M,App1 K,Trilinos X)

Time


Regulated Backward Compatibility: Guidelines• Prepare users for the break in backward compatibility:

– Deprecate code to create compiler warnings in previous versions (see __deprecated__ attribute in GCC and Intel)

• Fail big and hard when backward compatibility is dropped:– Code should not even compile, and compile errors should be clear– Otherwise, code should not run at all and fail hard– Generate good compile or runtime error messages

• Provide for safe straightforward upgrades:– Provide sed-like scripts for making baulk changes?

• Example: Change all class and files names ‘VectorBase’ to ‘Vector’, etc.– Allow namespace changes?

• Example: TpetraNew::Vector => Tpetra::Vector for Map refactoring

Take Home Message: Plan for it and think about the user!


Deprecation Approaches• Deprecate classes and functions using the standard <UCPACAKGENAME>

DEPRECATED macro:This macro expands to the GCC deprecated attribute when Trilinos is configured with Trilinos SHOW DEPRECATED WARNINGS = ON.

• Deprecate macros by calling dummy deprecated functions using the standard <UCPACAKGENAME> DEPRECATED macro:

#define OLD_MACRO_NAME(…) { SomeDeprecatedFunction(); NEW_MACRO_NAME(…) }

• Deprecate old class names using deprecated typedefs or macro defines:Preferred (nontemplates):

typedef UCPACKAGENAME_DEPRECATED NewClassName OldClassName;Templated classes: #define OldClassName NewClassName;

• Must also deprecate the header file with #warning (see below)

• Deprecate header files by inserting protected #warning directives:#ifdef __GNUC__# waning This header OldHeader.hpp is deprecated. Please use NewHeader.hpp#endif


Other Considerations


Risk Analysis and Acceptance Testing• NOTE:

– Trilinos provides tools for creating end-user applications but has no end-user applications– Risk analysis must be driven by specific end user use cases

• Customer responsibility:– Do their own risk analysis– Communicate their risks down to Trilinos developers as requirements (through Agile

feedback)

• Trilinos Developers responsibility:– Write good algorithms with good tests, good user input checking, etc.– Listen to customers through Agile feedback


Grandfathering of Existing Packages• ???


Next Steps?• How to identify and enable software based on maturity level:

– Allow users to set the minimum maturity level:• -D Trilinos_ENABLE_MINIMUM_MATURITY_LEVEL=PRODUCTION_MAINTENANCE_STABLE• (or PRODUCTION_GROWTH_STABLE or RESEARCH_STABLE)

– Assign a maturity level to each package in TrilinosPackages.cmake– Assign a maturity level to each subpakage in a package (High maturity level package can

have some lower maturity level subpackages)– Allow code to be otherwise ifdefed etc. to control enables based on

Trilinos_ENABLE_MINIMUM_MATURITY_LEVEL.

• How to assess maturity levels?– Define standards and metrics for various areas to help define maturity levels?– Set up process to review packages and assign maturity levels?– Let package teams pick their own maturity levels?

• Other issues:– How do we deal with existing legacy packages? => All modifications should be done

according to Legacy Software approach (1. cover with tests, 2. add functionality, 3. refactor for simplicity)

Trilinos Lifecycle Model 2.0

Documents

Transcript of Trilinos Lifecycle Model 2.0