Design Driven Cost Estimation via Design Driven Testing Doug Rosenberg Copyright © 2011 ICONIX.
-
Upload
samara-bramble -
Category
Documents
-
view
217 -
download
1
Transcript of Design Driven Cost Estimation via Design Driven Testing Doug Rosenberg Copyright © 2011 ICONIX.
Design Driven Cost Estimationvia
Design Driven Testing
Doug Rosenberg
Copyright © 2011 ICONIX
“How can we count lines of code before coding?” asked Alice
Copyright © 2011 ICONIX
CoCoMo meets DDT
Copyright © 2011 ICONIX
1987 2011
ICONIX was heavily involved with CoCoMo from the mid-
80s to the mid-90s
• We sold a CoCoMo tool for about 10 years• What did we learn?• A) Estimation is a lot easier when you
know how many lines of code you will be building
• B) It isn’t always easy to know how many LOC you’ll have up front
• Our CoCoPro tool calibrated estimations to a history file of past projects, to converge to better estimation results over time
Copyright © 2011 ICONIX
We know how to count code after it’s written
Copyright © 2011 ICONIX
• Q: how can we get a line of code count before coding? • A: Count lines of unit test code, generated from model
Cost estimation via testing?Isn’t that backwards???
• In today’s “agile” universe, tests are sometimes written before code (Test Driven Development)
• With Design Driven Testing we generate unit test code from UML/SysML models before code is written
• We can line-count this unit test code and use it as a basis for estimation
Copyright © 2011 ICONIX
Copyright © 2011 ICONIX
DDT flips “agile” test driven development around
• That’s good because driving design from unit tests is 180 degrees backwards
• It’s better to drive testing from design
• Cost estimation can be driven in the same way
Copyright © 2011 ICONIX
DDT addresses unit testing by developers and acceptance
testing by QA folks
• We’ll focus on unit testing in this talk
• DDT automates generation of skeleton unit test code from UML models
• This unit test code can be line-counted and used as a basis for LOC estimates
Copyright © 2011 ICONIX
Unit tests are auto-generated from sequence diagrams
• This is good because we can count lines of unit test code and use it as a basis for estimation, before code is written
• How does it work?
Copyright © 2011 ICONIX
1. Generate a test case for each message on a sequence diagram
Copyright © 2011 ICONIX
2. Define test scenarios for each test case
Test with good data, test with bad data, etc.
Define as many scenarios as are needed to fully exercise the operation being tested
More complex operations will have more test scenarios
Copyright © 2011 ICONIX
3. Transform test cases to unit test classes
• A test method is created for each test scenario
• These test methods can be code generated
Copyright © 2011 ICONIX
Transforms are instantaneous
• setUp and tearDown methods are also generated, in a addition to a method per test scenario
• Automation is key to the whole DDT approach because it’s too much work if it’s not automated
Copyright © 2011 ICONIX
4. Test class -> test code
• Unit test code is generated automatically
• The amount of code is proportional to the complexity of the operation being tested
• This unit test code can be line-counted and used as a basis for COCOMO estimation
• NOTE: Not a 1:1 ratio between unit test code and production code
Copyright © 2011 ICONIX
We have an early-lifecycle line of code count (before coding begins)
This line of code count is directly generated from the UML or SysML model, and is directly proportional to the complexity of the code that will be written, because more complex methods will have more test scenarios
No handwaving
Copyright © 2011 ICONIX
Oh, BTW…No gaps in test coverage
• Minor side benefits of DDT include better designs and better test coverage
Copyright © 2011 ICONIX
What if we could start from an earlier lifecycle artifact than
sequence diagrams?
• Sequence diagrams represent a very detailed level of design.
• We probably need to estimate earlier in the lifecycle.
• We can generate test code from conceptual designs, as soon as use cases are defined
Copyright © 2011 ICONIX
Controllers are testable units of system behavior
• We identify logical software functions on robustness diagrams as a disambiguation device when writing use cases
• Each controller may expand to several messages on a sequence diagram
• Fewer tests are required• Controller tests can be
code generated just like sequence message tests
Copyright © 2011 ICONIX
Breadth-first test coverage across a scenario
• And an earlier-lifecycle artifact to drive test code from
• This can provide a “first guess” estimate
• Re-estimation during detailed design will provide more accurate estimates
Copyright © 2011 ICONIX
Add test scenarios for each test case as before
• The conceptual design will not include the “infrastructure” or “scaffolding” code, but will reflect the core functionality that needs to be built
• This is OK for a first-pass estimate
Copyright © 2011 ICONIX
Transform to classes and generate code
Since robustness diagrams are drawn just after use cases are written, we have a line-countable code base very early in the project lifecycle
The earlier we can estimate accurately, the better we can plan budgets and schedules
What about hardware/software
systems?
• Does all this still work if we’re modeling an embedded system using SysML?
• Yep.
Copyright © 2011 ICONIX
DDT/Systems extends DDT/Software for SysML• DDT/Software is a great place to
start system testing, because most systems are software intensive
• DDT/Software (DDT) includes both Developer Testing and QA/Acceptance Testing
• DDT Scenario Testing applies directly to “system of systems” testing
Copyright © 2011 ICONIX
What is DDT/Systems?
• A comprehensive approach to validation and verification for systems and system-of-systems
• A systematic way to drive V&V from SysML models
• An extension of Design Driven Testing for software to include full HW/SW system V&V
Copyright © 2011 ICONIX
DDT/Systems extends DDT for the full range of SysML models,
including interface and state testing
Copyright © 2011 ICONIX
SysML Example Model: Pothole detecting spacecraft
• Orbits the earth• Takes pictures of city streets• Downlinks images to ground station• Detects potholes via image analysis• Emails images with annotated
potholes to subscribers• (Thanks to Dr. Mike Sievers of
NASA/JPL for his help with the
POTHOLE example, and with
defining DDT/Systems)
Copyright © 2011 ICONIX
How do we V&V our spacecraft from this SysML model?
• Generate tests from requirements• Generate interface tests from ports• Generate scenario tests from use cases• Generate state machine tests from triggers,
do/entry/exit behaviors• Generate unit tests from sequence messages• Generate unit tests from class operations• Generate parametric tests
Copyright © 2011 ICONIX
We’ll have one sequence diagram per use case
Copyright © 2011 ICONIX
Sequence diagrams show all system behavior within each use case
Copyright © 2011 ICONIX
These test cases are transformed into unit test code and can be an input to CoCoMo cost models
Copyright © 2011 ICONIX
State machines include triggers, and entry/exit/do behaviors
Thanks to Dr. Mike Sievers of NASA/JPL for assistance
Copyright © 2011 ICONIX
We can generate all of these behavior tests automatically from a SysML model
Thanks to Dr. Mike Sievers of NASA/JPL for assistanceCopyright © 2011 ICONIX
Interface tests are generated from Ports
Thanks to Dr. Mike Sievers of NASA/JPL for assistance
Copyright © 2011 ICONIX
Interface Tests for Non Volatile Memory include testing the Buffer Control Interface, Instrument Data Input, and Xband Output
Thanks to Dr. Mike Sievers of NASA/JPL for assistance
Copyright © 2011 ICONIX
Current Status
• DDT is described in Design Driven Testing (Apress 2010)
• DDT/Systems is described in ICONIX Process Roadmaps (Fingerpress 2011)
• DDT/Software currently automated in Sparx Enterprise Architect; DDT/Systems extensions not yet released
• MagicDraw plug-in currently under development, scheduled release 4Q 2011
Copyright © 2011 ICONIX
Copyright © 2011 ICONIX
Summing up
Early lifecycle line of code estimates are generally beneficial for cost estimation
Design Driven Testing provides a basis for early LOC estimates from conceptual and detailed design views of a use case
DDT/Systems extends this to cover state and interface testing from SysML models
LOC estimates are generated directly from the design models, thus all handwaving is eliminated
WAGs become substantially more scientific (SWAGs are better than WAGs)
Copyright © 2011 ICONIX
Questions?
Copyright © 2011 ICONIX
Thank You