Implementing Zero-Debt Continuous Inspection in an Agile Manner A Case Study Brian Chaplin October...
-
Upload
jason-jacobs -
Category
Documents
-
view
218 -
download
3
Transcript of Implementing Zero-Debt Continuous Inspection in an Agile Manner A Case Study Brian Chaplin October...
ImplementingZero-Debt Continuous Inspection in an Agile Manner
A Case Study
Brian ChaplinOctober 8, 2013
Agenda
• What is technical debt?• What is continuous inspection?• A scalable framework to automate technical
debt reporting• How to control technical debt using the
framework• Best practices• Lessons learned from 2 case studies
– Agile lessons learned• What you can do tomorrow
Technical Debt Metaphor
• Ward Cunningham at OOPSLA 92 : “Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite...”
• However,– Every minute spent on debt counts as interest
Consequences of Technical Debt
• Product atrophy• Decreased
predictability• Underperformance• Universal frustration• Decreased customer
satisfaction
• Unpredictable tipping point
• Increased time to delivery
• Significant number of defects
• Rising development and support costs
From Essential Scrum, Kenneth S. Rubin
Types of Debt• Naïve
– reckless debt (Fowler 2009), unintentional debt (McConnell 2007), and mess (Martin 2008).
• Unavoidable– Although this debt might be predictable it’s not
preventable because we can’t foresee how the component might evolve
• Strategic– deliberately make a strategic decision to take shortcuts
during product development to achieve an important short-term goal, such as getting a time-sensitive product into the marketplace.
Naïve Technical Debt DefinedCoverage Uncovered conditions
Uncovered linesComplexity Average method complexity
Total class complexityCompliance 140 static code analysis rules
Critical, major, minor weights
Duplication Duplicated lines
Organization Circular dependencies
Design Fan-out, 6
What is Continuous Inspection
• an approach to running automated code analysis as part of a build in order to find code quality problems
Why Continuous Inspection?
• 5. Sustainable development, able to maintain a constant pace
• 9. Continuous attention to technical excellence and good design
• can help reduce the time spent in long, manual code review sessions
Continuous Inspection Tooling
• SonarQube www.sonarqube.org
• CAST www.castsoftware.com
• Microsoft Visual Studio/Team Foundation Server
Automating Continuous Inspection Reporting
• Necessary for large projects–Agile and timely reporting within hours
• Necessary for zero-debt–Architects only have time for the big
issues–Computer handles the smaller quality
defects
Business Context
• Large project, semi-Agile• IT history of past quality initiatives
– OO and pair programming– Automated unit test generation
• Management goal of 80% test coverage• Continuous integration build/deploy
Characteristics of 2 Case StudiesCategory Java
Open SourceC# dot Net
Developers 100 175LOC 2,200,000 1,600,000Commits per day 30 36Classes 14,000 13,000Unit tests, Coverage 41,000 83% 22,000 63%
File changes per day 225 500Technical debt per day 20 50Code quality since 2009 Dec. 2012
Architecture Requirements for Code Quality (CQ) Statistics
• Must come from existing continuous integration process
• Available within 24 hours• Meaningful to developers, leads and
management• Must involve reviewers• Must tie to accountability points
– Leads, reviewers, submitters, project, business function
• Extract, transform and load
• Match the committer with the quality change
Architecture ETL Overview
Architecture Overview• Extract/Transform/Load
(ETL)• Input
– Source code management system (SCM)
– Project management system
– Code quality metrics
• Output– SQL database– Excel pivot table reports– Defect tickets (project
management system)– Email– Ad hoc management
reports– Program scores and
developer contribution
Establish and Maintain the Quality Database
• Guard against corrupted quality data– Test or build failures– Sonar may record lower stats– Rollback a bad build
• Design for mash-ups• Combine code quality with runtime defect
reporting
Maximize Report Flexibility• Enables timely and actionable reporting• Excel pivot table as data-marts• Refreshed Excel reports accessed via Email
hyperlinks or served up by web server
Bake Code Quality into the Build and Sprint
Notificationsin every phase Sprint
Code Quality ChangeTechnical DebtOpen Tickets To Fix DebtDaily
Integration Tests
Code Quality Champions Personalized CQ ReportContinuous
Auto Build Unit Test Deploy
Email Code Quality Warnings 18
Fast Feedback
Technical Debt Information Radiator
1. Fast feedback: identify and notify Immediately
–Inside the IDE–After the commit
2. Track until it’s fixed
1. Notify Immediately• Email each debt violation within hours
– After each continuous inspection build• 3 times a day or after every CI build
• Notify both committer and code reviewer• Daily contribution
– Best contributors summary– Personalized contribution detail to each
committer• Hyperlink to more reports and wiki
– Direct Sonar hyperlink to the degraded file
Fast Feedback
Project ID
Change ID
Prog. Name
UncoveredLines or Branches
Static Code
Analysis Debt
Excess Complexity
Class Function
ticket1 580932 4/18/2013 Class1 9 9
ticket1580902
4/18/2013 Class3 4 0 14 2.8
ticket1 580902 4/18/2013 Class5 23 0 13 2.6
Submission Warning(s)You are receiving this message because even though the code quality may have been enhanced, the submission(s) below have decreased the code quality (coverage, compliance, and/or uncovered complexity) or they didn't meet the standards for a new class. Please review these submissions with your reviewer and take the appropriate action.
Sample Violation Email
• Metric• Submitter• Lead• Reviewer• Class• Sprint• Date
2. Track Until It’s
Fixed
2. Track Debt Until It’s Fixed
Making Technical Debt Visible
• Estimate the cost of remediation• Assign owner• Report weekly progress by owner• Report trends• Monetize it, various methods
– SQALE (Software Quality Assessment based on Lifecycle Expectations) from Inspearit, Jean-Louis Letouzey
Monetize the DebtEstimated Remediation Costs
Cost Activity12 minutes To unit test uncovered branch or
line6 minutes To fix static code analysis violation8 hours To split a class that is too complex30 minutes To split a method that is too
complex2 hours To eliminate a block of duplicated
code
Current debt
Ratio of bad to good code
Weekly progress
Report Weekly Progress by Owner
Report Weekly Tech Debt Trends
• By project• Or by
owner
• Staff a code quality desk–Knowledge clearinghouse
• Use code reviewers• Stabilize build and project structure• Establish static code analysis rules and
rarely change them
Best Practices
• Developers must be able to clear unfixable debt from the tracking report
• Run the ETL at least daily• Keep the database accurate• Track both contribution and debt• Recognize code quality champions• Use uncovered line/branch count not
percentage
Best Practices, cont’d
Allow for Exclusions, False Positives
• Integrations, class rename, code moves
• Classes exempted from quality–Registries–Test support
• Unfixable debt–Unreachable test
cases–Exceptions to the
metrics standard–Static analysis false
positives• 0.5% are excluded
Case Study Lessons Learned• Communicate the benefits of code quality
– Maintainability– Less runtime defects, explain carefully– Working with a net
• Help new developers with un-testable code– Train and mentor– Consider refactoring to testable code first, then
write the test
Reviewers Guard Against Doing the Wrong Thing
• Meaningless comments• Unit tests just for coverage that assert
nothing• Allow some large case statements
Too Busy to Keep Debt Down?
Monthly debt rate
Managing the Debt
Rate
• Debt rate is about 15% for both projects
• Varies widely by developer
Techniques to Keep the Debt Rate Low
• Immediate feedback• Open defect tickets at
key points• Train• Assist developers one
on one• Encourage re-
factoring to write testable code
• Code a little, test a little
• Keep management aware
• Continually monitor, don’t let it get out of hand
• Recognize the champions
Count 282 8,447Average value NPE Not Function complexity 2.90 1.88 Lcom4 1.98 1.14 Weighted violations 7.52 2.70 Complexity 66 11 Coverage 85% 83% Duplicated lines 1.99 5.61 Comment lines density 28% 32% Statements 144 25 Score 1.53 1.73 Fan-out 9.45 3.16
Probability =exp( a + b*WEIGHTED_VIOLATIONS + c*COMPLEXITY -d*COVERAGE -
e*COMMENT_LINES_DENSITY - f*SCORE)
10/20 top NPEs
predicted
probability
class name CXTY SCORE
100%class1 528 1.13100%class2 523 1.12
99%class3 464 0.8999%class4 523 1.2599%class5 523 1.4598%class6 369 1.0796%class7 345 1.1896%class8 341 1.3395%class9 399 0.9595%class10 385 0.8393%class11 305 1.0592%class12 372 1.2491%class13 279 1.3290%class14 416 1.1288%class15 263 1.0288%class16 328 1.0287%class17 318 1.0986%class18 428 1.1085%class19 316 0.8984%class20 236 1.22
Does bad code cause defects?
Implement the “Boy Scout rule”
• Encourage Code Quality Improvement On Every Commit
• Score the program’s improvement• Score the committer’s improvement• Recognize the contributors• Ensure the numbers are used properly
Agile Lessons Learned
When and How to Create Debt Backlog Items
• Detailed appropriately– By metric, program file,
developer
• Emergent– At release code branch– Move to later Sprints as
necessary
• Estimated– Use Sonar tech debt
estimate
• Prioritize by– Complexity– Unit tests– Static code
Keep the Pressure Off
• Be positive!• Manage debt informally as sprint
issues• Use TDD and other techniques as
productivity aids• Interpret the contribution metrics
correctly
Fast Feedback
• Reviewers work with submitters to fix same day
• Quick emails with links to more information
• Allow developers to correct immediately after receiving an email
Small Batch Size
• Fix the debt as it’s incurred
• Don’t let it build up
• Minimize tech debt inventory1. Static code violations2. Code and branch
coverage3. Complexity
management4. Circular
dependencies5. Fan-out reduction
Individual vs. Group Accountability
• Idle work vs. idle workers–Attack the most significant debt first
• Fix your own debt quickly–You know the code best
• Fix debt in the code you’re working on–Leave it cleaner than you found it
Minimize Ceremony
• Backlog spreadsheet always available• Morning “warm-up”, fix a little every
day• Only open backlog tickets at release
or sprint boundaries
Code Quality Take-away Messages
Quality Motivates and Enhances Productivity
• Tom DeMarco & Timothy Lister, Peopleware: “Quality, …, is a means to higher productivity”
• Defect removal #1 cause of productivity loss
• Programmers are motivated by emphasis on code quality
Implement CQ Continuous Inspection• Establish CQ point of
accountability• Promote CQ• Track technical debt
– Require remediation• Scales to large projects
– Multi-branch– Multi-language
• Target the reports– End of Sprint or
production release branch
– By lead, committer, reviewer
– Commit quality trend
Embed a Code Quality Expert on the Team
• Unit testing techniques– Mocks– Corner cases
• When to override the warnings• When to clear, not fix, debt• Safe re-factoring
How Much Time for a Code Quality Czar?• 12% of a senior dev’r
cq48%
lead35%
devr9% mgr
5%build
er2%
Roles
Effort breakdownRole task Total
Code Quality Czar
monitor technical debt 13%clear unfixable debt 10%mentor/assist 6%scoring 5%monitor dev branches 4%manual db update/fix/check 3%ad hoc reporting 2%email warning help desk 2%standards setting 2%investigate numbers/database 1%
Lead developers
report review 16%performance appraisal 13%presentations 4%train 1%trend analysis 1%
Code coding 5% write SQL 4%Manage awareness 2% inform/escalate 2% sell 2%Build sonar 2%
13% more code per yearAll with 100% test
coverage
Keep the code base maintainable so velocity continues
You Can Maintain Velocity and have Zero Debt
What Can You Do Tomorrow?• Implement a code quality dashboard
– Sonar– MS Visual Studio or Eclipse CQ plugin
• Require unit testing• Peer review at check-in• Break the CI build if quality decreases
– Coverage drops below threshold– Critical or major static analysis violations