Measure It, Manage It, Ignore It - Software Practitioners and Technical Debt

© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

Measure it? Manage it? Ignore it? Software Practitioners and Technical Debt Neil A. Ernstwith Stephany Bellomo, Ipek Ozkaya, Robert NordSoftware Engineering InstituteCarnegie Mellon UniversityPittsburgh, PA 15213

and Ian GortonCollege of Computer and Information ScienceNortheastern University

2TD SurveySeptember 2, 2015© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

Copyright 2015 Carnegie Mellon University and IEEE

This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center.

References herein to any specific commercial product, process, or service by trade name, trade mark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by Carnegie Mellon University or its Software Engineering Institute.

NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.

This material has been approved for public release and unlimited distribution.

This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at [email protected].

DM-0002708

3TD SurveySeptember 2, 2015© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is UnlimitedDistribution Statement A: Approved for Public Release; Distribution is Unlimited

Background• Cunningham, 1992: “Shipping first time code is like going

into debt. A little debt speeds development so long as it is paid back promptly with a rewrite... The danger occurs when the debt is not repaid”

• Our definition: “the obligation that a software organization incurs when it chooses a design or construction approach that's expedient in the short term but that increases complexity and is more costly in the long term.” (McConnell)


Research Questions

• RQ1. Is there a commonly shared definition of technical debt among professional software engineers?

• RQ2. Are issues with architectural elements (such as module dependencies, external dependencies, external team dependencies, architecture decisions) among the most significant sources of technical debt?

• RQ3. Are there practices and tools for managing technical debt?


Methodology

• Pilot Survey• Full survey• Triangulate with:

• Multiple choice background

• Open-ended questions

• Likert agreement• Follow-up interviews

(opportunistic)


Demographics• 1831 surveys were started (across all three collaborators)

and 536 surveys fully completed (all questions answered), an overall response rate of 29%

• Respondents over 6 years experience• Roles selected included developers (42%), and project

leads/managers (32%)• Most projects were web systems (24%) or embedded

(31%). • Projects generally consisted of 10-20 people• The systems averaged 3-5 years old, but a significant

number (29%) were over 10 years old. • The systems between 100KLOC and 1MLOC in size.


Findings

1. Architecture is a key source of technical debt2. Technical debt is broadly temporally distributed (source

and symptom not synched)3. Issue trackers are currently the best tool approach to

managing technical debt.

Triangulate answers to these questions by looking at quantitative (closed-ended) questions, and coding of open-ended and interview data.


RQ1: Defining Technical Debt

• “(A2) I think the vocabulary of technical debt is useful for getting the interests aligned.”

• ‘convincing product managers and stakeholders on the value proposition of managing the debt.’


RQ2: Architecture choices key

9


Coding of open-ended questions


RQ2: Architecture as source


RQ2 (2)

“(B2) the work that we’re doing now to introduce a service layer and also building some clients using other technology is an example of, you know, decisions that could have been done at an earlier stage if we had had more time and had the funding and the resources to do them at the time instead of doing it now.”

“‘platform’ was not designed with scalability in mind”

“In retrospect we put messaging/communication ... in the wrong place in the model view controller architecture”


RQ2: DriftLehman’s concept of entropy present in our data:• “over the years, other sites would begin using the system and

would require changes to how the workflow operated” Weak association between system age and the perceived importance of architectural issues, using Yule’s Q. 89% of those with longer-lived systems (>=6 years old) agreed or strongly agreed with the notion that architectural issues were a significant source of debt, compared to 80% of those with newer systems (<3 years old).


RQ3: Management of TD

How tracked … Where tracked …


RQ3: Tool Use

15

None/Unknown: 58%


RQ3: Management of TD

“(B1) regarding static analysis we have the source code static analysis tools, but this is to assure proper quality of source code. But how architectural changes are impacting I don’t know. And, in fact, this is something we don’t do.”

“(C1) it showed up on Jenkins - the CI server - there’s a billion little warnings. And so it seems a little bit overwhelming.”

“[we track] occasionally by explicit tech debt items, usually by pain, or not at all...”


• Timeline for TD management

• Track TD items like we do defects • Connect Architecture Technical Debt to architectural analysis

(e.g., quality attribute prioritization)

Future Work and Open Questions


18

• Software practitioners agree on the usefulness of the metaphor

• Different interpretations of what makes up technical debt in particular contexts.

• Leading sources of technical debt are architectural choices

• take many years to evolve

• Managing this drift is vital in managing technical debt.

• Developers perceive management as unaware of technical debt issues

• Desire for standard practices and tools to manage technical debt that do not currently exist

http://bit.ly/sei-td – @neilernst – [email protected]

http://bit.ly/sei-td

http://bit.ly/sei-td


Call for Papers: Negative Results in Software EngineeringSpecial Issue of J. Empirical Software Engineering.

We welcome your well-conducted yet ‘negative’ empirical studies.We also are looking for suitable reviewers and reviewer experience reports.

Deadline for submission: October 7, 2015http://bit.ly/emse-negative

If you have questions/comments or would like to volunteer to be a reviewer of the papers, please contact the guest editors.Richard Paige [email protected] Jordi Cabot [email protected] Neil Ernst [email protected]

PlugPicture

(optional)

http://bit.ly/emse-negative

http://bit.ly/emse-negative

mailto:[email protected]




BackupPicture

(optional)


Coding Process

Id Statement Coder 1-A

Coder 1-B … Coder 2-C

1 Technical debt is built over years & multiple versions by the combination of factors like monolithic code design, mix of obsolete and new technologies, cost over quality etc.

Code Code … Code

Cohen’s K: 0.45


Code definitionsFinal Coding Term Definition Subsumed Codes

Interest The pain caused by technical debt, but not the debt itself.

Rework Additional work needed to remove technical debt ‘principal’

Architecture choiceWhile arch choice is pain caused perhaps by context shifts vs design shortcut that is deliberative we combine the two as “architecture choice” because participants may not even know whether it was deliberate or not. Must be a specific case/instance.

Design Shortcut, Bad Architecture Choice

Legacy modernization Changes and evolution in operating environment e.g. new tech or requirements changes. Obsolete Technology, Legacy, evolution, changing requirements, External Dependencies, Prototype become Product

Limited Knowledge Respondent had no good understanding of TD No clue

Awareness People in respondent’s org had no understanding of problems TD caused Management, avoidance strategy, culture, deferred integration design, metaphor

Defects Responses referring to problems in code externally visible. Interest code is more suitable when text refers to linking those bugs to original decision. Bugs, Maintenance, Software Quality

Time Pressure Must release to make deadline Schedule

Cost Pressure Lack of financial resources or motivation to fix problem New code

Code Problems Issues relating to code-level problems such as detected by FindBugs Overly complex code, inter-module dependencies, Code with Debt, Code Duplication

Process Some problems arising from poor development processes. Inadequate testing, Inefficient CM, Lack of Documentation

None A comment that is not possible to code

Measurement codes about need for measuring, things we might need to measure such as complexity and accuracy

Tools this category is about the things that are technical in nature including tool support for making intelligent decisions. Platform dependence, ways to find TD

Requirements Shortfall Requirements not changing significantly, but system does not meet them. Could be due to ambiguity, lack of clarity, etc.

Lack of documentationIssues due to incomplete understanding of the architecture which was often attributed to lack of documentation or some way to better understand the impact of making changes on the quality or maintainability of the system

Inadequate Testing This category covers inadequate testing due to issues such as inadequate test coverage, limited test resources, lack of test automation or not enough time to complete tests

Measure It, Manage It, Ignore It - Software Practitioners and Technical Debt

Technology

Transcript of Measure It, Manage It, Ignore It - Software Practitioners and Technical Debt