Fact or Fiction? What Software Analytics Can Do For Us
-
Upload
andy-zaidman -
Category
Science
-
view
349 -
download
1
Transcript of Fact or Fiction? What Software Analytics Can Do For Us
![Page 1: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/1.jpg)
Fact or Fiction? What Software
Analytics Can Do For Us
Andy Zaidman
Delft University of TechnologySPLASH-I 2016, Amsterdam
azaidman
![Page 2: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/2.jpg)
I like to tell stories…
Andy
![Page 3: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/3.jpg)
EVERYBODY LIES
![Page 4: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/4.jpg)
EVERYBODY LIES
![Page 5: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/5.jpg)
EVERYBODY
Andy Zaidman
![Page 6: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/6.jpg)
ANDYLIES
![Page 7: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/7.jpg)
EVERYBODY
Andy Zaidman
Moritz Beller
Georgios Gousios
AnnibalePanichella
Niels Spruit
![Page 8: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/8.jpg)
Let’s consider software testing
![Page 9: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/9.jpg)
ANDY’SLIE
Grant proposal
“It is well-known that testing consumes 50% of a developer’s time”
![Page 10: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/10.jpg)
![Page 11: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/11.jpg)
“it depends”
“50% of your time”
![Page 12: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/12.jpg)
What is the (likely) origin of the 50%?
![Page 13: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/13.jpg)
The Mythical Man-Month?Brooks “observed” that testing and debugging took 50% of the time
How? “Calendar research”
When± 40 years ago…
![Page 14: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/14.jpg)
ANDY’SLIE FACTS
Grant proposal
“It is well-known that testing consumes 50% of a developer’s time [1]”
![Page 15: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/15.jpg)
![Page 16: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/16.jpg)
Lack of critical thinking has allowed wrong
claims to take root in the discipline and become
entrenched.
![Page 17: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/17.jpg)
Chinese whispers / telephone game
![Page 18: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/18.jpg)
“Truth begins in lies”-House
![Page 19: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/19.jpg)
Lies begin in truth
“Truth begins in lies”-House
![Page 20: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/20.jpg)
Folklore/evidence test approximately 50%
Lack of knowledgeHow much time is actually
spent on testing?
![Page 21: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/21.jpg)
![Page 22: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/22.jpg)
Research Tool Software Analytics“Big Brother” in your IDE measures testing activities
• 416 Softw. Engineers, 68 countries• 5 month observation period• Java / Eclipse • 13.7 man years of development
observed
Immediate feedback for developers
![Page 23: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/23.jpg)
WatchDog Pre-Test Questionnaire
Can you estimate how much time you spend on engineering test code versus production code?
50% 50%test code production code
30% 70%test code production code
![Page 24: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/24.jpg)
WatchDog Pre-Test Questionnaire
Can you estimate how much time you spend on engineering test code versus production code?
48% - 52%test code production code
![Page 25: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/25.jpg)
After measuring ≥ 3 months
25% - 75%test code
engineering
Most participants overestimate their test engineering activities
production codeengineering
![Page 26: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/26.jpg)
Other observations
![Page 27: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/27.jpg)
Not all Java projects do unit testing
Likely, an underestimation…
![Page 28: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/28.jpg)
No test engineering
For projects with test code: 53% of developer did not execute, read, or modify a single test within five months.
![Page 29: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/29.jpg)
Do sets of unit test executions take…
< 5 seconds ≥ 5 seconds
![Page 30: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/30.jpg)
Swift tests
Execution of set of unit tests takes 0.54 seconds (median).
![Page 31: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/31.jpg)
Ad hoc test selection
Developers frequently select a specific set of tests to run in the IDE. In most cases, developers execute one single test.
![Page 32: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/32.jpg)
Test execution in the IDE is rare
In 85% of development sessions no tests were run…
![Page 33: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/33.jpg)
Tests fail
65% of test executions in the IDE fail.
![Page 34: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/34.jpg)
Test Driven Development
TDD is not widely practiced. Programmers who claim to do it, neither follow it strictly nor for all their modifications.
![Page 35: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/35.jpg)
• Be critical: are long-standing beliefs/insights/claims• Still valid?• Valid in your context?
• Triangulate• Developer opinions deviate greatly from
what we objectively measured• Mixed-method research
• Longitudinal studies
![Page 36: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/36.jpg)
• Estimating is difficult
• Estimating something that you don’t like is more difficult
• Beliefs are to be questioned all the time
![Page 37: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/37.jpg)
![Page 38: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/38.jpg)
Differential diagnosis- Tests are written, but not run- Tests are written, but run on Command Line- Tests are written, but run in Continuous Integration
![Page 39: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/39.jpg)
Differential diagnosis- Tests are written, but not run- (Tests are written, but run on Command Line)- Tests are written, but run in Continuous Integration
![Page 40: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/40.jpg)
• 1,359 Java and Ruby projects• 2,640,825 builds
![Page 41: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/41.jpg)
Testing in Continuous IntegrationIn our population of 1,359 Java and Ruby projects
![Page 42: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/42.jpg)
Builds with at least one failing testFrom 1,108 projects which executed tests
![Page 43: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/43.jpg)
Testing #1 reason for build to break From 1,108 projects which executed tests
Test failure
Other failure
![Page 44: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/44.jpg)
Tests are decisive for overall buildFrom 1,108 projects which executed tests
98.3% projectsTest fails Build breaks
![Page 45: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/45.jpg)
Integrating in multiple environmentsFrom 1,108 projects which executed tests
60% projectsDifferent build result for at least one build in
multi-environment setup
![Page 46: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/46.jpg)
Differential diagnosis- Tests are written, but not run- (Tests are written, but run on Command Line)- Tests are written, but run in Continuous Integration
- 60% failing tests in the IDE- <20% failing tests in CI- Command Line?
![Page 47: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/47.jpg)
ANDY’S LIES
1. 50% of time goes to testing
2. This talk was on software analytics
![Page 48: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/48.jpg)
Empirical Software
Engineering
Software Analytics
![Page 49: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/49.jpg)
Software analytics =
bait / intrinsic motivation
![Page 50: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/50.jpg)
![Page 51: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/51.jpg)
![Page 52: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/52.jpg)
From an experienced developer…“Estimated time working on tests: 20%,
actual time: 6%. Cool statistics! ”
Testing analytics helps developers to get a better understanding about their own
development behavior!
![Page 53: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/53.jpg)
Software analyticsEmpowers software engineers, software teams, managers
Make better tools
![Page 54: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/54.jpg)
Debugging
![Page 55: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/55.jpg)
WatchDog 2.0We are looking into how developers debug
Product improvement: In contact with Eclipse + IntelliJNew features coming to Eclipse based on our
analytics
Educational improvement: We need to teach our students to debug more efficiently (with the IDE debugger)
![Page 56: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/56.jpg)
What to expect?
![Page 57: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/57.jpg)
Software Analytics Challenges for the SPLASH community• What works in languages? What doesn’t?
• How should we design the next version of languages?
• How should we design the next generation of tools?
• How should we improve education?
![Page 58: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/58.jpg)
Software engineering
folklore
Perception w.r.t. dev. testing
Measured developer testing
48% - 52%
25% - 75%
Software analytics help engineers to better understand
their own behavior
azaidman
WatchDog Team
Challenges for SPLASH
![Page 59: Fact or Fiction? What Software Analytics Can Do For Us](https://reader033.fdocuments.us/reader033/viewer/2022042906/58a625841a28ab416c8b51b3/html5/thumbnails/59.jpg)
Pointers to literature1. Moritz Beller, Georgios Gousios, Andy Zaidman:How (Much) Do
Developers Test? ICSE 2015: 559-5622. Moritz Beller, Georgios Gousios, Annibale Panichella, Andy Zaidman:
When, how, and why developers (do not) test in their IDEs. ESEC/SIGSOFT FSE 2015: 179-190
3. Moritz Beller, Georgios Gousios, Andy Zaidman:Oops, my tests broke the build: An analysis of Travis CI builds with GitHub. PeerJ PrePrints 4: e1984
4. Fred Brooks: The Mythical Man-Month, Addison-Wesley, 19755. Laurent Bossavit: The Leprechauns of Software Engineering, 2012-
2014, https://leanpub.com/leprechauns