Software Analytics: The Dark Side and the Test Side

SEN Symposium 2016 Andy Zaidman, TU Delft

I like to tell stories…

Andy

Jurgen/Frank: Can you give a keynote on

software testing and software analytics at the

SEN Symposium?Me:

Sure I can, I’m honored by the invitation.

Jurgen/Frank: Can you give us a title and

abstract?Me:

<<PANIC>> what should I talk about? Subject so broad...

ARRGHHH...

1. “Software Engineering”

2. “Software Testing”

3. “Software Analytics”

I started by asking

I turned to again

The Dark Side

@ work (Dec. 2015)

http://www.popularmechanics.com/technology/infrastructure/a18721/washington-state-released-3200-inmates-early/

http://townhall.com/tipsheet/mattvespa/2015/12/22/good-news-computer-glitch-caused-the-irs-to-issue-46-million-worth-of-questionable-refunds-n2096374

Bad

http://deredactie.be/cm/vrtnieuws/cultuur%2Ben%2Bmedia/kunsten/1.2490758

vs.

3 - 0

Bring balance, you must!

Test more, you should!

“it depends”

“50% of your time”

What is the (likely) origin of the 50%?

The Mythical Man-Month?Brooks “observed” that testing and debugging took 50% of the time

How? “Calendar research”

When± 40 years ago…

Lack of critical thinking has allowed wrong

claims to take root in the discipline and become

entrenched.

Chinese whispers / telephone game

Folklore/evidence test approximately 50%

Lack of knowledgeHow much time is actually

spent on testing?

NWO Vidi project

Research Tool Software Analytics“Big Brother” in your IDE measures testing activities

• 416 Softw. Engineers, 68 countries • Java / Eclipse • 13.7 man years of development

observed

Immediate feedback for developers

WatchDog Pre-Test Questionnaire

Can you estimate how much time you spend on engineering test code versus production code?

50% 50%test code production code

30% 70%test code production code

WatchDog Pre-Test Questionnaire

Can you estimate how much time you spend on engineering test code versus production code?

48% - 52%test code production code

After measuring ≥ 3 months

25% - 75%test code

engineering

Most participants overestimate their test engineering activities

production codeengineering

Other observations

Not all Java projects do unit testing

Likely, an underestimation…

No test engineering

For projects with unit test code: 53% of developer did not execute, read, or modify a single test within five months.

Do sets of unit test executions take…

< 5 seconds ≥ 5 seconds

Swift tests

Execution of set of unit tests takes 0.54 seconds (median).

Ad hoc test selection

Developers frequently select a specific set of tests to run in the IDE. In most cases, developers execute one single test.

Tests fail

65% of test executions in the IDE fail.

Test Driven Development

TDD is not widely practiced. Programmers who claim to do it, neither follow it strictly nor for all their modifications.

Software Analytics?

What about

Testing Analytics?What about

Software analytics =

bait / intrinsic motivation

From an experienced developer…“Estimated time working on tests: 20%,

actual time: 6%. Cool statistics! ”

Testing analytics helps developers to get a better understanding about their own

development behavior

Hawthorne effect?

Hawthorne effect?“Keeping up appearances” > 3 months?

Takeaways

Triangulate your results mixed methods research

Be critical towards “folklore” Core might be true, but is it (still) valid,

valid in your context, etc.

Don’t talk testing, but act testing

Takeaways

Critique?• Only developer testing ( ≠ Brooks)• Only Java, only Eclipse• Only 416 developers • Only inside the IDE

Extending to IntelliJ and > 1500 developers

Also looking into Continuous Integration

Software engineering

folklore

Perception w.r.t. dev. testing

Measured developer testing

48% - 52%

25% - 75%

Software analytics help engineers to better understand

their own behavior

azaidman

WatchDog Team

References• M. Beller, G. Gousios, A. Panichella, A. Zaidman. When, How and Why

Developers (Do Not) Test in Their IDEs. Proceedings of the Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 179-190, 2015.