Natalia Juristo

University of Oulu &

Technical University of Madrid

Conduc'ngExperimentsinIndustry:TheESEILFiDiProProject

Project People Goal

Gaininsightintothechallengesofconduc1ngexperimentsintheso6wareindustry

ImproveunderstandingofdifferencesbetweenexperimentsinthelabandinthefiledforSE

Experimental Software Engineering Industrial Laboratory (ESEIL)

January 2013-December 2017

ExperimenttopicschosenbycompaniesUptothree

Eachexperimentreplicatedbyseveralindustrialpartners

Companiesrunning2-3experimentsover5yearsWithaminimumof1

Research Approach

CompletetheSEexperimentalpathAresoAwareindustrialexperimentsequivalenttofieldexperiments?

UnderstandthebarrierstosoAwareindustryexperiments

Learnwhetherexperimentscanbeusedfordecisionmakinginindustry

Understandthedifferencesbetweenstudentsandprofessionalsasexperimentalsubjects

ExternalvalidityofresultswithstudentsBehaviorofsubjects

ResearchGoals

Experiment Runs Design

An experiment on TDD

Experiment sold as hands-on exercises embedded in a training course

Limitations are placed on design

Participants are professionals but novices in the technology being evaluated

TRAINING EXERCISES TREATMENTS

DAY1 UTConcepts&Slicing

2SlicingExercises BASELINETASK(DoItYourWay)

DAY2 Slicing&TDD

2ITLExercises1TDDExercise

ITLTASK

DAY3 TDD 1TDDExercise TDDTASK

Organiza'on Country Date No.ofSubjectsUniveristyofOulu Finland may-14 48TechnicalUniversityofMadrid Spain mar-14;oct-14;oct-15 38UniversityofBasilicata Italy oct-15 20UniversityofSouthernDenmark Denmark jan-16 71TechnicalUniversityofValencia Spain may-14 32UniveristyofESPE Ecuador apr-14;apr-15;apr-16 43Elektrobit/Biaum Finland mar-14 9Ericsson Finland mar-15 21FSecure Finland&Malaysia oct-13 31Mapfre Spain jun-15 14Paf Finland mar-16 13PlayTech Estonia mar-14 18EcuadorianArmy Ecuador Apr-15 22

130 professionals 250 students

Today !I will not report !

results! J !

What we have learnt

Recruitment •  Hardtosignupasignificantnumberofpar1cipants

–  Developer1meismoney–  Numberofpar1cipantswaslowinallcases

•  8-20

•  Trainingwastheonlycarrotthatwefound

•  Companystructurehasamajorimpactonrecruitmentsuccess–  Companieswithbooked1mefortrainingwereeasiertoconvince

•  F-Secure

•  Projectleaderaschampionbederthaninnova1onmanagers–  PLadministersdeveloper1me

•  Mapfre&Playtech

Technologies •  Technologiesvaryacrosscompanies

–  Language,IDE,tes1ngframeworkweredifferentacrosscompanies

–  Experimentalinstrumentshadtobeadaptedseveral1mesfordifferentcompanies

•  OriginallyforJava,EclipseandJUnit(foracademicseang)•  AdaptedtoC++,C#,Boost,GoogleTest,IntelliJ

– Wemissedsomeinteres1nginstruments•  Astreatmentconformance(onlyavailableforJava)

Design •  Par1cipantsvola1litythreatscontrol

– Feweradendeesthansignedup– Moredrop-outs

•  Missingdatapointsthatthreatenvalidity–  Paf

– Adendeessome1meshaddifferentprofilethanexpected

•  Redesignonthefly–  Ericsson

@PafPar1cipants

–  Planned:14subjects–  Real:13subjects–  Useful:8subjects

•  Dataremoved:5subjects–  4adendedonly1session–  Group3hadonly1subject

TheimportanceofstayingontoperformallexperimentaltaskswasnotwellunderstoodLoosingagroupmeantthatwewereunabletocomparealltreatmentsforseveraltasks

DAY1YW

DAY2ITL

DAY3TDD

GROUP1 BSK SS MR

GROUP2 SS MR BSK

GROUP3 MR BSK SS

Design •  Par1cipantsvola1litythreatscontrol

– Feweradendeesthansignedup– Moredrop-outs

•  Missingdatapointsthatthreatenvalidity–  Paf

– Adendeessome1meshaddifferentprofilethanexpected

•  Redesignonthefly–  Ericsson

@EricssonPlannedDesign

Expectedsubjects:experiencedinC++,Eclipse,Boostandunittes1ng

TRAINING EXERCISES TREATMENTS

DAY1 Tes1ngToolConcepts

2ToolExercises1Mo1va1onalExercise(ITL)

BASELINETASK(DoItYourWay)

DAY2 Slicing 2SlicingExercises CONTROLTASK(ITL)

DAY3 TDD 3TDDExercises TREATMENTTASK(TDD)

TRAINING EXERCISES TREATMENTS

DAY1 Tes1ngToolConcepts

2ToolExercises1Mo1va1onalExercise(ITL)

BASELINETASK(DoItYourWay)

DAY2 Slicing 2SlicingExercises CONTROLTASK(ITL)

DAY3 TDD 3TDDExercises TREATMENTTASK(TDD)

RealDesignSubjects:veryinexperiencedinBoost&unittes1ng;inexperiencedinC++

Characteriza1onofthepar1cipantsMostsubjectsare:•  VeryinexperiencedinBoost•  VeryInexperienced/inexperiencedinunittes1ng•  InexperiencedinC++•  AlltypesforOO,programmingandIDE

Behavior •  Professionalsarelessmo1vatedthanstudents

–  Adendanceofatrainingcourse<>grading–  Preoccupiedwithworkissues–  Usedtoflexibleschedule–  Youngpar1cipantsmoreac1veandenthusias1cthanolderones–  Theremightbeseveralotherpsychologicalissues

•  Treatmentcomplianceislowerthanforstudents–  Studentsappeartobemorewillingtoabidebytherulesdefinedbyinstructors

–  Professionalstendtohavetheirownideasaboutwhattheyexpecttogetfromthecourse/experiment

•  Professionalsmightbeafraidofbeingassessed–  Somesubjectsremovedtheircode

Results Reception •  Managersverymuchwelcomedthefigures

–  Theywereamazedbythequan1ta1veinforma1onaboutdevelopment

•  Significancewashardtograsp–  Theytendedtofocusontheaverageandneglectedsignificanceandpower

– Wetriedoutdifferentrepresenta1ons•  Chartswereveryuseful

•  Repor1ngneedstodifferfromresearchpapers–  Focusondiagramsratherthannumbers–  Statethefindingsinwords– Discusstheconsequencesofresultsintheircontext

Means and Error Intervals Nonsignificant Significant

15.9%

46.4%

22.5% 48.3%

58.4%

Results Reception •  Managersverymuchwelcomedthefigures

–  Theywereamazedbythequan1ta1veinforma1onaboutdevelopment

•  Significancewashardtograsp–  Theytendedtofocusontheaverageandneglectedsignificanceandpower

– Wetriedoutdifferentrepresenta1ons•  Chartswereveryuseful

•  Repor1ngneedsdifferfromresearchpapers–  Focusondiagramsratherthannumbers–  Statethefindingsinwords– Discusstheconsequencesofresultsintheircontext

Impact of Findings •  Someadoptedideasfromtheexperiment(ifnottheresults)–  Toimprovetheirdevelopmenttools

•  EBadoptedinstrumentstomonitordevelopers•  EvenwhenresultsconvincedmanagersandtheyoptedtoadoptTDDtheyfacedreluctancefromdevelopers–  Conceptsfromtechnologytransferareneeded

Conclusions

Theconceptoffieldexperimentneedsmoreresearch

Itsadapta1ontoSEisnotsimpleStrategiestofacethreatstointernalvalidity

BothtypesofexperimentsareneededAr1ficialhighlycontrolledenvironmentAndnaturalenvironments

Needtoimproveunderstandingonthevalidityofsubjects

Students,althoughnovices,mightpossiblybenotasbad

aswethoughtasexperimentalsubjects

Natalia Juristo

University of Oulu &

Universidad Politécnica de Madrid

Conduc'ngExperimentsinIndustry:TheESEILFiDiProProject