1 14.Sep.2010 Systematic Literary Reviews: Experience and Advice Systematic Literary Reviews:...

14.Sep.2010

Systematic Literary Reviews: Experience and Advice

1

Systematic Literary Reviews: experience and challenges

ISERN discussion session14 Sept. 2010, Bolzano, Italy

Reidar Conradi 1), Mohammad Ali Babar 2)

1) NTNU, NO-7491, Trondheim, Norway2) ITU, Copenhagen, Denmark

http://www.idi.ntnu.no/grupper/epos/isern/slr-comments-isern-14sep10.ppt

[email protected], [email protected]

20.04.2010

X-discipllnarity in OSS; GoOpen, 20 Apr. 2010

2

Table of contents

0. Some challenges of SRLs

1. What is so special about SE and ESE/EBSE?

2. Availability and Utility of current SLRs

3. Utility of SLRs for Agile incl. Pair Programming

4. SLR: status, challenges, future advice

5. Future: OSS and networked development

6. References

20.04.2010


3

0. Some challenges of SRL

Summary of challenges:

• What is currently known about the strengths and limitations of systematic reviews for software engineering?

• How rigorous and relevant are the systematic reviews reported so far?• What are the challenges of the adoption of this methodology in software

engineering?• What are the improvements proposed for maturing this methodology?• What is the role of systematic reviews in software engineering research and

practice in the coming ten years?• How can ISERN members contribute to the evidential assessment of the

rigor, effectiveness and relevance of systematic reviews in software engineering?

• What kinds of Methods are required to ensure the rigor and relevance of systematic reviews for software practice?

• What can be done to address the most common challenges (such as forming optimal search strategy, assessing primary studies, and synthesizing the evidence) involved in adopting systematic reviews in software engineering research and practice.

20.04.2010


4

1.1 What is so special about SE and ESE/EBSE?

• Still looking for the silver bullet; cf. Conradi’s own “odyssee”: PLs, ADT/OO, PEs, SE DBs, SCM, Process, Quality/SPI, Reuse/CBSE and gradually open source software (OSS) -- plus ESE/EBSE.

• SE is both “hard” and “soft“ – so tough to improve predictibly:

still 5+-2 SLOCs per person-hour for large software; 4+2+4 rule.

• Five trends:– Massive global reuse/CBSE by OSS - half million software components available– New cooperation and innovation processes around OSS– Flood of mobile/media applications w/ fancy and fast-moving technologies– Some new SE technologies: Agile methods, SOA, cloud computing, …– Empirical track: what works how, where and why – takes time w/ narrow focus – so

are we not studying the most relevant issues?

20.04.2010


5

1.2 Don’t forget relevance: PITL vs. PITS

• Relevance vs. rigor: PITL vs. PITS, case studies vs. experiments?

• Ex. top-10 project killers are PITL-related (Boehm89, Jones02, Charette05):

– Goals/commitment

– Project management

– Communication

– Bad requirements

– Changing requirements

– Estimation, tracking

– Product quality

– Technical personnel

– Development technology

– Insufficient software reuse

• So try mix agile + OSS + mobile … – and less experiments?

20.04.2010


6

1.3 The EBSE challenge

• How to synthesize relevant experience in a meta-analysis?

How to find and retrieve relevant experience – via data mining and semantic searches e.g. by Google or Case-Based reasoning. Costly, but effort not reported!

Ex. Medline portal receives 2000 new journal articles per day!

Industry: how to motivate for an empirical approach.And yet much turbulence in companies and their contact persons.

COMMENTS:• Status of available SLRs…..• Comments on SLR on Agile methods (Dybå&al.2007).• Detailed comments on SLR with a statistical meta-analysis of Pair

Programming (Hannay&al.2009).• Recommendations.

20.04.2010


7

2.1 The utility of existing SLRs, v1 • Controlled Experiments (Sjøberg2005):

– Found 121 such, or 2% of all empirical studies,– 20 repeated experiments:

• 7 of 11 had same results internally, but only• 2 of 9 had same results externally (unclear context factors?).

• Quasi-experiments (Kampenes20xx): need randomized subjects.• Effect size (Dybå200x): often too small to avoid Type-II errors, i.e.

need a β between 60 and 80.• Theory use (Hannay20xx): marginal, as most experiments have their

own agenda, and it is hardly not “feasible” for an experiment to be repeated N > 100 times. Cf. 3p physics papers with 150 co-authors, and similar in medicine with over 100,000 subjects.

• => Merge above insights in a shared recommendation!!• Means that future EBSE experiments must be more closely

coordinated – but by whom and how?

20.04.2010


8

2.2 The utility of existing SLRs, v2 • Estimation of effort/schedule (Jørgensen20xx): stable

30% overruns in the last 30 years. Domain expert will usually beat a general and formal tool.

• Knowledge management (Dingsøyr&Bjørnson20xx): not covering actual use.

• Agile methods (Dybå&Dingsøyr2007): see later.• PP meta-analysis (Hannay&al.2009): see later.• Open Source Software (Hauge&al.2010): Industrial OSS

adoption.

• I.e. a Scandinavian discipline??

20.04.2010


9

2.3 The utility of existing SLRs, v3

From the Univ. Keele portal: ++??add more

• Design inspections (Runeson20xx): generally more effective than unit testing.

• Testing (Juristo20xx): Unit testing showing diverse results.

• Software Architecture (Travassos20xx): ...

• Requirements engineering (20xx): ... ...

20.04.2010


10

2.4 Some quasi SLRs, v4

• SCM:– SCM overview (Mili and Mili and Mili)

– Version models (Conradi&Westfechtel2007)

– ICSE IMPACT paper on SCM (Estublier&al.2007)

• Reuse:– HP?? (Wayne Lim)

– Industrial reuse (Mohagheghi&Conradi2008)

• Inspections and testing:– ICSE IMPACT paper on defect discovery (Biffl&al.2007)

– Grady’s book??

– Gilb&Graham book??

20.04.2010


11

2.5 More conventional SLRs, v5

• ICSE: SE roadmap book (2000), FOSE roadmap book (2007).

IMPACT papers• ESE community:

laws: Endres/Rombach

Dagstuhl seminars 2007 (Basili et al.)

• Tutorials, keynotes, textbooks, industrial seminars

• Make comparison table of some of this?

20.04.2010


12

2.6 How to find more SLRs: what queries work?

Repeat for i=1..n over the entire Google universe (not limited to ACM, IEEE, …) the following query:

• Software and SE-subarena_i and

(Literature Review or Review or Overview or Survey of or Lessons learned or Experience With or …).

• Where SE_subarena_i: <empty>, engineering, requirements engineering, design, architecture, coding, implementation, programming, testing, project management, configuration management, …

• But much manual post-processing of the query results!

20.04.2010


13

3.1 Two SLRs on Agile Methods

• Agile manifesto: improvisation and quick changes, not plan-based and stiff waterfall regime.

• Agile technologies: XP (12 techniques incl. PP and TDD), Scrum, EVO, DSDM, rapid prototyping, …

• Much overselling, with an anti-metrics culture.• Recent challenges with global, distributed/mobile

teamwork.

• (Dybå&Dingsøyr2007): SLR of 33 primary research papers on Agile technologies.

• (Hannay&al.2009): Meta-analysis of 18 primary research papers on PP.

20.04.2010


14

3.2 SLR on Agile Methods

(Dybå&Dingsøyr2007): SLR w/ no clear conclusion of XP benefits, but with perceived positive attitudes.

Study Context Productivity of XP vs. traditional

Quality of XP vs. traditional

S07 (Dalcher05), experiment

4 diff. life-cycle models, stud., 1 y, 15 teams, t.size: 3-4

3 -> 13.1 LOC/h for XP, but XP gives 3.5x more code

No recorded (n SLR)

S10 (Ilieva04), case study

Two similar projects, prof., 900h, t.size: 4

3.8 -> 5.4 LOC/h for XP, mostly in 1st of 3 releases

13% fewer defect reports by XP

S14 (Layman04), case study

Reimpl.proj., prof., 3.5 months, w/ 3 of 10 prev. team-memb.

300 -> 340 LOC/month for XP

Much better pre/post-delivery quality/defects by XP

S15 (Macias03), experiment

10 XP vs. 10 trad. teams, stud., 1 semester, t.size: 4-5

No difference in productivity

No difference in quality/defects

S32 (Wellington05), experiment

One XP and one trad. team, stud.,1 semester, t.size: 16

Trad. team: 78% more code than XP

XP team delivered much better quality

20.04.2010


15

3.3 Reflections on the two Agile SLRs

• In Agile: Project management and requirements engineering most importent – not PITS-inspired factors like productivity or defect density.

• In first SLR on Agile: 3 XP (PP?) experiments in 2003-2005.

• In second SLR on PP: 18 other experiments, 9 from 1998-2005, 9 from 2006-2007.

20.04.2010


16

3.4 SLR on Pair Programming (PP): intro

• PP: One of the 12 methods in XP, with roots in generous teamwork elsewhere – cf. book by (Weinberg1971).

• So, what do we know on PP: either less effort, shorter duration, better “quality”– or just happier people with more shared knowledge and experience, learning from another.

• Moderate effects on programming-related factors: +- 15%.

• (Hannay&al.2009): A statistical meta-analysis of PP experiments in 18 primary research papers. Technical emphasis: quality, duration, effort – not longitudinal studies of social/cognitive factors to promote learning.

20.04.2010


17 ”3.5 SLR must consider ”Project-triangle” – three competing variables or aspects

Result:

Functionality / Quality

”Project-

Triangle”

Effort Duration/ deadline

20.04.2010


18

3.6 SLRs on Pair Programming (PP) – in detail

• No common experiment organization or treatment wrt.:• Goals and objectives.• Team composition: software expertise (student vs. practitioner, programming

skills), soloists vs. pairs, application task, use of external reviewers, …• Software technology: Eclipse; UML; Java, C, C++• Randomized vs. self-chosen team composition.• And here: no social/cognitive aspects studied.

• Tree major dependent variables: – Quality (hmm?) – see next slides

– Duration (hours) – elapsed time

– Effort (person-hours) – often just named hours

– No volume / productivity measures (SLOC, FP, …), calculated by tools.

14.09.2010


19

3.7 SLRs on PP – in detailStudy Quality (14 non-empty slots) Duration (11 non-empty slots) Effort 11 non-empty slots)

P1 Arisholm et al. (2007) Q D E

P2 Baheti et al. (2002) Q D -

P3 Canfora et al. (2005) - Q missing D E

P4 Canfora et al. (2007) Q D E

P5 Domino et al. (2007) Q (not elsewhere) - -

P6 Heiberg et al. (2003) Q D -

P7 Madeyski (2006) Q (not elsewhere) - -

P8 Madeyski (2007) Q (not elsewhere) - -

P9 Müller (2005) Q D E

P10 Müller (2006) Q D E

P11 Nawrocki & Wojciechowski (2001) - Q missing D E

P12 Nosek (1998) Q D E

P13 Phongpaibul & Boehm (2006) Q - E

P14 Phongpaibul & Boehm (2007) Q - E

P15 Rostaher & Hericko (2002) - Q missing D (only mentioned here) -

P16 Vanhanen & Lassenius (2005)++ reduce col. height??

- Q missing E (only mentioned here)

P17 Williams et al. (2002) Q (not elsewhere) - -

P18 Xu & Rajlich (2006) Q D E

14.09.2010


20

3.8 SLRs on PP – the Quality resultsStudy Quality (14 non-empty slots) Effort 11 non-empty slots)

P1 Arisholm et al. (2007) Q P10 Müller (2006) Q

P2 Baheti et al. (2002) Q P11 Nawrocki & Wojciechowski (2001) - Q missing

P3 Canfora et al. (2005) - Q missing P12 Nosek (1998) Q

P4 Canfora et al. (2007) Q P13 Phongpaibul & Boehm (2006) Q

P5 Domino et al. (2007) Q (not elsewhere) P14 Phongpaibul & Boehm (2007) Q

P6 Heiberg et al. (2003) Q P15 Rostaher & Hericko (2002) - Q missing

P7 Madeyski (2006) Q (not elsewhere) P16 Vanhanen & Lassenius (2005)++ reduce col. height??

- Q missing

P8 Madeyski (2007) Q (not elsewhere) P17 Williams et al. (2002) Q (not elsewhere)

P9 Müller (2005) Q P18 Xu & Rajlich (2006) Q

20.04.2010


21

4.1 The future: how to consider OSS?

• OSS: changes the whole paradigm of software development and associated economic patterns and interactions – by cooperative, distributed innovation.

• Norwegian software-intensive companies and public institutions must undergo this economic and cultural revolution:

– novel Innovation Models: new products and services– novel Business Models: make money on these

• Need professional partnership communities to establish and evolve their needed software, itself being OSS.

• Move away from “unpredictable” volonteers working for free.• OSS: adopted by private and public policy makers: e.g., IBM and

Sun Microsystems, IKT-Norge, …, Skattedirektoratet, KS, …• St.meld. 17 (2006-07) by Norwegian Government recommends

OSS and open standards.• www.FriProg.no competence center. Also www.FriNett.no (NFR).

20.04.2010


22

4.2 From linear to networked paradigm

Before:

Now:

20.04.2010


23

4.3 Networked development actors

Emerging network paradigm for OSS will affect:• Long-term vs. short-term planning,• Cross-company cooperation and dependence:

(re)negotiation vs. dictate, • Control mechanisms: many risky parts, • (Open) Innovation: new combinations, spin-offs, …• Legal: IPRs and licensing, • Economical: new business models,• Marketing: close to development,• => Requirement Engineering reborn!• See [Ayala09] [Hauge10] [Lindman09].

20.04.2010


24 4.4 OSS Ecosystem for support - by Friprog-.no

20.04.2010


25

4.5 Summary of X-disciplinary aspects in OSS

• Situated selection of promising components, pragmatic approach, later auto-generated ontologies for functionality properties.

• New ways of cooperation and sharing: new work organization, very interesting paradigm – “The world is flat”, global cooperation. Ex. Cooperate on shared, domain-specific middleware.

• Open Innovation model: altruism gives the highest long-term profit, by combining several open technologies into new products and services [Browning08] [Chesbrough03] [Chesbrough03a] [Chesbrough06].

• Business model: and making money of this - proven to work!– Apache: open and shared source supported by a cooperative

foundation (“IBM”); separate payable services. – eZ: dual model with free previous version, payable current version

plus services and support.

25

20.04.2010


26

6. References (1)

• [Ayala09] C. Ayala, Ø. Hauge, R. Conradi, X. Franch, et al.: ”Challenges Using the OSS Component Marketplace in the Industry”, Proc. OSS'09.

• [Bowker99] Geoffrey C. Bowker and Susan Leigh Star: ”Sorting Things Out - Classification and Its Consequences”, MIT Press, Boston, USA, Oct. 1999, 389 pages, ISBN 978-0-262-02461-7.

• [Browning08] L. D. Browning, Alf Steinar Sætre, K. Stephens and J.-O. Sørnes: ”Information and Communication Technologies in Action: Linking Theory and Narratives of Practice”. New York: Routledge, 2008.

• [Chesbrough03] Henry W. Chesbrough: OPEN Innovation: The new imperative for creating and profiting from technology, Harvard Business School Press, 2003.

• [Chesbrough03a] Henry W. Chesbrough: “The era of open innovation”, MIT Sloan Management Review, 44(3):35-41 (2003).

• [Chesbrough06] Henry W. Chesbrough: OPEN Business Models – How to Thrive in the New Innovation Landscape, Harvard Business School Press, 2006.

• [Hauge10] Ø. Hauge, D. Cruzes, R. Conradi et al.:"Risks and Risk Mitigation in Open Source Software Adoption", Proc. OSS'2010.

• [Lindman09] Juho Lindman et al.: “Beyond the Business Model: Incentives for Organizations to Publish Software Source Code”, Proc. OSS'2009.

1 14.Sep.2010 Systematic Literary Reviews: Experience and Advice Systematic Literary Reviews:...

Documents

Transcript of 1 14.Sep.2010 Systematic Literary Reviews: Experience and Advice Systematic Literary Reviews:...