Issues and challenges of Agile Development: Some perspectives from Academic Research

1

Dr. Sridhar Nerur

University of Texas at Arlington

Presentation OutlinePresentation Outline

Are Two Heads Better Than One – Pair Programming Study

Pairs versus Individuals on Design Tasks: A Distributed Cognition Perspective

Case study of XP (Extreme Programming) adoption

Other research projects

2

Are Two Heads Better Than One?Are Two Heads Better Than One?A Pair Programming Study A Pair Programming Study

(Balijepally, Mahapatra, Nerur & (Balijepally, Mahapatra, Nerur &

Price, 2009)Price, 2009) Methodology development driven by

practitionersEffectiveness reports were anecdotal or

experientialNo systematic effort to provide

theoretical explanation of practicesInconsistent findings on the

effectiveness of pair programming Lack of theoretical and methodological

rigor (e.g., Nosek 1998, Williams 2000)

3

Effectiveness of Pair Effectiveness of Pair ProgrammingProgramming

Research Objectives◦To examine the efficacy of pairs vis-

à-vis individuals in a programming task

◦To study the effect of pairing on a programmer’s task satisfaction and confidence in performance

◦To understand how the above are affected by programming task complexity

4

Theoretical BackgroundTheoretical BackgroundTask Typology

◦Based on Steiner’s typology (1972) programming tasks are unitary (not divisible among group members), optimizing (quality is more important than speed), and disjunctive (group must arrive at a single solution)

◦Tasks may be intellective or judgmental ( Laughlin and Ellis 1986)

◦Programming tasks are relatively intellective with high solution demonstrability

5

Theoretical BackgroundTheoretical BackgroundNominal Pair (Laughlin et al.

2002)◦A nominal pair is created by

randomly pairing individuals who worked alone

◦Comparing the performance of a collaborating pair with the best and the second best members of a nominal pair provides better insight into group performance

6

Theoretical BackgroundTheoretical BackgroundGroup Performance

◦Group members have access to a larger pool of resources (Flor and Hutchins 1991; Hinsz et al. 1997)

◦Group processes may suffer from process losses due to social loafing (Karau and Williams 1982)

◦Performance is contingent on task type◦With intellective tasks, groups outperform

an average individual but rarely perform better than the best individual (Hill 1982, Laughlin et al. 2003)

◦Quest for assembly bonus effect (when the group outperforms the best individual) still continues (Collins and Guetzkow 1964)

7

Research ModelResearch Model

8

H1

H3

H2

H4

Perceptual Outcomes

Task

Complexity

Programming Setting

Individuals or Pair*

Satisfaction

Software quality

Confidence in Performance*1 – Best in a nominal pair

2 – Second-best in a nominal pair

3 – Collaborating pair

Research Method and Data Research Method and Data AnalysisAnalysis

Lab Experiment2 (Individual vs. pair) X 2(Simple Vs.

Complex task) factorial design120 student subjects (30 pairs and 60

individuals)Pilot done to check experimental protocol

and to determine time needed for the experimental task

15 minutes for the warm up task and 2 hours for the experimental task

MANCOVA followed by ANCOVA for each dependent measure

Programming ability used as a covariate9

ResultsResults

10

Hypothesis Result ExplanationSoftware Quality

1 Programming performance of a collaborating pair measured in terms of software quality will be higher than the performance of the second-best member of a nominal pair

Supported (p < 0.01)

A collaborating pair outperforms the second-best member of an independently working nominal pair.

Satisfaction2 A. The mean satisfaction with

the programming task of a collaborating pair will exceed the level reported by the best member of a nominal pair.

Supported(p=0.04)

Collaborating pairs are more satisfied than each member of an independently working nominal pair.

B. The mean satisfaction with the programming task of a collaborating pair will exceed the level reported by the second-best member of a nominal pair.

Supported (p = 0.02)

ResultsResults

11

Confidence in Performance

A. The mean confidence in performance of a collaborating programming pair will exceed the level reported by the best member of a nominal pair.

Not Supported(p = 0.15)

Collaborating pairs are only as confident of their performance as the best member of an independently working nominal pair, but more confident than its second-best member.

B. The mean confidence in performance of a collaborating programming pair will exceed the level reported by the second-best member of a nominal pair.

Supported (p < 0.01)

Moderating Effect of Task ComplexityTask complexity affects the performance, in terms of software quality, of the best member of a nominal pair more adversely than that of a collaborating pair

Not Supported (p = 0.18)

Task complexity affects the performance of collaborating pairs and nominal pairs in similar ways.

Hypothesis Result Explanation

ContributionsContributionsOne of the earliest theoretically

grounded empirical research studies on pair programming.

Offers insight into performance of programming pairs

Examines the contingent effect of task complexity on programmer’s performance

Creates a new benchmark to evaluate pair programming performance through the use of nominal pairs

12

Pairs vs Individuals on Pairs vs Individuals on Design Tasks (Mangalaraj, Design Tasks (Mangalaraj,

Nerur, Mahapatra, & Price – Nerur, Mahapatra, & Price – under review)under review)

Motivation◦Research on pair designing is sparse

◦Design tasks are cognitively more challenging

◦Very limited empirical research on design patterns in tasks that are design-centric

13

Theoretical FoundationTheoretical FoundationAccording to the theory of

distributed cognition, cognition is distributed across problem-solvers (i.e., social interactions), artifacts (i.e., embodied cognition), and time

Two aspects of distributed cognition that are relevant to our research: social and structural (or embodied)

We use pairs (social) and design patterns (cognitive artifacts) in our study

14

Research ObjectivesResearch ObjectivesTo study the role of design

patterns in facilitating the solution of a software design problem

To assess the effectiveness of pairs vs. individuals working on a software design task

15

Theoretical BackgroundTheoretical BackgroundDesign problems are ill structured,

ambiguous, and have no pre specified solution path (Simon 1973)

Key activities in solving a design problem are◦ Problem structuring◦ Searching through the solution space

Problem structuring is facilitated by availability of partial solutions and external representation media (Guindon 1990, Zhang and Norman 1994)

Expert developers use their design experience (stored as design schemas) to constrain the search space (Guindon et al. 1987)

16

Research ModelResearch Model

17

Individual/Pair

With design patterns/without design patterns

Outcome Variables

Solution quality

Task completion time

Task satisfaction

H4, H5a, H5b, H6

H1, H2, H3

Research MethodResearch Method

18

Research Method & Data Research Method & Data AnalysisAnalysisLab Experiment2 (Individual vs. pair) X 2 (Design Pattern

Vs. No Design Pattern) factorial design92 professionals with no prior pattern

experiencePilot done to check experimental protocol

and to determine time needed for the experimental task

Participants were provided a free seminar on patterns

ANOVA/ANCOVA used for analysisObject-oriented programming experience

was used as a covariate

19

ResultsResults

20

Hypothesis ResultH1: Quality of the design solution will be higher when design patterns are used during software design.

Supported

H2: Time taken to complete a design task will be shorter when design patterns are used.

Supported

H3: Task satisfaction will be higher when design patterns are used in a software design task.

Supported

H4: Solution quality of collaborating pairs will be higher than that of the second best member of a nominal pair in a software design task.

Supported

H5a: Time taken by collaborating pairs to complete a software design task will be longer than that of the best member of a nominal pair.

H5b: Time taken by collaborating pairs to complete a software design task will be longer than that of the second-best member of a nominal pair.

Not Supported

Supported

H6: Task satisfaction will be higher among collaborating pairs when compared with the average level of task satisfaction of nominal pairs.

Supported

ContributionsContributionsOne of the earliest theoretically

grounded empirical research on the impact of design patterns on design performance

Demonstrates the benefits of using design patterns ◦ Improves design quality◦ Reduces task completion time◦ Enhances developer satisfaction

Consistent with group literature, pairs were found to be better than second-best individuals of nominal pairs

21

SynthesisSynthesisTheoretically grounded research provides

a deeper understanding of the performance of pairs in software development

Studies demonstrate that pairs outperform second-best individuals of nominal pairs regardless of task type and experience level.

This confirms the long-standing research in group theory, but is counter to the dominant belief in the agile community

The use of nominal groups in this stream of research is a novelty. It permits a more fine-grained analysis that is more meaningful to practitioners

22

ImplicationsImplicationsPairing should be used judiciously

◦Performance gains are realized when low-performing rather than best developers are paired

The use of design patterns is beneficial

Our studies have established a benchmark for theoretically grounded research in software engineering

23

Table 1: Core XP practices adopted by ABC Corp.XP Practice Definition* Relative

Importance

Release Planning

Part of the planning game (original XP practice) in which the customer specifies the requirements for the project and creates the release plan based on the developer’s cost/time estimates.

0.139

Customer Team

Customer is part of the development team. He/she determines requirements and performs acceptance tests.

0.126

Collective Ownership

Members of the project team collectively own the code and any one can change any of the code at any time.

0.122

Pair Programming

Pair programming involves two programmers working side by side at one computer, collaborating on the design, coding and testing on a continuous basis.

0.109

Iteration Planning

A part of the planning game that involves planning for the two-week iterations.

0.100

Sustainable Pace

Teams work at a sustainable pace, usually 40 hour per week. Team members work overtime only when needed.

0.098

Case Study – Problems with Adoption ofXP practices at ABC Corp.

Refactoring Ongoing redesign of software to improve its responsiveness to change.

0.049

Simple Design Simplest design is done for the functionality that has been defined and not in anticipation of future requirements.

0.046

Unit Tests Involves the development of the test module prior to writing the code.

0.040

Small Releases

Software is released in small increments that address the most valuable business requirements.

0.040

Acceptance Tests

Customer writes the acceptance tests. Testing is largely automated.

0.037

Continuous Integration

Builds are done every few hours to lessen code integration problems.

0.028

Metaphor It is the common vision of how the system works. The metaphor is intended to provide a broad view of the project’s goal.

0.023

Open Workspace

Developers work in an open lab-like environment with no cubicles.

0.022

Coding Standards

Rules that everyone in the team must follow in writing code. 0.020

* These definitions are based on the following sources: (Highsmith, 2002; Beck, 2000; Jeffries, 2001)

Table 2: Comparison of the two projectsCharacteristics Project X Project YApplication type New project Maintenance and enhancementLanguage/Technologies

Java C, C++, Java, Motif

Project size Approximately 30,000 lines of code.

Several hundred thousand lines, with more than 10 year old code base.

Tools used IntelliJ and CruiseControl

C++ editor, SlickEdit, and ClearCase

Team Size 6 members 7 membersXP Score (Maximum 4.0)

2.64 1.58

Table 3: Comparison of core XP practices across the two projects.XP Practice Project X Project YRelease and iteration planning

Release planning is done periodically.

Iteration planning involves all members of the team. Developers have the freedom to choose stories they wish to work on.

Release planning is done periodically.

Though iteration planning involves all members of the team, stories are assigned based on skill specialization.

Customer team

Proxy customer writes user stories and collaborates in acceptance testing.

No dedicated customer/proxy customer is involved.

Collective ownership

Team members are collectively responsible for code development and maintenance.

Limited collective code ownership due to skill specialization.

Pair-programming

It is done to a greater extent.

In general, team members seemed to enjoy pair-programming and they hold positive attitude towards it.

Pair programming was seldom done.

Examples of differences between the two projects

Individual Factors•Attitude towards XP•Knowledge of XP

Team Factors•Task management•Team leadership

Technology Factors•Compatibility•Tools support

Task Factors•Type of application•Project size

Environmental Factors•Budget constraints•Time constraints

Acceptance of Extreme Programming

Figure 3. Extreme Programming Acceptance Framework

Other research studiesOther research studiesEfficacy of pairs vs individuals with

and without Test-Driven Development

Impact of problem representation on solution quality

Understanding cognitive processes – mental models, cognitive ability

Agile Project Management challenges

29

30

Questions??