GRID Computing: Meeting CERNs Future Computing Challenges

Agile Software Development using ScrumDerek MathiesonGroup LeaderAdvanced Information SystemCERN Geneva, SwitzerlandCERN European Organization for Nuclear ResearchGS Department Administrative Information ServicesCERNGS-AIS1Speaker BackgroundCurrently:Group Leader of AIS since January 2010Previously:Section Leader EDH (2000)Software Developer at CERN (1994)Software Developer at SSC in Texas (1992)CERN Fellow (1990)CERN Technical Student (1989)Software Developer (1986)Frequently Asked Questions on WWW

FREQUENTLY ASKED QUESTIONS ON W3An FAQ list is really a cop-out from managed information. You should be able to find everything you want to know by browsing from the WWW project page, as everything should be arranged in a logical way. Here though are things which maybe didn't fit into the structure, with pointers to the answers which maybe did. Its an experiment, started May 92. The questioners are anonymous. I am just starting: how do I find out more?[1] How does www keep track of the available servers?[2] How does W3 compare with WAIS and Gopher[3] ? How do I create my own server[4] ?1-10, Up, for more, Quit, or Help:CERNGS-AIS2AgendaTraditional Software DevelopmentWhat is Agile?The Agile ManifestoAgile MethodsSCRUMSCRUM @ CERNCERNGS-AISTraditional DevelopmentCERNGS-AISCERNGS-AISWaterfall ModelRequirementsDesignImplementationVerificationMaintenanceTimeCERNGS-AISThe first formal description of the waterfall model is often cited as a 1970 article by Winston W. Royce, though Royce did not use the term "waterfall" in this article. Royce presented this model as an example of a flawed, non-working model.

For example, in one key study of 1,027 IT projects in the United Kingdom, Thomas [2001] reported that scope management related to attempting waterfall practices was the single largest contributing factor for failure. The studys conclusion:

[T]his suggests that . . . the approach of full requirements denition, followed by a long gap before those requirements are delivered is nolonger appropriate. The high ranking of changing business requirements suggests that any assumption that there will be little signicant change to requirements once they have been documented is fundamentally awed, and that spending signicant time and effort in ghting the to maintain the maximum level is inappropriate.

5Waterfall in PracticeSafeguard - Ballistic Missile Defence System

The project was delivered according to specificationsCost: $25 Billion (not adjusted)1969-1975, 5407 person yearsOperational for 133 days Project terminated in 1978By the time the 6-year anti-missile system project was completed, the new missiles were faster than the antimissile missilesCERNGS-AISWaterfall in Practice

Safeguard - Ballistic Missile Defence System

By the time the 6-year anti-missile system project was completed, the new missiles were faster than the anti-missile missilesCERNGS-AISCost: $25 Billion (not adjusted)1969-1975, 5407 person yearsOperational for 133 days - Project terminated in 1978

By the time the 6-year anti-missile system project was completed, the new missiles were faster than the anti-missile missiles

7Spiral Model1.Determineobjectives2.Identifyandresolverisks3.DevelopmentandTest4.PlanthenextiterationProgressCumulativecostRequirementsplanConceptofoperationConceptofrequirementsRequirementsPrototype 1Prototype 2OperationalPrototypeDraftDetaileddesignCodeIntegrationTestImplementationReleaseTestplanVerification&ValidationDevelopmentplanVerification&ValidationReviewCERNGS-AISThe spiral model was defined by Barry Boehm in his 1986 article "A Spiral Model of Software Development and Enhancement".

The US military has adopted the spiral model for its Future Combat Systems program. The FCS project was cancelled after six years (2003 - 2009), it had a 2 year iteration (spiral). 8Why Software Is Different?Software is ComplexTechnology Changes RapidlyRequirements are IncompleteChange is Considered EasySoftware Design is ResearchConstruction is Actually DesignChange is Inevitable

CERNGS-AISSoftware is ComplexComputer programs are the most intricate, delicately balanced and finely interwoven of all the products of human industry to date[James Gleick 1992]CERNGS-AISSoftware is ComplexCERNGS-AISSoftware is ComplexCERNGS-AISSoftware is ComplexCERNGS-AISSoftware is ComplexCERNGS-AISWhy Software Is Different?Software is ComplexTechnology Changes RapidlyRequirements are IncompleteChange is Considered EasySoftware Design is ResearchConstruction is Actually DesignChange is Inevitable

CERNGS-AISTechnology Changes Rapidly

CERNGS-AISRequirements Are IncompleteCERNGS-AISAn ExperimentCERNGS-AISCERNGS-AISProject DescriptionPersonal Transport Device

Usable outsideWeather proof.Reasonably StrongUsable for several years.Stable, relatively easy to useProbably 4 wheels?

CERNGS-AISImplementation

?CERNGS-AISMany projects fail because their developers fail to build the right thingGrady Booch CERNGS-AISCERNGS-AISWhy Software Is Different?Software is ComplexTechnology Changes RapidlyRequirements are IncompleteChange is Considered EasySoftware Design is ResearchConstruction is Actually DesignChange is Inevitable

CERNGS-AISCost of Change Curve

CERNGS-AISWhy Software Is Different?Software is ComplexTechnology Changes RapidlyRequirements are IncompleteChange is Considered EasySoftware Design is ResearchConstruction is Actually DesignChange is Inevitable

CERNGS-AISIntroducing Agile DevelopmentCERNGS-AISCERNGS-AISWhat is Agile?

CERNGS-AISWhat is Agile?Agile:Having the faculty of quick motion; nimble, active, ready.

Agile software development:A group of software development methodologies based on iterative and incremental development, where requirements and solutions evolve through collaboration between self-organizing, cross-functional teams.(Oxford English Dictionary)(Wikipedia)CERNGS-AIS27Why Agile?CERNGS-AISCERNGS-AISThe Iron TriangleScopeQualitySchedulePickTwoTraditionalDevelopmentCERNGS-AISFred Brooks The Mythical Man-Month (1975):Adding manpower to a late software project makes it later. - Brooks's law

The Chaos Report is the first survey made by the Standish Group.29Software DeliveryThe Chaos ReportStandish Group (1995)CERNGS-AIS31.1% of projects will be canceled before completion. 52.7% of projects will cost over 189% of their original estimates.16.2% of software projects are completed on-time and on-budget

30The Agile Manifesto (2001)Early and continuous delivery of valuable software Welcome ChangeDeliver OftenCustomers and developers must work togetherBest possible people, tools and workplaceEmphasis on face-to-face communicationWorking software is the best measure of progressConstant sustainable progressFocus on technical excellence and good design SimplicitySelf-organizing teamsRegular reflection on improvementsCERNGS-AISIn February 2001, 17 software developers met at a ski resort in Snowbird, Utah, to discuss lightweight development methods. They published the "Manifesto for Agile Software Development" to define the approach now known as agile software development. Some of the manifesto's authors formed the Agile Alliance, a non-profit organization that promotes software development according to the manifesto's principles.31The 4 Agile ValuesIndividuals and interactions over processes and toolsWorking software over comprehensive documentationCustomer collaboration over contract negotiationResponding to change over following a planCERNGS-AISThe Iron TriangleScopeQualitySchedulePickTwoTraditionalDevelopmentAgileDevelopmentCERNGS-AISFred Brooks The Mythical Man-Month (1975):Adding manpower to a late software project makes it later. - Brooks's law

The Chaos Report is the first survey made by the Standish Group.33Iterative DevelopmentRegular releases to customerTime-boxingNormally 2 6 weeksAdjust design as the project progresses

RequirementsPlanningAnalysis & DesignImplementationTestingEvaluationInitialPlanningDeploymentCERNGS-AISAgile MethodsScrum Feature Driven Development (FDD)LeanExtreme Programming (XP)RUPKanbanCERNGS-AISIn 1995, Sutherland and Schwaber jointly presented a paper describing Scrum at OOPSLA 95 in Austin, Texas, its first public appearance. Schwaber and Sutherland collaborated during the following years to merge the above writings, their experiences, and industry best practices into what is now known as Scrum. In 2001, Schwaber teamed up with Mike Beedle to describe the method in the book "Agile Software Development with Scrum".

35SCRUMCERNGS-AISCERNGS-AISWhat is SCRUM?

CERNGS-AISWhat is SCRUM?Scrum is a framework for iterative, incremental development using cross-functional, self-managing teams. It is built on industry best practices, lean thinking, and empirical process control.Ken Schwaber, 2006co-creator of SCRUMCERNGS-AISMethod ComparisonWaterfallSpiralIterativeScrumDefined processesRequiredRequiredRequiredPlanning & Closure onlyFinal productDetermined during planningFixed during planningSet during projectSet during projectProject costDetermined during planningPartially variableSet during projectSet during projectCompletion dateDetermined during planningPartially variableSet during projectSet during projectResponsiveness to environmentPlanning onlyPlanning primarilyAt end of each iterationThroughoutTeam flexibility,creativityLimited - cookbook approachUnlimited during iterationsKnowledge transferTraining prior to projectTeamwork during projectProbability ofsuccessLowMedium lowMediumHighJeff Sutherland, The Scrum Papers 2010co-creator of SCRUMCERNGS-AISSCRUM in Pictures

CERNGS-AIS40SCRUM in PracticeCERNGS-AISCERNGS-AISEDH Statistics14,500 active users25k Documents/month60k Signatures/month-5,000 10,000 15,000 20,000 25,000 Documents per monthDistinct Users per month0.000.501.001.502.002.503.00-10,000 20,000 30,000 40,000 50,000 60,000 Signatures per monthRatio Signatures/DocumentCERNGS-AISEDH Development Team4 Staff3 Project Associates1 Fellows1 or 2 Students (9 month contract)

1.8 million lines of code~1000 3rd line support calls/year

CERNGS-AISEDH Development B.C.B.Constant Developer InterruptionsLow efficiencyDelivery was often latePoor estimation many unknownsScope CreepSpecification constantly changingEverything is FreeSome features never used

Before SCRUMC.CERNGS-AISSCRUM VocabularyProduct OwnerProduct BacklogScrum TeamSprint PlanningScrum MasterDaily ScrumSprint BacklogSprint Review Meeting

CERNGS-AIS45Chickens and Pigs...

InvolvedCommitted

CERNGS-AISA number of roles are defined in Scrum. All roles fall into two distinct groupspigs and chickensbased on the nature of their involvement in the development process. These groups get their names from a joke about a pig and a chicken opening a restaurant:A pig and a chicken are walking down a road. The chicken looks at the pig and says, Hey, why dont we open a restaurant? The pig looks back at the chicken and says, Good idea, what do you want to call it? The chicken thinks about it and says, Why dont we call it Ham and Eggs? I dont think so, says the pig, Id be committed, but youd only be involved.So the pigs are committed to building software regularly and frequently, while everyone else is a chickeninterested in the project but really indifferent because if it fails theyre not the pigsthat is, they werent the ones that committed to doing it. The needs, desires, ideas and influences of the chicken roles are taken into account, but are not in any way allowed to get in the way of the actual Scrum project.

46The Product OwnerTypically a Product Manager, Internal Customer, etc.Responsible for:Providing and maintain a prioritised Product BacklogResponsive to questions during a sprint

CERNGS-AIS47The Product BacklogA list of all desired work on the projectUsually a combination of story-based work: let user search and replacetask-based work: improve exception handlingPrioritised by the Product OwnerPriority should be (ideally) based on Business ValueCost assigned by the Scrum TeamCERNGS-AIS48The Scrum TeamTeams are self-organisingCross-functionalQA, Programmers, UI Designers, Technical Writers, etc.Assign Cost to each Item on the Product Backlog

Commit to the Sprint Goal

CERNGS-AIS49Agile EstimationPlanning Poker

CERNGS-AISSpeed up the estimation process by limiting the number of choices (i.e. number of cards). Avoid a false sense of accuracy for high estimates.Encourage the team to split large stories into smaller ones.50The Scrum MasterResponsible for enacting Scrum values and practices (The Process)Main job is to remove obstacles which affect the teamTypical obstacles could be:My ____ broke and I need a new one. I still haven't got the software I ordered. I need help debugging a problem with ____. I'm struggling to learn ____ and would like help. The GL has asked me to work on something else "for a day or two."

CERNGS-AIS51The Sprint Planning MeetingAttended by:Product Owner, Scrum Master, Scrum Team, and any interested and appropriate management or customer representatives.

Product Owner describes the highest priority features to the team.

Collectively the Scrum Team and the Product Owner define a Sprint GoalCERNGS-AIS52The Sprint GoalA short theme for the sprint:

The SCRUM Team commit to this goal.Create Working Form.Implement Workflow.Create Reports.Implement Bulk Emailing.CERNGS-AIS53The SprintFixed Time-Box (we chose 2 weeks)

Product is designed, coded, and tested during the sprint

Daily Scrum Meetings

Produce demonstratable, working, new functionality.CERNGS-AIS54The Daily ScrumAnyone InvitedLed by Scrum Master15 minutes, every dayNot for problem solving

Three questions:What did you do yesterdayWhat will you do today?What obstacles are in your way?

CERNGS-AIS55Process repeats...2 Weeks Pass

CERNGS-AISCERNGS-AISThe Sprint Review MeetingTeam presents what it accomplished during the sprintTypically takes the form of a demo of new features or underlying architectureParticipantsManagementProduct OwnerOther engineersCERNGS-AIS57Release SprintConcentrate on preparing for production:No new featuresLast minute bugs, typos, layout issues, etc.Translation (if not done already)Desktop IconsCommunication, Bulletin Articles, etc.Sprint 1Sprint 2Sprint 3ReleaseSprintCERNGS-AIS58Scrum value driven not plan drivenEmpower lean teams to deliver more software earlier with higher quality.Demonstrate working features to the customer early and often so the customer can inspect progress and prioritize change.Deliver exactly what the client wants by directly involving the customer in the development process.Provide maximum business value to the customer by responding to changing priorities in real time.

Jeff Sutherland, 2007co-creator of SCRUMCERNGS-AISSCRUM in IndustryThe most profitable software product ever created (Google Adwords) is powered by Scrum.The most productive large project with over a million lines of code (SirsiDynix) used a ... Scrum implementation.Jeff Sutherland, 2010co-creator of SCRUM

CERNGS-AISSCRUM in IndustryState of the IT Union Survey, July 2009Agile Adoption Survey, March 2008Organizations using Agile methodsCERNGS-AISSCRUM in Industry10.9%6.0%3.8%3.4%2.9%2.1%1.8%1.6%1.3%0.9%0.3%0.2%0.2%16.3%2.7%1.6%8.4%2.5%2.5%30.6%ScrumAgile ModelingFeature-driven development (FDD)Test-driven development (TDD)eXtreme Programming (XP)Lean developmentMicrosoft Solutions Framework (MSF) for AgileAgile Data MethodAdaptive Software Development (ASD)Six SigmaCrystalBehavior-driven development (BDD)Dynamic Systems Development Method (DSDM)Do not use a formal process methodologyIterative developmentRational Unified Process (RUP)SpiralWaterfallCapability Maturity Model Integration (CMMI)ISO 9000Waterfall, 13%Iterative, 21%Agile, 35%Base: 1,298 IT professionalsSource: Forrester Research, Inc. 2009CERNGS-AISVisible benefits of SCRUMTime-Boxed:Maximum investment known up-frontTackle most valuable features firstFocus on working, tested, documented product featuresCERNGS-AIS63ConclusionsProduct Owner:Active ParticipantCan see product evolveKnow the cost of each feature

Good Product Owners can be hard to findCERNGS-AIS64ConclusionsTeam:Work closely with Product OwnerKnow the Value of each FeatureKnown Start and End of ProjectEfficient, highly focused development

Strong Team SpiritCERNGS-AIS65Why SCRUM?What I wanted:Manage Product RequirementsProvide Visibility to ClientsBetter manage developer timeA more repeatable development processWhat developers wanted:Something lightTask managementCommunicationCERNGS-AISWhat did we adapt?2 week SprintRelease SprintNot everyone SCRUMsFull time support staffTechnology(Almost) Everyone does support tooSome people have several rolesCERNGS-AISImplementation BarriersSome clients insist all features must be in final productDaily Sprint meetings = InterruptionPoor Product OwnerNot final decision makerDoesnt want to be involvedMore than one (that dont agree!)

ScopeQualitySchedulePickTwoCERNGS-AISLessons LearnedBe careful of the choice of Product OwnerUse tools to simplify adminExcel, whiteboards, ScrumWorks, JIRA,

CERNGS-AISProduct Backlog Window

CERNGS-AISSprint Detail Window

CERNGS-AISWeb Client

CERNGS-AIS

JIRA + GreenHopper

CERNGS-AISJira IDE Integration (IDEA)

CERNGS-AISDoes it Increase Productivity?Probably

Did it make development work easier?YesCommunication is betterEstimates are betterPlanning is easierCustomers are happierCERNGS-AISThank You

CERNGS-AIS76Yes butI like writing software, but I dont like doing the other development stuff which we are not forced to do here.

SCRUM lets you:Focus on valuable development

Use tools to minimise admin

CERNGS-AISYes butIt might help, but we have multiple projects per person.

So do weIts simpler to have only one, but sometime schedules dont allowTime-boxing helps to reduce parallel activities.CERNGS-AISYes butManagement wont agree

SCRUM offers:Better PlanningDeadlines metMinimise unnecessary developmentHappy ClientsCERNGS-AISYes butOur clients wont agreeTricky oneSCRUM needs Client commitmentSCRUM exposes the cost of featuresSCRUM makes the client chooseIn return they get:TransparencyLicense to change their mindsMet deadlinesCERNGS-AISYes butI like X from Scrum, but not Y, I might try X.

Do X!

CERNGS-AISYes butYou are trying to get us to work more for less! No way!

SCRUM lets you:Focus on useful work

CERNGS-AISYes butOur project X is special and not industry so we dont need a process.

CERNGS-AISThank YouCERNGS-AISCERNGS-AIS84