1of32

Agility is a Stable Requirement

Graham McLeod inspired.orgDecember 2017

Abstract

Changeintechnology,businessandsocietyiseverpresentandaccelerating.Itisveryunlikelytoslowdown,thusitisastablerequirement.Ourmethodsofdoingstrategy,devisingfuturearchitecturesanddeliveringsystemscapabilitiesinsupportofbusinessprocesses,capabilitiesanddeliveryofservicesandproductsthereforneedtoaddressthis.AgreatdealofefforthasbeenappliedinAgileMethodsoverthepasttwodecadestoacceleratethesystemdevelopmentprocess,i.e.doingthingsfaster.Nomatterhowquicklytheydeliver,however,thesemethodsoftenproducesomethinginDlexible.Thispaperarguesforabroaderapproach,whichlooksat:thecontext(muchofthechangerequiredisoutsidethesystemdeliveryspace;thefocus(whatshouldwebechangingandwhy?);andthreeapproachestoachievingthechangewithrespecttosystemdeliver:doinglessthings(de-scope,usepackages,libraries,components,frameworks);dothingsfaster(agilemethods,automation,generation)andmakemoreDlexiblethings(runtimeadaptableordomainmodeldrivensystems).Thelastoftheseisanunconventionalapproachthatholdspromise,evenifyoucurrentlydon’tpractice,orsucceedwith,agilemethods.Finally,wealsoaddressthedilemmaofaccelerateddeliverywhiledealingwithlargelegacyapplicationlandscapes.

2of32

Intentionallyblank

3of32

TableofContents

Abstract 1AgilityisaBusinessRequirement 5DeDiningAgility 5Figure1-DimensionsofChangetoAchieveBusinessAgility 5

AgileMethods(Doingthingsfaster) 6Figure2-AgileManifesto 7Figure3-TypicalAgileLifecycleSource:ScrumAlliance 8

Dangersof“Agile” 9AchievingAgilityinSoftwareSolutions 10Figure4-Typical“Waterfall”Lifecycle 10Figure5-ContrastingWaterfallandAgileLifecyclesPMO=ProjectManagementOfDice 11

DeliveringMoreRapidly 11Figure6-TraditionalWaterfallLifecycle 11Figure7-IterativeIncrementalLifecycle 12Figure8-AgileLifecycle 12Figure9-DevOpsLifecycle 13

LeveragingPriorEffort(Doinglessthings) 13Figure10-PartofanoldstonebuiltfoundryinMoratacPark,NorthCarolina 14UseComponentLibrariesandFrameworks 14Figure11-RapidConstructionwithPre-BuiltComponentsSource:https://i.pinimg.com 15Figure12-PrefabricatedStructuredeliveredtosite 16

UseReferenceModelsandPatterns 16Figure13-Ahospitalarchitectureblueprint,orpatternSource:www.hhbc.in 17Figure14-FrameworxfromtheTelemanagementForumprovidesgenericmodelsforTelcos 17

GenerateDetail 18Figure15-3DPrintedHouseSource:https://all3dp.com 18

DomainSpeciDicModelingandGeneration 19Figure16-DomainSpeciDicModelingActivitiesSource:MetaCase 19Figure17-DSMlanguageandgeneratordevelopmentdoesincuranoverhead,butitcanbesmall.Source:MetaCase 20Figure18-DSMdevelopmentmayberecouped,eveninaninitialprojectSource:MetaCase 20Figure19-DSMProjectsvsConventionalDevelopmentSource:MetaCase 21

ReuseandEnsureAdaptability 21DoLessThingsOurselves 22MakemoreFlexibleThings 22Figure20-CapeTownInternationalConventionCentre-AconDigurablefacilitySource:CTICC 23

RuntimeAdaptableSystems 24Figure21-RelativeProductivityofDifferentLanguagesSource:SoftwareProductivityResearch 25Figure22-EVANetmodelerArchitectureSource:inspired.org 26Figure23-RelativeFlexibilityandProductivityofTechnologies 27

AgilityandLegacy 27

......................................................................................................................................................................................................................................................................

............................................................................................................................................................................................................................

.........................................................................................................................................................................................................................................................

......................................................................................

....................................................................................................................................................................................................................................

...........................................................................................................................................

...............................................................................................................................................................................................................................................................................................................................................................

...................................................................................................................................................................................................................................................................................................

...........................................................................................................................

................................................................................................................................

............................................................................................

.........................................................................................................................................

.....................................................................................................................................................................

............................................................................................................................................................................................................................

...................................................................................................................................................

........................................................................................................................................

...............................................

......................................................................................................................................................................................................................................................................................................................................................

.....

..................................................................................................................

...........................................................................................................................................

.....................................................................................................................................

4of32

Figure24-SoftwarevisualizationwithMoosetools 28

Harvest,ReModel,ForwardGenerate 28Figure25-CodeHarvestingandForwardGenerationwithDSM 29

TheLists 30Figure26-Factorsaffectingagility 30

PromotingOrganizationalAgility 30ReferencesandFurtherReading 32Papers 32Presentations 32Videos 32Websites 32

.............................................................................................

...................................................................................................................................................................

........................................................................................................................................................................................................................................................................................

................................................................................................................................................................................................................

............................................................................................................................................................................................................................................................................................................

......................................................................................................................................................................................................................................................................................................................

5of32

AgilityisaBusinessRequirement

Veryfewwoulddisputetheneedforagilityintoday’sbusinessworld,governmentorganizationsandNGOs.Agilityisneededtorespondtoincreasedcompetition,globalisation,governance/regulationandchangesintechnology,customerexpectationsandbusinessmodels.NoonewantstobeaKodakinadigitalphotographyworld,oranEncylopaediaBritannicawhenEncartaorWikipediaobliterateyourbusinessmodel.Giventhattechnology,businessandsocietalchangeiseveraccelerating,itishighlyunlikelythattheneedforagilitywilleverdecrease.ThisleadstothesurprisingconclusionthatAgilityisaStableRequirement.Ifitisastablerequirement,weshouldbeabletodevisewaysofachievingit.Therestofthispaperexaminestheimplicationsofthisrealisationandexploreswaystoachieveagility.

DefiningAgility

Forthepurposesofourdiscussion,wedeDineagilityasTheabilitytosuccessfullyadapttonewcircumstancesoropportunities

inarapidandsustainablemanner

Forabusinessthisincludestheabilitytoadaptoradjust:• BusinessModelandStrategy• OrganizationStructure,CultureandStafDing• BusinessProcesses• ProductsandServices• ChannelsforDelivery• ArrangementswithBusinessPartners• SkillsandTechniques• SupportingSystems,InformationandTechnologyWecanrepresentthedifferentfocusareasinthefollowinggraphic:

Figure1-DimensionsofChangetoAchieveBusinessAgility

6of32

NoticethatIT/ISisonlyoneofmanyaspectsthatrequireagility.Achievingagilityintheotherareasismostlyaconcernofdisciplinesincluding:• Strategy/BusinessArchitecture• ChangeManagement/ProgrammeManagement• OrganizationDesign• ProcessEngineering• ProductandServiceDevelopment• Contracting• Training• EnterpriseArchitecture• ProjectManagementCultureisalsoaveryimportantelementinachievingagiity.AnempoweredworkforceofsmallteamswithinterdisciplinaryskillsworkinginalearningorganizationculturewilldeDinitelyoutperformanautocraticformalone.FortheIT/ISelement,agilityincludesadapting:• Technologiesinuse• Infrastructuresupportingthebusiness• Applicationsystemscapabilities• Informationcapture,storage,processing,sharing,analysisandvisualisation

capabilitiesBearinginmindthatIT/ISisonlyafractionofthefullpicture,wewillzoominonapproachestoachievingagilityinthisdimension.Weplantoaddresstheotherdimensionsinfuturepapers.

AgileMethods(Doingthingsfaster)

ManyorganizationshaveadoptedAgileMethods(e.g.Scrum,DisciplinedAgile,ScaledAgile,ExtremeProgramming,Crystaletcetera)asameansofachievingagility.Theearliermethodsandversionsofmethodsprimarilyfocussedontherapid,incrementaldeliveryofapplicationsystems.Whilerelativelysuccessfulinthisgoal,wecanimmediatelyappreciatefromtheforegoing,thatthisisasmallpartoftheoverallscopementionedabove.TheAgileManifesto,draftedbyagroupofsoftwaredevelopmentluminariesin2001,setoutsomeimportantprinciples.

7of32

Figure2-AgileManifesto

Notethatitemphasisesthepriorityofitemsontheleftoverthoseontheright.Itdoesnot,asmanyadherentsproclaim,eliminatetheitemsontheright.Notetoo,thatitaddressesSoftwareDevelopment.Inourview,amethodshouldtargetSolutionDevelopmentwhichisbroaderthansoftwareonly,inthatitwilladdressotherelementsrequiredtosolvethebusinessproblem,including:• DocumentationsufDicientfortheinstallation,operation,supportandenhancement

ofthedeliveredsystembypartiesotherthanthedevelopers• Proceduresaroundthesystemtoensureitssuccessfuloperationinitsintended

context• Conversionofdatatothesystem• Integrationofthesolutiontothecontextinwhichitmustoperate• IdentiDicationofreusablecomponentsandservicesthatmaybeleveragedinother

efforts• Qualityassurancethroughtestingandothermeans(e.g.inspections)toensurethe

correctness,robustnessandcomplianceofthesolutionAgileMethodstarget,interalia,thefollowinggoals:• Rapiddeliveryofsystemscapabilitytothebusiness• Highlevelsofsponsoranduserengagementindecisionmakingduringthedesign,

buildandtestingactivities• ClariDicationofunclearorunknownrequirementsthroughmock-ups,prototyping,

facilitationanditeration• Gainingexperienceofnewapproachesortechnologiesearlytofacilitatelearning

andeliminaterelianceonuntestedassumptions• Visibilityofprogress

8of32

• CollaborativeteamworkTheyuseavarietyoftechniquesandpracticestofacilitateorachievethesegoals,including:• FeatureRequirementsListsorIssueBacklogs-tocaptureandagreerequirements

andallowgroupingofthesetoscopeaniteration,agreeingprioritieswithsponsorandusers

• Sprints-intenseburstsofactivitywithveryfocussedgoals,includingdeliveryofworkingsoftware

• Standupmeetings-toshareprogress,encouragecollaboration,reducemeetingtimesandkeepadministrationoverheadlow

• Visiblereporting(e.g.Burndowncharts)tomakeprogressandstatusvisibleandsharedandtofocusefforttowardsgoals

• Pairprogramming-toapplymorethanonemindtotheproblemandgeneratebackupskills/providecoaching

• Continuousplanning,integrationand(automated)testing• TestDrivenDevelopment(TDD)wheretestsaredevelopedDirst,andsoftware

subsequentlytopassthetests• Leanconceptswhichincludetheideasofdoingjustenoughtobeusefulandgauging

successbeforecommittingmoreresources,orpivotingtoamoreusefuldirection.Theseandkanbanconceptsalsofavourlimitingworkinprogresstoensurethatfocusishighandproductivityisimprovedbylimitingmulti-tasking

• Eliminationofany“mightbeneeded”requirements-thetermYAGNI(Youaren’tgonnaneedit)hasbeencoinedtosummarisethis,inthebeliefthatusers,analysts,designersandprogrammersarepooratpredictingthefuture,soweshouldnotwastetimeonanyrequirementsthatarenotabsolutelyknowntobeessential

Figure3-TypicalAgileLifecycleSource:ScrumAlliance

9of32

Agilerequirescertainfactorstopermititssuccess,including:• Highleveloftrustbetweenthesponsoranddevelopmentorganization.Afterall,

thereisnocontractedvalue/scopeforthemoneythatwillbespent,asintraditionalcontracting/waterfall.Rather,thesponsorcommitstofundingthedevelopersatacertainrateinthefaiththattheyarecompetent,willbeproductiveandwillfocusontherightissuestodelivervaluetothebusiness.Thiskindoftrusttakestimeandgoodhistorytoestablish

• Agilemethodsarenosubstituteforgoodskills.Infact,successfuluseofAgilewilloftenrequirehigherskills,sinceteammembersmustexercisemuchmoreindividualjudgementanddiscretion.Agileshouldthusnotbeattemptedwithateamofjuniorswithlittleexperience.Theycancertainlybeincludedinateamwithmoreexperience,buthavingexperienceonhandisapre-requisiteforsuccess

• Agilereliesalotonpeople.Continuityofstaffandfulltimecommitmenttotheprojectarenecessarytoensurethatthetacitknowledgeinplayisnotlostordiluted

Dangersof“Agile”

Agileissometimesusedasanexcuseforvariousills.CommonissuesarisingfromtheabuseormisunderstandingofAgileinclude:• Lackofarchitecture-assumingthatwecan“builditaswego”,resultinginpoorly

architectedsystemsthatdonotadheretogoodprinciplesandwillbedifDiculttomaintaininfutureorwhichdonotDitintothecontextwheretheymustoperate.Infact,betteragilemethodsdocaterforarchitecturalrequirements,e.g.bydoinganarchitecture“spike”beforecommencingdetaildevelopment

• Lackofdocumentation-AlthoughAgilefavourspeopleandinteractionoveradministrationanddocumentation,itdoesnotsayweshouldnotdocument.Rememberthatweneedtoproduceaninstallable,operable,maintainablesolution

• Rehashingrequirementsthatwealreadyknow.Yes,AgileisagreatwaytoreDineunclearrequirements,ordiscoverunknownones.Thatdoesnotmeanthatallourrequirementsareunknownoruniquetothisproject.Wecancertainlysaveahugeamountoftimeandeffortbysimplydocumentingourknownrequirementsorusingestablishedframeworks,referencemodelsandpatterns

• Under-deliveringonscope.Agileprojectswilloftencutscopetomeetadeadline.Thatcanbeagoodtechnique,providedthatwedonotdoittothedetrimentofthebusinessrequirementsandpriorities.WehaveseenAgileprojectsthatreducedthescopebeyonda“minimumviableproduct(MVP)”todeliversomethingnotusefultothebusiness

• Focusonfunctionalityattheexpenseofotherrequirements,suchasintegrationtoothersystems,performance,robustnessorscalability.Wemustpayadequateattentiontonon-functionalrequirementsaswell.Failuretodosocanultimatelydoomtheprojectandsolution

NewerAgileMethods(suchasScaledAgile(SaFe)DisciplinedAgile(DaD))areaddressingissuesbeyondjustsoftware,andincreasingly,issuessuchasarchitectureandDittoorganizationalobjectives.Theseareencouragingdevelopments.InthenextsectionwewilllookatwaystoachieveagilityandwhereAgileMethodsDitinthis.

10of32

AchievingAgilityinSoCwareSoluEons

Theprimaryapproachfollowedtoimproveagilityinsoftwareistotrytoreducethetimebetweenidentifyinganeedanddeliveringasolution.Thetraditional“waterfall”lifecyclehadaseriesofsteps,moreorlessinsequencewitha“Dlow”fromearliertolaterstages.Discoveryofnewrequirementsordisprovingassumptionslaterinthelifecycle,oftenrequired“cyclingback”toanearlierstageandameasureofreworktocontinue.Theoveralllifecyclecouldtakeawhiletocomplete,oftentothefrustrationofbusinessunderpressureforrapidcalendartimedelivery.

Figure4-Typical“Waterfall”Lifecycle

Iterativeandincrementalformswereevolvedtoimprovethedelivery.Intheseapproaches,asubsetofrequirementswouldbeselected,developedanddelivered,beforecyclingbacktotacklemorerequirements.Thishadtheadvantageofearlierdeliveryofpriorityitemsandbeingabletoapplylearningfromearlieriterationstolaterones.Theseapproachesworkedwell,providedthattherequirementswerefairlywellunderstood.However,manyclassesofproblemsarose(especiallyasInformationSystemsandTechnologyexpandedintonewtypesofapplicationsneverbeforecontemplated)wheretherequirementswerenotunderstoodattheoutset.Inthesecasesamorefacilitative/prototyping/experimentalstyleemergedthatledto“Agile”methods,whichassumethatrequirementsarenotunderstoodandwillonlybecomeclearduringtheproject.Agileemphasiseshighsponsor/userinvolvement,teamwork,communication,deliveryandfeedbackinrapiditerations.Scopemaybeadjustedineachiterationinconsultationwiththesponsorandlearningfromearlieriterationsisfedintosubsequentones.Eachiterationhaselementsofrequirements,design,development,testingandmaydeployworkingsoftware.Wecontrastthewaterfallandagilelifecyclesbelow:

11of32

Figure5-ContrastingWaterfallandAgileLifecyclesPMO=ProjectManagementOfQice

DeliveringMoreRapidly

AlotofmethodemphasisovertheyearshasbeenonreducingthetimetodeliveryfromidentiDicationofabusinessrequirement.Wecanseethisprogressionasfollows:

Figure6-TraditionalWaterfallLifecycle

Inthetraditionalwaterfallcycle,deploymentoccursquitelateinthecycleandalongtimeafterrequirementsareDirstdiscussed.Thiscanbeaproblemforbusinessiftherequirementisurgent,e.g.complyingwithasuddenlegislativechange.Otherproblemsthatcanarisearethattherequirementchangesorbecomesobsoletebeforethedeliveryofthesoftwarecapability.Thiscanleadtoagreatdealofwastedeffortandinvestment.Iftherequirementsarepoorlyunderstood,orunclear,theriskofdeliveringsomethingthatisnotoptimalisveryhigh.Fixingitcanalsoincuralongdelayandhighcostaswewaitforanothercycle.Toaddresstheseconcerns,theiterativeincrementalapproachbrokerequirementsupintopriority(anddependency)subsets,eachofwhichcouldbetakenthroughthelifecycleinashorterperiod.Thisisillustratedbelow.

12of32

Figure7-IterativeIncrementalLifecycle

Notethatthephasesarestillthere,buttheyareshortersinceeachistacklingasmallerscope.Thereisearlierandmorefrequentdelivery.AsintheDigure,thereisalsoopportunitytooverlapthedevelopmentifmultipleteamsareavailable.Agilemethodsrecognisethatrequirementsareoftenunclearormayevolveduringthelifecycle,sotheyaccommodaterequirementschangebothwithinandbetweeniterations,Seebelow.

Figure8-AgileLifecycle

Byallowingthis,theyovercometheproblemofwastedeffortonincorrectorobsoleterequirements.Aswementionedpreviouslythough,thereareanumberofpreconditionstogettingAgileright,aswellasdangersfromitsmisapplication.ThecombinationofAgileMethods,Virtualisedinfrastructure,IntegrationbetweenAgileDevelopment,AutomatedTestingandDeploymenthasledtotheDevOpsmovement,whereDevelopmentandOperationsarebroughtmuchclosertogether.

13of32

Figure9-DevOpsLifecycle

Therequirementstodeploymentcycleisshortenedbyintegratedmethods,highcollaborationacrossfunctionalareas,andadvancedintegratedtoolchainsthatcantaketheoutputfromdevelopmenttools,feedittotestsuitesandtheitemsthatpasstestingtooperationsinaseamlessprocess.Continuousdeploymentisanextremeformofthiswhereeachfeatureorchangecanbedeployedimmediatelyitistestedina“minirelease”.Thiswilloftenalsorequiretheadoptionofserviceorientedapproachessothatdeploymentofcomponentscanoccurwithoutneedingacompletemonolithicproducttobebuiltorlinked.Inadvancedshops,suchasFacebookorAirBnb,therecouldbeseveraldozen“pushes”intoproductiononadailybasis.Google,forexample,maydeployseveraldifferentchangescenariosinparallelandmonitorwhicharemoresuccessfulinproduction,beforedecommissioningthelesssuccessfulcandidates.

LeveragingPriorEffort(Doinglessthings)

Theslowestwaytobuildsomethingisdoingitfromrawmaterials,uniquelyforthecurrentrequirement.Ananalogyherewouldbebuildingastonestructurewherestonesareindividuallyprepared.

14of32

Figure10-PartofanoldstonebuiltfoundryinMoratacPark,NorthCarolina

Thinkabouttheeffortinvolvedinconstruction:• Findingandtransportingstone• Cuttingstonetosizeandshapebyhammerandchisel• Fittingeachuniquestonecarefullywithitsneighbours• Cuttinglumber,planingitintoplanks,makingdoorframesanddoors• Obtainingbrassforhinges,cutting,beatingandDilingitintoshapetoformhinges,

handlesetc.Thisiscertainlynotarapidprocess.Itrequireshighskillsandagreatdealofeffort.Changingoradaptingtheresultingstructurewillnotbeeasyorquick.InITterms,thisisanalogoustodevelopingasystemfromsourcecodeonly,withoutusinganypre-existinglibrariesorinfrastructuresoftware.Letslookatsomeoptionstoachievegreateragility.

UseComponentLibrariesandFrameworks

Ifwehaveaccesstopre-madecomponents,wecanfocusmoreonconstructionandlessonthebasiccomponentmaking.ConsidertheDigurebelow.Herewehavebricks,cement,tiles,timbertrusses,doorframesetc.deliveredtosite.Theprimaryefforthereisincombiningtheseintouniquecombinationstomeettheclient’srequirements.Effortismuchlowerandtimetobuildmuchless.Wecanprobablyalsogetbywithlowerlevelsofskill.

15of32

Figure11-RapidConstructionwithPre-BuiltComponentsSource:https://i.pinimg.com

IntheITenvironmenttodaywehaveverylargelibrariesofcomponentsdesignedtoworktogetherandtoprovideforthebuildingofnearlyallcommonsolutionsandcapabilities.Frameworksarelargecomponent(alsoclassandresource)librarieswhichprovideintegratedandcompatiblesupportaswellasarchitecturalguidanceonhowtolayerandcombinetheelementssuccessfully.ExamplesincludeMicrosoft.Net,Java2EnterpriseEdition(J2EE)andApple’sCoreandApplicationServices.Eachoftheseprovidesdeveloperswithaplethoraofcapabilitieswhichcanbeaccessedusingcommonapplicationprogrammingparadigmsandfacilities.Theyinsulatedevelopersfromagreatmanytechnicaldetails.Takingpre-fabricationtothenextlevel,wecanmakeuseofcompletepre-fabbuildings,deliveredtositeandjustconnectedtotheinfrastructure(e.g.water,electricityandsewage).InITterms,thisequatestotheuseofpackagesforatleastpartofthesolution.Theideaisthatweusecommoditydesignsandconstructionsforgenericrequirements,andonlyadaptordevelopbespokeelementswherenostandardonesareavailablethatwilldothejob.Packagescanreducetimetomarketandinvestmentrequireddramatically,iftheysuitrequirementswellandarenotheavilymodiDied.Themorewemodifythem,themorewelosetheadvantagesweseekandslipbackintodevelopment,sometimeswithrequirementsforrareandexpensiveskillsinesoterictools.

16of32

Figure12-PrefabricatedStructuredeliveredtosite

UseReferenceModelsandPa:erns

Componentsreducetheefforttobuildthesolution.Wherewehavetodesignthesolutionorelementsofit,wecansavetimebyusingreferencemodelsandpatterns.Patternsareatadesignorarchitecturelevel,ratherthanthephysicalcomponentlevel.Theycaptureknowledgeofprovenapproachesthatworkandthatcanbeeasilyadaptedtoouruniqueneeds.Anexamplewouldbethepatternusedforaclinic,wheretherewouldbeareceptionarea,assessmentarea,consultationrooms,treatmentarea,dispensingfacility,testareaandablutionfacilitiesforstaffandpatients.Thepatterncouldbescaledupordowndependinguponthebudgetandthenumberofclientsitisexpectedtoserve.

17of32

Figure13-Ahospitalarchitectureblueprint,orpatternSource:www.hhbc.in

AnITequivalentwouldbetheModelViewController(MVC)patterningraphicaluserinterfacesystems,whichprovidesarchitecturalguidanceinhowtosplitresponsibilitiesbetweenlayersofthesoftware.Thiscanbeadaptedfordesktop,webormobileuse.Wherepatternsaddressparticulardomainknowledge,theyaretypicallyreferredtoasreferencemodels.ExamplesinindustryincludetheBAINreferencemodelsforservicesinBanking,theACCORDmodelsinInsurance,theARTSmodelsinRetailandtheFrameWorxmodelsforprocess,dataandapplicationsinTelecommunications.

Figure14-FrameworxfromtheTelemanagementForumprovidesgenericmodelsforTelcos

Thesecanembodyhundredsofyearsofanalysiseffortandcansaveagreatdealofworkduringanalysis,architectureanddesign.Theyalsotypicallyfacilitateinteroperabilityofoursolutionswithbusinesspartnersandpackagesobtainedfromindustry.Theirusecanreduceriskandtimetodelivery,aswellascost.Thesemainlyreducetimeinanalysisanddesign,butcanalsoshortenbuild,testandintegrationtimebyusingprovensolutionsandmodels.

18of32

GenerateDetail

Developmentnormallyincludestheactivitiesofrequirementselicitation,design,buildingandtesting.Requirementsanddesignarebestdoneasmodelstoensuretheircompleteness,rigourandaccuratecommunicationbetweenparties.Thebuildandtestactivitiesareoftenthemostlabourintensive.Softwaregenerationapproacheseliminatealotofthiseffortbygeneratingthecodefrommodelscapturingtherequirementsanddesign.Traditionally,thishasbeentoutedasamajorboontoproductivity,buthasseldomdeliveredinpractice.ApproachesinthisspaceincludeModelDrivenDesign(MDD)fromtheObjectManagementGroup(OMG).Problemsinachievinghigherproductivitywerefoundtobemostlyduetothemodelsbeingsolution/technologyorientedandgenericratherthanrequirement/domainorientedandspeciDic.Ananalogyherewouldbemodelingahouseintermsoffoundations,brickwalls,doorandwindowframes,raftersandrooDingsheets.Thesearegenericcomponentsatafairlylowlevel.Specifyingthemodelatthisleveldoesnotsavealotofeffortoverbuildingit.However,ifwecouldspecifythehouseatthelevelofaconceptualarchitecture(styleandDloorplan/roomsonly)andgeneratetheDinishedhousefromthatautomatically(e.g.by3DPrinting)thenwecouldseemajorgainsinproductivityandrapiddevelopment.

Figure15-3DPrintedHouseSource:https://all3dp.com

19of32

ThesystemsequivalentofthisisknownasDomainSpeciDicModeling(DSM).Requirementsaremodeledusingdomainconcepts,suchasCustomer,Product,Order,Branchandassociatedprocessesandevents.Codegenerationproducesworkingsoftwarerespectingissuessuchasarchitecture,standardscompliance,securityandothercontextualrequirementsandtargetingthetechnologyplatformofchoice.Therearemajorsavingsintimeandcost,sincetheefforttocreatethearchitectureandgeneratorsisexpendedonlyonceandthegeneratedcodeiserrorfreesinceitfollowspreviouslytestedpatterns.

DomainSpecificModelingandGeneraEon

Figure16-DomainSpeciQicModelingActivitiesSource:MetaCase

Inindustrialprojects,DSMhasprovenhighlyproductive.Itisusedextensivelyintheautomotiveindustry,forexample,torapidlygeneratenewsoftwarefornewgenerationsofvehicles.Anewmodelcouldbelaunchedeverytwoyearsorsorequiringmillionsoflinesofcustomcodetomanagefuel,emissions,powerdelivery,transmission,traction,braking,entertainment,comfortsystems,lighting,navigationandamyriadotherthingsreliably.Codingthisintheconventionalwayintherequiredtimeandwiththerequisitereliabilityandqualityissimplynotfeasible.Otherindustrieswhichrelyheavilyuponthesetechniquesincludetelecommunications(e.g.developmentofswitches),cellular(developmentofhandsetsoftware)andotherelectronicsmanufacture,wheredevicesareincreasinglydrivenbysoftware.DSMisnotrestrictedtotheseDields,buthasalsobeenusedeffectivelyinbanking/Dinance,insurance/assurance,agriculture,plantautomationandotherapplications.

20of32

DSM Solution Development Time

Man days

Figure17-DSMlanguageandgeneratordevelopmentdoesincuranoverhead,butitcanbesmall.Source:MetaCase

ThereisanoverheadincreationofthedomainspeciDiclanguageandassociatedgeneratorsfortheDirstproject(s),butthesavingsdownstreampaybackhandsomely.

Figure18-DSMdevelopmentmayberecouped,eveninaninitialprojectSource:MetaCase

21of32

Results of the studies � Laboratory study

– Measuring time: at least 750% faster – Asking opinions: results (scale 1-5, 5 best):

� Pilot – Measuring time:

>900% faster

Figure19-DSMProjectsvsConventionalDevelopmentSource:MetaCase

Overallproductivity,qualityandmatchtodomainrequirementscanbeimprovedsubstantially,aswellastimetomarket.

ReuseandEnsureAdaptability

Anotherwaytoreduceeffort,andthustime,todelivery,istoreusewhatwehavedonepreviously.Thiscanoccurintheformofcomponents,frameworks,patternsandservicesasdetailedpreviously.Itcanalsooccurintheformofreusingcode,whichwehaveensurediswellstructuredenoughtobeintelligible,welldocumentedenoughtochangesafelyandwellstoredinasafeplacesothatitiseasilyDindable.Afurtherrequirementforreuseisthatthecode,components,frameworksetc.areofgoodquality.Wedonotwanttoreuseandproliferateproblemsandbugs!Achievingreuseisnoteasyandrequirespremeditation,discipline,skillsandtoolsupport.Someareasweneedtopayattentiontoinclude:• Architecture,sothatcomponentsarecoherentandfocussedonachievingone

purposewell• Design,sothatelementsarewellconceived,havecleaninterfacesandcaninter-

operate• Standards,sothatthingsarecompatibleandcompliant• Serviceorientationandloosecoupling,sothatitisrelativelyeasytoswap

componentsinorout,ortoreusethingsindifferentcontexts• Reliability,toensurethatwecansafelyusethingswithoutexperiencingproblems• Performance,sothatelementscanbeusedindifferentscaleapplications• Security,toensurethatwedonotexposetheorganizationtounduerisk

22of32

• Documentationandknowledgemanagement,sothatcomponentsareeasilydiscoverableandpotentialuserscanassesstheirsuitabilityquicklyandeasily

DoLessThingsOurselves

Anotherwaytoacceleratedeliveryistodolessthingsourselves.Wehavepreviouslydiscussedsavingeffortbyusingcomponents,libraries,frameworksandreferencemodels.Wecanalsoachieveitbygettingsomeoneelsetodosomeofthework.Thismaynotsaveeffortorcost,butitcanalleviateresourceconstraintsandreducecalendartimetodelivery.Apre-requisiteforthisisthatwehaveagoodideaofrequirementsandarchitectureatahighlevel,sothatwecanapportiontheworkandresponsibility.AfurtherrequirementisthatwehaveaproductiveandtrustedoutsourcepartnerwhowillDitintoourworkapproachanddeliverproductsandcomponentscompatiblewithourenvironment.Unfortunately,Dindingasuitablepartnerandbuildingthenecessarycollaborativeworkmethodsmaytakealotofeffortandtime.Itabitlike:“Godgrantmepatience,buthurry!”.

MakemoreFlexibleThings

ADinalwayinwhichwecanbecomemoreagile,isdifferenttothoseproposedabove.Essentiallythepreviousapproachesconcentratedmostlyonbeingabletodelivertheendproductmorequickly.ThereisanorthogonalapproachwhichinsteadshiftsthefocustodeliveringaproductwhichisitselfmoreDlexible,obviatingtheneedfornewdelivery.Thisharksbacktoourtitleof“AgileisaStableRequirement”.Aphysicalexampleofthiswouldbethesituationwherewebuildaconferencecentre.Weknowupfrontthatthefacilitymustbeabletoadapttoaccommodatetheneedsofawidevarietyofevents,rangingfromtradeshows,tomusicconcerts,agriculturalshows,sportsevents,businessconferencesandindustrycoursesandtutorials.Weneedtobeabletoaccommodatetheeventsinshortorderandinanysequenceofvariety.Thisrequiresthatthefacilitywedesignandbuildmust,itself,beabletoadaptveryquickly.ThiscanbeaccommodatedbyhavinglargecoveredspaceswhicharereconDigurableviamovablewallsintodifferentspaces,whilecateringinmorepermanentwaysfortheneedswhicharecommonacrossallfunctions,suchascatering,parking,ablutionsetc.AgreatexampleofthisistheCapeTownInternationalConferenceCentre(CTICC).ItcanbeconDiguredinlessthan24hourstocaterforalltheabovementionedtypesofevents.Thisismorerapidthananyconstructionmethodcouldpossiblyhopetoachieve.ThereconDigurationisachievablebecausethefacilitywasdesignedwiththislevelofadaptabilityinmind.Thefocusisonanadaptableresultingproduct,ratherthanonthebuildprocess.

23of32

Figure20-CapeTownInternationalConventionCentre-AconQigurablefacilitySource:CTICC

InITterms,thiscanbeachievedintwomainways:• RuntimeadaptablesystemswhicharedrivenbyuserconDigurablemetamodels,

businessrules,outputparametersandformulae.Theycanbeadaptedbyuserswhoareknowledgableaboutthebusinessdomainwithoutneedingtoknowtheunderlyingtechnology

• ModelDrivenDevelopment,especiallyDSMasdiscussedabove.HerethesystemspeciDicationismaintainedasasetofintegratedmodels.Whenchangesarerequired,thenecessarychangesaremadeatthebusinessdomainmodellevelandnewcodeisgenerated,potentiallydirectlytoproduction

TheformerstyleallowsgreatDlexibilityandcancaterformanychangestobusinessrequirementswithoutneedingtechnicalsystemchanges.AlimitationisthatthesystemwilloftenbelessefDicientthanalessDlexiblesystem,sothisapproachmaynotbeappropriatewhereveryhighvolumesmustbeprocessedorresponsetimesareverycritical.ThesecondstyleisalmostasDlexible,butdoesstillrequirethegenerationanddeploymentsteps,althoughthesecanbeautomated.Itisabletoaddresshighdemandsforvolumesandcriticalperformance.Ineffect,theabovedebunkstheYouAren’tGonnaNeedIt(YAGNI)principleadvocatedinAgilemethodstoreducescope.WereplaceitwithYAGNI,butmeaningYouAREGoingtoNeedIt,the“it”beingagility.Thegoals,howeverarenotincompatible.RememberthatAgileevolvedtoassistintherapiddeliveryofbusinesscapabilities.YAGNIwasappliedtoreducescope,sothatwecouldreduceefforttodelivermorerapidly.So,ifwenowturnthataround,areweincreasingscopeandeffort?No,butthereisatrick.Ifwesimplyaddedmorefunctionforthingsthatwemightneed,then,yes,scopewould

24of32

increaseandthatwouldbebad.BUTifweworkatahigherlevelofabstractionandidentifythekindsofthingsthatwewillneednowandinthefuture,wecanoftenreducethescopeofthesystemfurther,thussavingeffortandtimeintheoriginaldevelopmentwhilealsoequippingthesolutionwiththeabilitytoadaptinproductionusewithoutmoredevelopment.So,thisisawin-win.Thecaveatisthatwemayrequirehigherskillsinconceiving,designingandbuildingtheoriginalsolution.

RunEmeAdaptableSystems

AnexampleofsuchasystemfamiliartomostofusisMicrosoftExcel(orotherspreadsheets),whichprovidesarangeofgenericfacilitiesthathelpuserstailorthesystemtotheirneedsviaformulas,headingsandothercapabilities,suchaschartdeDinitions.Thegenericcapabilitiesanticipatethefactthatuserswillhaveaverybroadrangeofrequirementswhichdifferinthedetails,buthaveunderlyingsimilarities(e.g.theneedtoworkwithnumbers,manipulatethemwithformulas,organisethemintotablesandsorttheminvariousways.Usersareabletousetheapplicationtomeettheirneedsinshortorderwithoutrecoursetodeveloperstomakechanges,orthedelayandcostofsoftwaredevelopment.SpreadsheetshaveconsistentlymeasuredveryhighproductivitylevelswhenempiricaldataisanalyzedfordeliveryofgivenrequirementsinspeciDiclanguageenvironments.WitnessthefollowingstatisticsfromCapersJones:

LANGUAGE

1st Generation default

2nd Generation default

3rd Generation default

4th Generation default

5th Generation default

ABAP/4

Access

ANSI BASIC

ANSI COBOL 74

ANSI COBOL 85

ANSI SQL

Assembly (Basic)

Assembly (Macro)

C

C++

CICS

LEVEL

1.00

3.00

4.00

16.00

70.00

20.00

8.50

5.00

3.00

3.50

25.00

1.00

1.50

2.50

6.00

7.00

AVERAGE SOURCE STATEMENTS PER FUNCTION POINT

320

107

80

20

5

16

38

64

107

91

13

320

213

128

53

46

25of32

Figure21-RelativeProductivityofDifferentLanguagesSource:SoftwareProductivityResearch

LanguageLevelisarelativeproductivityindicatorforcomputerlanguages.Alanguageratedas12wouldhalvetheprogramming(build)phaseofaprojectrelativetoalanguagerated6.Ifthelanguageisalsoaccessibletodomain/businesspersonnelratherthanprofessionalcomputersystemdevelopers,itmayalsoreduceeffort,timeandcostintheanalysisanddesignphasesaswell.AnotherexampleofaruntimeadaptablesystemistheEVANetmodelerenterprisemodelingandknowledgerepositorytoolsetfromInspired.org.ThisallowsruntimedeDinitionofametamodeldescribingtheconcepts,propertiesoftheseandrelationshipsbetweenthem,ofrelevancetoauser,group,domainorenterprise.Itthenusesthisinformationtomodifyuserinterfaces,reports,visualizationtoolsandotheraspectsdynamicallyatruntime.

COBOL

Common LISP

Crystal Reports

DELPHI

EXCEL 5

FORTRAN 77

Haskell

HTML 3.0

JAVA

Machine language

Object-Oriented default

Objective-C

PASCAL

PERL

PowerBuilder

Reuse default

RPG III

SMALLTALK

Spreadsheet default

SQL

Visual Basic 5

3.00

5.00

16.00

11.00

57.00

3.00

8.50

22.00

6.00

0.50

11.00

12.00

3.50

15.00

20.00

60.00

5.75

15.00

50.00

25.00

11.00

107

64

20

29

6

107

38

15

53

640

29

27

91

21

16

5

56

21

6

13

29

26of32

Figure22-EVANetmodelerArchitectureSource:inspired.org

Inoneinstancewemetwithaclientinthebankingindustryandtheirindustryconsultantstodeterminerequirementsinsupportofastrategy,architectureand“re-baselining”projecttobeconductedacrossthreecontinents.Wedrewtheseasametamodelonawhiteboardduringalongworkshopday.IntheeveningwecapturedthemetamodeltoconDigurethetooltoaddresstheproblemandsetupaninstanceonaserveravailableviatheInternet.Thefollowingdaywedemonstratedthetoolsupportfortheproblemsbeingtackledtotheconsultantsandgottheirusercredentials.Wecapturedtheseintothetoolthateveningandthefollowingdaythetoolwasbeingusedtocaptureinformationfromliveworkshopsonthreecontinents.Thekeytothisrapiddeploymentwasagainthefactthatthetoolsetisdesignedtocaterformanyandvariedrequirementsbyfactoringoutwhatiscommontothemandsupportingthatinawaythatuserscantailortotheirneeds.Technically,thetoolhaspatternsforbusinesslogicanduserinterfaceinvariousstyles,andinjectsthestructurestowhichtheseshouldapplyintothecodegeneratedfromthese,whichisservedtoabrowserviathenetwork.Moredetailcanbefoundin[McLeod,2001].

27of32

Figure23-RelativeFlexibilityandProductivityofTechnologies

Agility andLegacyAchievingagilitygoingforwardistoughenough.Butmostsiteshavealargecollectionoflegacyapplicationswhichrepresentasubstantialasset(oratleastexpenditure)andunderpinoperations.Oftenthesearepoorlydocumentedandhave“unravelled”tosomeextentthroughpoormaintenance,sothattheiroriginalarchitecturesanddesignhasbecomeobscured.Howdoweleveragetheseandmakesuchanenvironmentmoreagile?Wecanveryseldomaffordtothrowitawayandstartfromscratch.Evenifthiswasaffordableeconomically,itwouldprobablynotbepracticalfromatimeperspective.WemayalsoDindthatthedomainknowledgeisnotavailableintheenvironmentandusercommunity,butisembeddedintheoldapplications.Conventionalmaintenancereliesalotonreadingcode.Thisisadauntingprospectwhenweconsiderthatalegacyapplication(ofwhichtheremightbescoresorhundreds)caneasilyrepresentseveralmillionlinesofcode.Surveysshowthatmaintenanceprogramersspendasmuchas80%oftheirtimereadingcode.Thisactivityhasn’tchangedmuchsincethe1950’s,withtheexceptionthatthevolumeofcodehasincreasedexponentially.Thisisapoorwaytounderstandanexistingsystem.Itdoesnotscale.Fortunately,thereisanewmovementadvocatedbyTudorGirbaandcolleagues,knownasHumaneAssessment.Theideaistounderstandsystemsrapidlywithoutbeingcrueltoprogrammers.Wealreadyapplysophisticatedtoolstounderstandourbusinessdata

28of32

andtogeneratevisualisationsandinsightsfromvastdatacollections(“bigdata”).Heretheprogrammersareliketheshoemakerschildrenwhoarebarefoot.Canwenotdobetterandusebettertoolsandtechniquestosolveourownproblemsaswellasthoseofusers?Indeedwecan.Techniquesknownassoftwarevisualizationhavebeendevelopedoveraperiodoftwentyyearsandthetoolssupportingthese,includingtheMooseanalysisplatformanditsaddonsandextensionshavebecomeverysophisticated.

Figure24-SoftwarevisualizationwithMoosetools

Essentially,wecanscanexistingsystems(code)toidentifyrelevantdata(e.g.ClassesordeDinitionsrepresentingconceptsanddatastructures;Functions,ModulesandMethodsrepresentingfunctionsoractionswhichoperateonthedatastructures;andtherelationshipsbetweenthesee.g.Whichcodedependsuponwhichotherpartsandwhichcodeuseswhatdata).Thedatacanthenbevisualizedusingavarietyofavailabletechniquestoidentifystructure,problemsandopportunitiesforimprovement.Thiscanfacilitaterapidmaintenancewithlesseffortandrisk.

Harvest,ReModel,ForwardGenerate

Takingitfurther,wecansupplementtheanalysiswithsomehumanexpertisetoderivedomainknowledge,semanticinformationandothervaluableassetsfromtheexistingsoftware.Thesecanbeusedtocreatedomainframeworksandreferencemodelsforforwardengineering.

29of32

WecanthencoupletherecoveredmodelswithDomainSpeciDicModelingtechniquestocreateapowerfulApplicationRenewalmethod.

HarvestKnowledge

AugmentKnowledge

GenerateSolution

LegacySourceCode

DomainSpecificModels

TechnicalArchitect

DSMDriven

Application

NewSolutionCode

NewSolutionRuntime

TargetIDE

ReferenceModel

DomainExpert

ContinuousBusinessSolution

Improvement

ContinuousTechnicalSolution

Improvement

High PerformanceSolution Required

Low PerformanceSolution Required

UserCommunity

Figure25-CodeHarvestingandForwardGenerationwithDSM

Inspirediscurrentlydevelopingandpioneeringthesetechniquesandisseekingcommercialpartners/customerstocollaborateontheseefforts.WeareengagedwiththeauthorsofcodeharvestingandDSMtechniquesandtooling.Ifyouareinterested,pleasecontactthewriter.

30of32

TheLists

Inthissectionwesummarisesomeofthecriticaldimensionsthatpromoteorinhibitagility.Inmanycasesthesewillhaveincrementaleffectswhenusedtogether.Inourculture,training,methods,managementpracticesandprojectsweshouldtryanddoasmanyofthePromoteitemsaspossible,whileavoidingasmanyoftheInhibititemsaspossible.

Figure26-Factorsaffectingagility

PromoEngOrganizaEonalAgility

Theforgoingdiscussionmayseemdaunting.Afterall,therearemanydimensionsandmanyassociateddisciplines,skills,cultures,techniquesandtoolstomasteroradopt.Wheredowestart?Itcanbeusefultotakealeafoutofthebookofcontinuousimprovement,orthemoreformaldisciplineofSixSigma,whileavoidingtherigorousstatisticalsideofthelatter.Effectivelywefollowasimplecycle:• EstablishIntent-Decidewhatwewanttoachieve(e.g.OrganizationalAgility)and

whatthatwillrequire,viadecomposition(e.g.AgileFriendlyCulture;AbilitytoRapidlyDeliverSystemsCapabilityinProduction;AbilitytoFlexBusinessModeletc.)

• Identifythebiggestbottleneckstoachievement.Whatareourcurrentworstperformingareas?Maybeitisaveryautocraticculture,maybeitisaninabilitytoaccuratelydeDinerequirements,maybeitisaqualityprobleminthedeliveredproduct.ThisisbestdonebylookingatspeciDicperformancemeasuresand

Promote AgilityHigh SkillsUser InvolvementFrequent Short MeetingsEmpowered StaffHigh Level of AbstractionDomain Specific ModelingIntegrated MethodsModel OrientedSmall TeamsAutomationHigh QualityGood InfrastructureLearning CultureColocated TeamsVisual Work, ProgressCollaboration. MentoringInformal with high trustDevelopers have Domain KnowledgeUsers have IT KnowledgeContinuity of StaffReuseTeam of TeamsRuntime adaptable productVirtual infrastructure

Inhibit AgilityLow SkillsUser Dis-engagementDelayed Large MeetingsCentralised or Remote Decision MakingLow Level of AbstractionGeneric ModelingNon-Integrated MethodsDocument OrientedLarge TeamsManual WorkLow QualityPoor InfrastructureDo What You are Told CultureDistributed TeamsHidden Work, ProgressPunitive, AuthoritarianFormal with low trustDevelopers have only IT KnowledgeUsers have only Domain KnowledgeDisrupted or Part Time Allocation of StaffBuilding AnewCommand and ControlDesign time adaptationTraditional infrastructure

31of32

associatedbenchmarks,comparedtoourperformance.E.g.Ifcompetitorscandeliveranewservicetothemarketin3monthsandwetake10.Findthetop3-5areaswhichareunderourcontrolandamenabletochange.Focusjustonthoseforthenextwhile

• Improvetheunderperformingareasbyfacilitatingculturalchange,improvingmethods,skills,techniquesandtools,buildingtrustandwhateverotherfocussedmeanswilladdresstheissue

• Iterate-oncechangeisachievedinanareaandveriDiedbynewmeasurement,selectthenextmostproblematicarea(s)forattentionandkeepdoingit

Inthiswaywecanbeassuredthatwearealwaysmakingprogress.ItmayseemslowatDirst,butitwillbeincrementalaschangescombinetheirimprovementsinamultiplicativeway.Wecanacceleratechangebytheapplicationofmoreeffort,resourcesormoney,providedthattheyarealwaysfocussedondoingthingsbetter,notjustfaster.

32of32

ReferencesandFurtherReading

PapersMcLeod,Graham.“Pamela:AProto-PatternforRapidlyDelivered,RuntimeExtensibleSystems.”EvaluationofModellingMethodsinSystemsAnalysisandDesign(EMMSAD)(2001)availablehere

PresentaEons

SoftwareVisualization,TudorGirba

Videos

AgileVisualisationinMondrianSoftwareEnvironmentalismIntroductiontoDomainSpeciDicModelling20DomainSpeciDicModellingExamples

Websites

http://www.inspired.orghttp://www.moosetechnology.orghttp://www.metacase.com