Post on 14-Apr-2018
Improvingthemanagementofstatisticalprocessestheuseof
QualityGatesPaulSchubertandSiu-MingTam
July2015
Outline• BackgroundonQuality
Gates• EffectivenessofQuality
Gatesinreducing‘qualityincidents’
• Addressingbarrierstomorewidespreadandeffectiveimplementation
• Abroaderapproachtostatisticalriskmanagement
2
‘Statisticalrisk’andqualitygates
• Statisticalrisk: • Qualitygates:
checkpointsordecisionpointsatvariousstrategicplacesinastatisticalprocessatwhichthequalityoftheprocess(anddata)atoruptothatpointisexplicitlyassessed
thechanceorlikelihoodthatsomethingcouldgowronginthestatisticalprocessaffectingthequalityorintegrityofthestatisticaloutputs
Overviewofqualitygatescomponents
• Placement• QualityMeasures• Roles• Tolerances• Actions• Evaluation
Areasofrisk
• changestoprocesses,systems,methodologies• datatransformations• overlap/coordination/integrationwithotherareas
• knowledgemanagement
Reviewofqualityincidentsandreleaseincidents
0
1
2
3
4
5
0 1 2 3 4 5 6 7 8 9 10 11 12
Freq
uency
NumberofReleaseIncidentsintheQuarter
ReleaseIncidentsperQuarter
EffectivenessofQualityGatesinpreventingqualityorreleaseincidents
14.3%
61.4%
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
100.0%
Collections withQualityGates Collections without QualityGates
ChanceofaReleaseIncidentOccuringDuringtheReferencePeriodforPLaSSandEESG
collections(Jan2010-Jun2014)
Independentaudit
• AssesstheeffectivenessofQualityGatesinimprovingqualityandmitigatingrisk
• Understandandsuggestsolutionstothebarrierstomorewidespreadadoptionbyteamsthroughouttheorganisation
ExtentofqualitygateadoptionTeam 1 2 3 4 5 6 7 8 9 10Qualitygatesinplace?
P P Partially Ï Ï Ï Partially Ï Ï Ï
Placement P P P P P Ï P Ï P Ï
QualityMeasures
P P P P P Partially P P P P
Roles P P P P P Partially P P P Partially
Tolerances P P P P Partially Ï P Ï Ï Ï
Actions P P Partially Ï Ï Ï Ï Ï Ï Ï
Evaluations P P Partially P P Ï P Ï P Partially
Findings– benefitsofQualityGates
• valuabletoolinmaintainingqualityandminimisingrisk
• clearlyassignsaccountabilityandresponsibility• promotesbetterprojectmanagementpractices• createsefficiency• formalisesprocesses• aidsinknowledgesharingandsuccessionplanning
Findings– barrierstoadoption
• lackoftimeandstaffresources• teamswithrelativelyinexperiencedstafforimmatureknowledgemanagementpracticesstrugglewiththeconceptandapplicationoftheQualityGateprocess
• perceptionofaddingalargeadministrativeburdentoexistingprocedures
• easiertoimplementformeasuresthatarequantitativeratherthanqualitativeinnature
Recommendations
• developaQualityGatepolicy– specifyteams,collectionsand/orcircumstanceswhereQualityGatesaretobemandatory
• improvesupportandtrainingmaterial• utilisefacilitatedworkshops• improveknowledgemanagementofbusinessprocesses
• developKPIstomonitortheeffectivenessofQualityGates
Abroaderdefinitionofstatisticalrisk
• Context– plannedchangestosystems,
processesormethods– errorsinsystems,processesor
methods– changesintheavailability(or
expertise)ofresources– changesintheavailabilityor
qualityofsourcedata– ‘realworld’eventsthatmay
notbecapturedadequately– asampleoradministrative
datasetthatbecomesunrepresentative
– achangeinusers’viewsorperceptionsofthefitnessofpurposeofABSstatistics
TheriskthattheABSpublishesstatisticsthatarenotconsideredbyusersasfitforpurpose.Thisincludes,butisnotlimitedto,timeseriesthatarenotcoherentandstatisticsthatarenotcoherentwithrelatedstatistics.
AbetterapproachtostatisticalriskmanagementintheABS
• impossibletoeliminatestatisticalrisk• mitigationofriskscanofteninvolvepotentiallydifferentapproaches– assessonthegroundsofcost-effectiveness
• mitigationapproachesneedtobeassessedcollectively
aimtoensurethatstatisticalrisksareunderstoodandthatthecollectiverisksaremanagedinacost-effectiveway
Elementsofmanagementofstatisticalrisk
• integrationwithbroaderriskmanagementapproach• mitigationappropriatetoimportanceofoutputs• determinationof‘fitforpurpose’ofeachstatisticsorclassofstatistic
• documentedstatisticalriskmanagementplans– includingQualityGates,contingencies
• clearidentificationofriskmanager• independentassessmentandreportingbyMethodologyDivision
• engenderingtherightculture(ledbyaDeputy)
ReferencesABS2010,QualityManagementofStatisticalProcessesUsingQualityGates,cat.no.1540.0,ABS,Canberra.Dufty,D2014,ReviewofQualityGatesandQualityIncidentsintheABS,internalreport,ABS,Canberra.Oakton2015,ApplicationofQualityGates,internalaudit,ABS,Canberra.Harper,P2015,Statistical riskmanagementintheABS– Abetterapproach, internalpaper,ABS,Canberra.
17
Questions?
paul.schubert@abs.gov.au