Inforum 2016 Keynote: Data and Information Quality
-
Upload
jay-zaidi -
Category
Data & Analytics
-
view
245 -
download
4
Transcript of Inforum 2016 Keynote: Data and Information Quality
Change your Organization's Culture to Make Data and Information Quality a Part
of it’s DNA
inForum 2016Perth, Australia
September 13, 2016
Jay ZaidiManaging Partner
My Books, About Me, and Contact Details
Contact DetailsEmail – [email protected]
LinkedIn - http://www.linkedin.com/in/javedzaidiWeb – http://www.alydata.com/
2
My books on “Data-Driven Leadership” launched worldwide on Amazon and Kindle in July 2016.
About Me
Founded AlyData after two decades in the industry. In my last corporate role I reported directly for five years to the Chief Data Officer of the largest financial services company in the world. Worked for PriceWaterCoopers LLC, Commerce One, and DOW Chemical Company prior to that.
AGENDA
Part 1 – Age of DataPart 2 – Quality is Job 1Part 3 – There is a Trust Deficit that must be OvercomePart 4 – Change Culture or Get DisruptedPart 5 – Call to Action
3
Our World’s Being Turned Upside Down
4And You Should Think About What This Means to You!
The Fourth Industrial Revolution
“We stand on the brink of a technological revolution that willfundamentally alter the way we live, work, and relate to one another. Inits scale, scope, and complexity, the transformation will be unlikeanything humankind has experienced before. This is the FourthIndustrial Revolution or the digital revolution that has been occurringsince the middle of the last century. It is characterized by a fusion oftechnologies that is blurring the lines between the physical, digital, andbiological spheres.” – Klaus Schwab, Executive Chairman of The WorldEconomic Forum
5
I’ve labeled the Fourth Industrial Revolution the “Age of Data.”
Massive Disruption Is Happening In Every Sector - Your Company May Be Next
6
The Common Thread Across Disruptors are Data and Insights! High Quality Data is required for best insights.
5 Pillars of the New Business Model1. VarietyandDecentralization:Social,Mobile,Analytics,andCloud
(SMAC)driveoperations2. BetterInsights:Nearrealtimeinsightsfordecisionmaking,risk
management,andtogaincompetitiveadvantage3. Agility:TransformationoftheoperatingmodelfromSDLCtoAgileand
introductionofautomatedprocesses4. Transparency:Sharingeconomyrequiresasharingculture.Changein
teamdynamicstobecomemoretransparentandsharedataandalgorithms.
5. Innovation:Innovateusingdata,people,algorithms,andprocess.Newareassuchasartificialintelligence(AI),deeplearning,intelligentconversationengines,speechrecognition,and imageandpatternrecognition.
7
The Intellectual Capital of this new world is Algorithms, Data, and People.
A New Leadership Paradigm -Leadership 2.0
8
Leaders and aspiring leaders must become “data savvy” and pivot on “data” not IT.
Leadership1.0 Leadership2.0
Quality Is Job 1
9
10
Quality is a State of Mind and Has To Be Incorporated Into All Data Processing Steps.
• Inthe1980’sJacquesNasserwasCEOofFordandintroducedthisslogan
• WantedtotransformFordintotheleadingconsumerproductscompany
• Thisinitiativechangednotjusttheculturebutthequalityoftheendproducts
• ResultedinaUS$300Millioninreducedscrap,rework,andnonvalueaddedactivities.Thisisequivalentto$754Millionintoday’sdollars–asignificantsavings.
11
https://www.youtube.com/watch?v=X3ecGr_-2vQ
https://www.youtube.com/watch?v=X3ecGr_-2vQ
12
https://www.youtube.com/watch?v=Sx_6OoFmycohttps://www.youtube.com/watch?v=Sx_6OoFmyco
https://www.youtube.com/watch?v=Sx_6OoFmyco
4 Eye Opening Facts
• DataQuality:Atleast6%to10%ofIToperatingbudgetwastedduetore-workandinefficientprocessing• DataWrangling:70%to80%ofdataprocessingtimeandcostisassociatedwithdatawrangling• DarkData(acquiredbutneverused):85%ofdataacquiredisn’tusedforanythingofvalue• Metadata(context):Inabilitytofinddata,understanddatasemantics,anddatarelatedrulesresultsinmassiveinefficiency
13
Companies need data quality founders, evangelists, and data quality owners across all departments.
Companies Require Small and Big Data To Succeed
Traditional(SmallData) Data-driven(BigData)Highlystructureddata Structured, Unstructured,Semi-structured
Pre-defineddataschemas FlexibledataschemasPre-defined datamodels(schemaonwrite) Undefineddatamodels(schemaonread)Relationaldatabase managementsystems HadoopandNoSQL datastores
Silosofdata BigDataLakes(consolidateddatasets)Performance andscalabilitylimitations Infinitescaling
Mostlyonpremise data Highlydecentralizeddata(Cloud)DataMiningandBusinessIntelligence Predictive,Prescriptive Analytics,DeepLearning
andArtificialIntelligence
14
The introduction of Big Data into the “Data driven” business model requires a culture change - new data quality and project management
skills, new execution capabilities, and agility at its core.
There Is A Trust Deficit That Must Be Overcome
15
Trust Deficit
16
Business transactions are built on trust. Unfortunately, there is a trust deficit today and it must be addressed.
Producers and Consumers Operate On Trust
17
Government agencies define standards and policies that producers must follow to ensure quality and transparency.
Labels
Food labels present nutritional and other information to help consumers make safe and healthy food choices. Some labelling information is mandatory, while others are voluntarily added by manufacturers. Labelling must include a list of ingredients and food additives, as well as any potential allergens. A nutrition panel outlining levels of key nutrients is also required.
We use food labels:
• For health reasons• To avoid particular ingredients or food additives• For personal beliefs, such as avoiding genetically modified foods or
foods containing animal products or to buy items grown locally.• Food labels must tell the truth and include:• Name or description of the food• Nutrition information panel• Ingredient list• Percentage labelling• Food additives• Country of origin• Food recall information• Directions for use and storage• Information for allergy sufferers• Legibility requirements• Date marking.
When Was the Last Time You Were Provided a Label with your Data Sets?
18
Seems logical. But it’s never done in the industry. Shouldn’t it?
Data Set Labels
Data Set labels present information that describes the content of the data set and other information to help consumers understand what’s in it and to make choices on its usage. Some labelling information should be mandatory, while others are voluntarily added by producers. Labelling must include a list of key data ingredients and any data enrichment performed on it (additives), as well as any potential transformations (allergens). A data panel outlining levels of sensitive and personally identifiable data should also be required.
We use data set labels:
• for business transaction reasons• to be aware of particular sensitive or personally identifiable data so that we can handle them with care
Data labels must tell the truth and include:
• Name or description of the data elements in the data set• Data consumption information panel• Data element list• Data quality labelling• Any enrichments• System(s) of origin• Directions for use and storage• Information for special handling• Date marking.
19
An Inspection Regime Is Needed
One must trust but verify quality. Companies must implement an inspection regime to audit data quality at every step.
Data Quality Process Flow
20Repeatable Data Quality Processes must be Implemented.
Australian Bureau of Statistics Quality Framework
21
Suggestedquestionstoassessinstitutionalenvironment:• Whichorganization(s)hassuppliedthedata?• Whatsortoforganizationisthis(e.g.,public,commercial,non-governmentorganization)?• Underwhatauthorityorlegislationwerethedatacollected?• Whatproceduresareinplacetoenableaneedforastatisticalproducttobeevaluatedwithrespecttoitsscope,detailorcost?
• Towhatextentarequalityguidelinesdocumentedbytheagency?• Isstatisticalconfidentialityguaranteed,andifso,underwhatauthority?• Towhatextent,andhowquickly,areanyidentifiederrorsinpublishedstatisticscorrectedandpublicized?
22
Australian Bureau of Statistics Recommendation
These questions can be tailored for companies and their departments.
Change Culture or Get Disrupted
23
“Culture eats strategy for breakfast, technology for lunch, and products for dinner, and soon thereafter everything else too.” – Business Management Guru Peter Drucker
24
This culture change is critical to winning with data. No amount of strategizing will work otherwise. However, most companies are in
denial that there is a culture problem.
25
“Succeeding with data isn’t just a matter of putting Hadoop in your machine room, or hiring some physicists with crazy math skills. It requires you to develop a data culture that involves people throughout the organization.” - DJ Patil, Chief Data Scientist of the U.S.
Winning with data isn’t about Hadoop or new technology. It requires you to develop a data culture that involves everyone.
Let’s Define Culture First
Aculture isawayoflifeofagroupofpeople--thebehaviors,beliefs,values,andsymbolsthattheyaccept,generallywithoutthinkingaboutthem,andthatarepassedalongbycommunicationandimitationfromonegenerationtothenext.Culture issymboliccommunication.
26
Here Are 11 Characteristics of a Data Culture1. MissionAlignment:Data’sroleinthecompany’soverallmissionandgoalsisclearlyarticulated.Openlydiscussing
strategiesandinnovationgoalsprovidesemployeeswithaclearviewofdata’sroleinthecompany’soverallmissionandreinforcestheirconnectiontothelargerorganization.
2. DataQualitySavvy:Managementandstaffthataredataqualitysavvy– understandallthefoundationalelementsofdataqualityandwhydataiscriticalforsuccess
3. DataQualityProcesses:Definequalityrequirements,measurequality,andproactivelyaddressqualityissues4. Behaviors:Everyonemakesevidence-baseddecisions(notbasedongut)5. RightQuestions:Leadersandstaffareempoweredtoasktherightquestionssuchas– whatisthesystemofrecordfor
data?,what’sbeendonetoit?,canItrustit?,whoisaccountableforspecificdata?etc.6. InformationSupplyChain:Departmentalsilosofinformationarethenemesisofthrivingdatacultures.Topromotethe
viewofdataasaflexibleassetthat’susablebymultipledepartments,organizationsneedtoeducateemployeesonhowthedatatheyusedailyripplesthroughotherpartsoftheorganization.Employeesneedtoseethebigpicture.
7. RewardsandRecognition:Datasuccessesaresharedandindividualsandteamsresponsibleforthemarerewardedandrecognized
8. RightIncentivesandAlignment:Cross-functionalsolutionteamsarecompletelyalignedongoalsandincentivesbetweenIT,Data,andBusinessstaff
9. DataSharing:Thereissharingofdataandinformationbetweendepartmentsandtotaltransparency– nodatahoarding.Athrivingdataculturedependsonanenvironmentinwhicheveryonecanshareinformationwithoutbeingperceivedasnegative.
10. KPITransparency:Availabilityanduseofkeydatametricsandmeasuresviacomprehensivedashboard– dataquality,dataissuemanagement,datagovernance,datasecurityandprivacy,datalineage,etc
11. RobustDataPlatform:Arobustdataplatformhasbeenbuiltanditsupportsthetypesofanalyticsrequiredtomakedecisions,managerisk,andinnovate
27
The Culture Change Framework
OrganizationalCultureChange
BusinessCaseForChange
Creatingtheclimate forchange
Implementingand
SustainingChange
Engagingandenablingthe
organization
1. Establishingasenseofurgency2. Creatingtheguidingcoalition3. Developmentofachangevision
1. Communicatingthevisionforbuy-in2. Empoweringbroad-basedaction3. Generatingshorttermwins
1. Neverlettingup2. Incorporating
changesintotheculture
1. Establishingtheneedforchange2. Tyingculturechangeinitiativetobusinesspriorities3. Articulatingthevisionandtangibleresults
Team-Building ProcessesTeam building is a development process where the course is divided
into 4 phases plus a resolution phase.
Performan
ce
Courseoftime
Low
High
Orientationphase(forming)1
Growthphase(performing)4
Cooperationphase(norming)3Confrontationphase
(storming)2
Resolutionphase(adjourning)5
Call To Action
30
Every great dream begins with a dreamer. Always remember you have within you the strength, the patience and the passionto reach for the stars to change the worldHarriet Tubman, Abolitionist, Humanitarian & Spy (1822 – 1913)
“Mybiggestfearisnotcrashingonabike…
It’ssittinginachairat90andsaying,
‘IwishIhaddonemore’.”GraemeObree
The Climb– It’s Tough But Helps You Win
1. Companyleadershipneedstoelevatedataqualityasatoppriorityandtieittotangiblecustomer,product,andemployeebenefits.
2. Dataproducersandconsumersmustagreeanamutualcontractanddeliveronit.Anindependentauditregimechecksadherencetocontractterms.
3. Developaroadmapforculturechangeandimplementit.4. Investintraining.5. Buildworldclassdataqualityprocessexecutioncapabilities.
Appendix
34
17 Articles I’ve Authored On Data Quality
• 5Reasons WhyMoreCompaniesDon'tHaveDataQualityProcessesInPlace?• DataQualityisJob1andHere'sWhy?• SeriousImplicationsoftheDarkSideofBigData• Whatisthisthingcalled"DataQuality"?• HolisticDataQuality(HDQ)- ANewParadigmInEnterpriseDataQualityManagement• TheHolisticDataQualityFramework- Version1.0• HighQualityData+Analytics=DeepInsights• Here'sWhyYourDataDoesn'tReconcile?• 5DataQualityBestPractices• ShouldUsersSwitchFromOfficeProductivityToolsToCommercialDataQualityTools• 7RootCausesFortheHighCostofBadData• BadDataIsCostingtheU.S.AtLeast6%ofItsGDP• 6CoreComponentsofaDataQualityProgram• 3ActionsCanSaveYourOrganizationMillions• OrganizationCanSaveMillionsByApplyingThisDataRule• 4IngredientsForProducingTrustworthyBigData• UntrustworthyDataIsSpawning"ShadowITandDataNinjas"
35
Here’s What’s Required To Win1. Companyleadershipneedstoelevatedataqualityasatoppriority
• Everyoneinthecompanyshouldbecomedataqualityconsciousandincorporatequalityintoeveryprocessstep• Benchmarkthecompany’sdataqualitypracticesagainstbestinclasscompaniestoidentifygaps• Assessmaturityofdataqualitycapabilities(i.e.,people,process,technologyanddata)toidentifyareasofimprovement
2. DataProducersandConsumersMustAgreeOnAMutualContract• Contractprovidestransparencyintodataset,itscontents,anditsqualityfromproducers• Producersmustcertifydatasets,basedonpre-definedconsumerexpectations• Anobjectivethirdpartyshouldauditthecontractsandtheartifactstoensurethatproducersandconsumersareworkingingood faith
3. Developaroadmapforculturechange• Galvanizeleadersandassociatestobecome“dataqualityfocused”andtoinvestindataqualitymanagement• Incrementallybuilddataqualityprofilingandremediationcapabilitiesinanopportunisticmanner(withfocusonbusinessresults)• Influencepeerstobuildacoalitionforchangetoimprovequalityofdata
4. Investintraining• Trainleadersandassociatesinchangemanagement• Trainleadersandassociatesonnewframeworks,technologies,executionstrategiesfordataquality• Educateandraiseawarenessaboutnewdataqualitycapabilitiesamongstpeersandleaders
5. BuildWorldClassExecution• Pickspecificareaswithinyourdepartmentwhereyoucanshowtangibledirectimprovementtothebottomlinebyapplyingthenew
executionmodel
36Becomingdataconsciousandimplementingqualitycontrolrequiresculturechange,
investments,andalongtermview.