1 DataIntegrationChallengesinOnboardingOperationalData

DataIntegrationChallengesinOnboardingOperationalData

WHITEPAPER

Latest Update: September 21, 2011

www.expressor-software.com

2 DataIntegrationChallengesinOnboardingOperationalData

Executive Summary

Organizationsinawidevarietyofindustriesmustimport,normalize,andintegrateoperationaldatafromvariousinternalandexternalbusinesspartnersonanon-goingbasis.Thisprocessofonboardingoperationaldataisoftenfraughtwithcomplexityduetovariationsinthewaypartnersandsystemsrepresentdata—andinconsistenciesfromoneonboard-ingprocesstothenext.Thesecomplexitiesoftenimpactanorganization’sabilitytoscaleitsprocessesandsupportitsbusiness.

Thiswhitepaperexaminesvariouschallengesinvolvedwithonboardingoperationaldata,andpresentsaneffectivesolutionbasedonapowerfulenablingtechnologyfromexpressorthatsimplifiesandstandardizesonboardingprocessestoenableorganizationstoovercomethesebusinesschallenges.

Introduction

Onboarding operational datareferstotheprocessofintegratingdatafromvariousinternalandexternalpartnersintoanorganization’sinformationsysteminfrastructuretosupportanorganization’sbusinessprocesses.Theabilitytoquicklyandaccuratelyonboardoperationaldataisoftenacore,mission-criticalrequirement.

Manycompanies,inawiderangeofindustries,havebusinessrequirementsforonboardingoperationaldatafromexternalsources—includingfromotherdepartmentswithinthecompany,aswellasfromthirdpartybusinesspartners—intotheirstructureddatamanagementenvironment.Whilesomeorganizationsonboarddatathatisdeliveredtotheminapredictableformat,manyorganizationsmustrespondinreal-timetonewdataformatsprovidedbynewpartners—creatingcomplexities,delays,anderrors.

Examplesoforganizationsthatmustquicklyandaccuratelyonboardoperationaldataincludethefollowing:

• B2BmanufacturersthatmustsynchronizeERPandotherdataacrossanetworkofsuppliers,subcontractors, anddistributors.• Pharmaceuticalvendorsthatneedtoanalyzethedistributionofdrugsbyprocessingthedailytransactionsofthe pharmaciesthatdispenseitsproducts.• Marketingservicesprovidersthatacceptdatafromcustomerstocreatenewcampaigns,performdataenrichment anddatacleansing,andotherservices.• Discountretailersthatdependonbeingabletorapidlyintegratewithnewsupplierstoquicklyofferandselltheir distressedinventory.• Servicesproviderssuchashealthinsurancevendorsor401-Kadministratorsthatmustquicklysupportnew enterprisecustomers.

Whileonboardingdataisoftenamission-criticalbusinessprocess,manyorganizationsdonottreatitasamission-criti-calcomponentoftheirenterprisedatastrategy.Asamission-criticalbusinessprocess,organizationscangaindramaticbusinessbenefitsbyimprovingaccuracyandefficiencyandreducingcostsandcomplexitiesassociatedwithdevelop-ing,executing,reusing,andmaintainingtheironboardingprocesses.

2www.expressor-software.com

3 DataIntegrationChallengesinOnboardingOperationalData

• Multiple Data Sources• Unique Artifact Types• Similar Data, Defined Differently

• Requires writing numerous programs & queries.• Can require modifying existing programs, creating a many custom programs.• Requires point-to-point data mapping.• Allows for limited development reuse across partners.• Increases in complexity with new data sources.• Is manual, costly, time consuming, and error prone.

• Data is successfully onboarded into an organization’s structured data management environment.

Custom Coding and “One-Off” Applications

Properly Structured and Transformed Data

Sourcing From Data Artifacts Typical Approach - Human Intervention Data Management Environment

XML

XLS

Challenges

Onboardingoperationaldataoftenrequireshandlingdatafromavarietyofsourcesandpartners,representedindissimilarformats.Coupledwithaneedtohandlemanydifferentdatasourcesandpartnersinarapidandefficientmanner,theprocessofonboardingoperationaldatatosupportbusinesspartnerscanbechallengingfororganizations.

Thedata artifactsthatareexchangedbetweenbusinesspartnersmaycontaindifferencesinthewaythatthedataisrepresented,andtheremaybeinconsistenciesfromoneonboardingtothenext,whichmakeitdifficulttodevelopstandardizeddataintegrationapplicationstohandletheprocess.Forexample,onepartnermaycreateanExcelspreadsheetwithadataelementcalled“Acct_Num,”definedasa16characterlengthalphanumericfield,whichisthedatabaserepresentationofthe“AccountNumber.”AnotherpartnermaycreateaCSVfilewith“Account_Nbr”definedasaninteger.Whilebothdataelementsarefunctionallythesame,theyarerepresentedindifferentformatsthatrequiredifferentprocessingatthedataintegrationlevel.Thesearethesortofinconsistenciesthatmakeitchallengingtocreateefficientandreusabledataprocessingapplicationsthatsupportmultipledatasourcesandbusinesspartners.

www.expressor-software.com

“Onboarding operational data is a source of pain for many organizations. We hear this again and again. Organizations need to normalize data from different clients or partners, and they need a solution that lets them adapt easily to each new onboarding requirement, which is where expressor can really help. expressor enables organizations to onboard new data much faster and more efficiently through a reusable data integration framework. With expressor, organizations will no longer need to write or modify one-off, custom programs and queries, which will simplify the onboarding process and reduce the maintenance associated with the traditional approach. Organizations will need fewer resources, while still reducing the time needed to add new customers or partners, enabling them to scale without adding significant costs to their business process.”

Wiqar Chaudry, expressor Product Manager

Figure 1: Workflow and Challenges Onboarding Operational Data

4 DataIntegrationChallengesinOnboardingOperationalData

• Creates a reusable onboarding infrastructure.• Simplifies data mapping with reusable Semantic Types.• Requires fewer technical resources.• Creates a more manageable data integration environment.• Supports a scalable process, which supports more files, more customers, and more data.

Map Data usingSemantic Data Types

Data is Now Onboarded Into Your Structured Data

Management Environment

Benefits of Semantic Data Integration

XML

XLS

Properly Structured and Transformed Data

Inaddition,apartfromthechallengesofintegratingnewdatafromnewpartnersandsources,themetadatathatdefinestheunderlyingdataisoftendifferentaswell.

Asaresult,itisdifficulttostandardizeandreuseexistingonboardingapplications,whetherdevelopedusingdataintegrationtoolsorcustomcodingtechniques,toaccommodatenewpartnersanddata.Organizationsareforcedtorepeatedlydevelopnewapplicationstohandle“one-off”dataartifactsincustomizedwaysforeachnewpartnerordatasource.Thisgrowinglibraryofone-offroutinesiscumbersome,expensive,andinefficientfororganizationstomanageandmaintain.Inaddition,thesubtledifferencesbetweenthedataelementsrequiretediousattentiontodetail,oftencausingdelaysanderrors.Traditionalapproachestoonboardingoperationaldataaretypicallymanuallyintensive,expensive,timeconsuming,anderrorprone.

Finally,asthevolumeofdata,dataelements,andfrequencyofonboardingeventsincrease,thechallengesandcostsassociatedwithprocessingthesedataartifactsaremagnified—creatingissuesaroundanorganization’sabilitytoscaleitsprocess,andultimatelyscaleitssupportofthebusiness.

expressor software: An Ideal Approach To Onboarding Operational Data

DataintegrationandETLtoolsareuniversallyrecognizedtobemoreefficientthancustomcodingandmanualdatamanipulationtechniquesfortheinitialdevelopmentandre-useofapplications.However,theuniquechallengespre-sentedbymanyonboardingtasksmakeapplicationre-usedifficultorimpossible,evenformostdataintegrationtools.

expressorsoftwareprovidesaradicallysimplifiedapproachtodataintegrationthatfarsurpassestheapplicationre-usecapabilitiesofeveryotherdataintegrationsoftware(aswellasthoseofcustomcoding),makingitidealforonboardingoperationaldata.expressorhaspioneeredauniquemetadata abstraction frameworkthatenablesunprecedentedre-useofdata,metadata,anddataintegrationartifacts,tofinallybringre-useandsimplicitytotheprocessofonboard-ingoperationaldatafrombusinesspartners.Perfectlysuitedtothisusecase,expressorisafast,reliable,easy-to-useplatformforaccessing,transforming,anddeliveringdatafromvirtuallyanybusinesssysteminanyformat.

www.expressor-software.com

Figure 2: Semantic Data Integration simplifies Onboarding Operational Data

5 DataIntegrationChallengesinOnboardingOperationalData

ThefoundationoftheexpressorplatformisapowerfulmetadatalayercalledtheSemantic Framework.TheSemanticFrameworksimplifiesandstandardizesthecomplexandoftenmessyprocessofdataintegrationapplicationdesign.ItemploysreusabledataobjectsreferredtoasSemantic Typesthatenableorganizationstodefine,inbusinessterms,howthedatashouldberepresented.SemanticTypeartifactsholdinformationsuchasfieldnamesanddatatypes,whichareautomaticallycreatedandmappedtoaschemaartifact,whichisareusablepieceofmetadatathatdescribestheexternaldatalayoutwhenanexternaldatasourceisread.Theauto-generatedSemanticTypeinheritsthecharac-teristicsoftheschemabydefault,simplifyingthetaskofworkingwithdisparateanddissimilardatasources.SemanticTypesarereusable,abstracted,internaldatastructuresthatcanbemappedtomultipledatasourcesortargets.TheyenableexternaldatatypestoautomaticallyadapttotheinternaldatatypesdefinedintheSemanticType,whichenablestheSemanticFrameworktobeeasilymodifiedtohandlenewdatasources,andautomaticallyinsulatesapplicationsfromschemachangesthatmightoccurinadatasourceortarget.

Semantic Types Deconstructed

SemanticTypesrepresentareusableabstractinternaldatastructurethatcanbeuserdefinedorautomaticallygeneratedfromthedata’ssourceschema.UserscanfurtherextendthecapabilitiesofSemanticTypesbyaddingconstraintstoeachattributeofaSemanticType.AnapplicationcanutilizevariousSemanticTypes:

Local Composite Type–Alocaltypeisanartifactthatisauto-generatedwheneveranewdatasourceisloaded.Alocaltyperepresentsaninternaldatastructurethatismappedtoaschemaartifact,whichdefinestheexternaldatastructureofthedatasourceortarget.

Shared Composite Type–Asharedtypecanbealocaltypethathasbeenpromoted,oritcanbeagenericstructurethatcanbedefinedwithintheStudioapplication.ThebenefitofthistypeofinternaldatastructureisthatitcanbereusedacrossmanyexternaldatasourcesandtargetswithoutaffectinganyoftheinternalETLlogicwrittenagainstit.

www.expressor-software.com

expressor Artifacts

Semantic Types define the structure of internal data.

Read Operator Transform Operator Read Operator

Semantic Types define the data type and constraints for data processed within a Dataflow.

Composite Types map to Schema records. Atomic Types are used in Composite Type attributes.

Attributes are mapped with rules in transform operators.

Dataflows Connections Semantic TypesSchemas

Schema Mappings Type Type Mappings SchemaAttributes Rules Attributes

Figure 3: Semantic Types in expressor

6 DataIntegrationChallengesinOnboardingOperationalData

Shared Atomic Type–AnatomictypeisthefundamentalbuildingblockofSemanticTypesandrepresentsasingleattributethatcanbeutilizedinoneormorecompositetypes.Constraints–Constraintsaredefinedonindividualattributesofacompositetype.Eachattributecarriesthefollowinginformationaspartofitsdefinition:

• Thenameoftheattribute,whichshouldbedescriptiveforhumanconsumption.• Aprimitivedatatypewhichisassignedtotheattribute(supporteddatatypes:string,integer,decimal,double, date-time,byte).• Constraints,whicharesetsofrulesthatthedatarepresentedbytheattributemustconformtobasedondatatype (exampleconstraints:minimumvalue,maximumvalue,minimum,regularexpressions,allowedvalues).

Onceaconstraintisdefined,theapplicationcanspecifythecorrectiveactionthatshouldbetakeniftheruleisviolated.Bydefault,allcorrectiveactionsaresettoescalateerrorstothedataflow,andtheapplicationcanreject,skip,orredirectdataandtakeappropriateactions.Anapplicationcanbesettocorrectdatawithintheconstraintdefinitionbyprovidingdefaultvaluesorotheravailableactionsbasedontheattributedatatype.

Constraintsareappliedtoattributesofcompositetypesoratomictypesandenabletheconstraintinformationtobecarriedforwardwhereveragivenattributeisused.Thisstreamlinesthedevelopmentprocessbydecouplingdatacleansinglogicfromanyactualtransformationthatneedstooccurtothedata.Allconstraint-relatedlogiciscentralizedandmanagedwithintheSemanticTypeartifactenablingeasiermaintenanceandartifactreusability,makingitidealforhandlingthewidevariationindataartifactsduringonboardingprocessing.

Simplified Data Mappings

TheSemanticFrameworksimplifiesthetypicallyhighlycomplexdatamappingprocessbyreplacingthepoint-to-pointmappingdoneintraditionalETLtoolswithaunique,streamlinedmappingstructureachievedthroughtheuseofSemanticTypes.Bymappingsourceandtargetdatasetstobusinessdataobjects,expressorautomaticallyreconcilesdatatypediscrepancies.

www.expressor-software.com

Figure 4: Constraints and Error Corrections Interface in expressor

7 DataIntegrationChallengesinOnboardingOperationalData

expressorispurpose-builttoalloworganizationstoreuseSemanticTypeartifactsasmanytimesasnecessary,enablingorganizationstobuildasingleapplicationthatcanbeappliedtothewiderangeofbusinesspartnersanddataformatsdictatedbyitsbusinessrequirements.SemanticTypedefinitionsincludemappings,datatypeconversions,constraintdefinitionsanderrorhandlingactions.expressor’suniqueapproachtodataintegrationensuresthatallchangesareautomaticallyandaccuratelypropagatedthroughalldataintegrationapplications.

Onboarding Operational Data Case Study: Managed Service Provider

Amarketingservicesprovider(MSP)servesalargenumberoffinancialinstitutions,providingvariousformsofemailcommunicationsandcampaignstoeachfinancialinstitution’scustomers.

Aspartofeachclientengagement,thefinancialinstitutionssubmittheirowncustomerfilestotheMSP,eachinauniqueformat.TheMSPmustvalidate,normalize,andimportthedataineachcustomerfileintoitsdatabasebeforecampaignscanberun.Thecompanyoperatesontighttimeframes,executingmultiplecampaignsforarangeofclientsacrossmultipletimezones.

Thecompany’sinitial,custom-codedsolutionforonboardingdatafromitsclientswasveryrigid.Sincenewmappingsandtransformationswererequiredforeachnewcustomerandcampaign,“developmentwasveryslow,andmainte-nancewasanightmare,”accordingtothecompany.Moreover,thecompanyhassevenemployeesthatcanfunctionasanalysts,butonlyonedeveloper.Thecompanyneededaflexibleonboardingsolutionthatwouldallowthesevenanalyststoquicklycustomizetheapplicationfornewcustomersandcampaigns.

Thecompanyimplementedexpressorsoftwaretodevelopaflexible,genericapplicationtoreadcustomerfiles,performvalidationanderrorhandling,mapandtransformthedataintheincomingfilestomatchtheschemaintheproductiontables,loadnewrecordstotheproductiontable,andwritethepriorrecordstoahistorytable.

www.expressor-software.com

Figure 5: expressor’s Simplified Mapping Process

Traditional Mapping Simplified Mapping

SemanticType

8 DataIntegrationChallengesinOnboardingOperationalData www.expressor-software.com

WhereastheMSPsprioronboardingapplicationrequired80hourstodevelop,theexpressorsoftwareapplication,builtusingexpressor’sgraphical,drag-and-drop,color-codedinterfacewasbuiltfourtimesfaster.

Withthepreviousapplication,eachnewcustomerorcampaignrequiredanewmapping,whichtookeighthoursofthedeveloper’stime.WithexpressorandtheuseofSemanticTypes,thenewfilesareincorporatedbyre-usingtheexistingapplicationthroughexpressor’sgraphicaldevelopmentenvironment.Thenewcustomerandcampaignmappingsarecreatedeighttimesfasterandarecreatedbythecompany’sanalysts,freeingthedevelopertoworkonmorecriticaltasks.

Thecompany’ssevenanalystsarenowabletoaccommodatetheirnewcustomersandcampaignsusingexpressor’sgraphicalenvironment,whichrequiresnocodingorcompiling,allowingthemtoeasilymapnewfileformatstothecom-monSemanticType.Andalloflogic,includingtransformations,fieldvalidations,anderrorhandlingareautomaticallymaintained—dramaticallyspeedingandsimplifyingnewclientengagementsandenforcingstandardization.

Benefits

expressorsoftwareprovidesanidealplatformforcompaniesthatmustperformonboardingofoperationaldatatosupportmultiplepartnersandcriticalbusinessprocesses.Organizationsusingexpressorsoftwareforonboardingoperationaldataareexperiencingmeasureablebenefits:

Speed of development. Withexpressor,customersdownload,install,build,andruntheirfirstapplicationinminutes.expressoriseasytodownload,install,learn,anduse.Itprovidesafamiliar,purpose-builtUIwithanMSOffice-likelookandfeel.Itprovidesdrag&dropconfigurationandusessimplified,graphicaltoolsfordefiningreusabledatamappingsandbusinessrulestospeedandsimplifythedevelopmentofdataintegrationapplications.

Ease of re-use.Withexpressor,customerscanreuseeverything,makingitidealforhandlingthechallengesandcom-plexitiesofonboardingoperationaldatafrombusinesspartners.Organizationscandefineandreuseanyprojectartifactfromsimplefileconnectionstocomplexdataflowsandbusinessrules.Userscandesignandreusetransformationoperators,dataflows,datasourceconnections,andtransformationlogic.Moreover,allprojectartifactscanbesharedandreusedwithinasingleworkspaceacrossmultipleprojects.

Accessible to a wider range of skill sets. Simplifiedre-usemeansthatlesstechnicaluserscanadapttheapplicationstohandlenewdatasourcesandbusinesspartners,freeingdeveloperstoworkonmorecriticaltasks.

Increased accuracy and standardization.Definingstandardapplicationsthatenforcedatamappings,constraints,anderrorhandlingactionsensuresconsistencywithnewdataandpartners,especiallycomparedwithcustomcodinganddevelopingone-offapplications.

Summary

Organizationsacrossawiderangeofindustriesthatmusteffectivelyonboardoperationaldataareexperiencingdra-maticbusinessbenefitsbyimplementingexpressorsoftware’sdataintegrationplatform,includingimprovedaccuracy,greaterefficiencies,andreducedcostsandcomplexityassociatedwithdeveloping,executing,reusing,andmaintainingitsonboardingprocesses.

9 DataIntegrationChallengesinOnboardingOperationalData©2010expressorsoftwarecorporation.expressorandredefiningdataintegrationaretrademarksofexpressorsoftwarecorporation.Allothertrademarksortradenamesarepropertiesoftheirrespectiveowners.Allrightsreserved.

©2011expressorsoftwarecorporation,allrightsreserved.Thefollowingaretrademarksofexpressorsoftwarecorporation:expressor,SmartSemantics,IntelligentLoadandGo.

expressorsoftwarecorporation1NewEnglandExecutiveParkBurlington,MA01803USAwww.expressor-software.com

About expressor

Atexpressorsoftware,we’resimplifyingdataintegrationbyprovidingsoftwarethatiseasytouse,powerful,andafford-able—enablingorganizationstoaccomplishmorewithfewerresources.Wedramaticallysimplifythevastmajorityofbasicdataintegrationtasks—makingitfareasiertomapandmovedatafromonelocationtoanother.And,whendataintegrationbecomesmorecomplicated—requiringvarioustypesofdatatransformations—wehelpmakedataintegrationandETLprojectsmoreflexibleandefficientbyenablingthecreationandsubsequentreuseofpowerfulartifacts,con-figurations,andbusinessrules.Thismeansthatwhenacomplexbusinessdatachallengeissolvedonce,thesolutioncanbereused.

expressorhasarapidlygrowingcommunityofover12,500uniqueexpressorStudiodownloaduserstoday.LeadingorganizationssuchasSKECHERSUSA,AmericanTower,Viverae,WesternWorld,LABVANTAGE,andClickRSVPhaveadoptedexpressor’saffordabledataintegrationplatformforserverdeploymenttohelpthemintegrateon-premisesandclouddatasourceswiththeirbusinesscriticalapplications.

expressorStudioisthedesigncomponentoftheexpressorDataIntegrationPlatform,andexpressormakesafreedownloadofStudioavailableatwww.expressorStudio.com.

expressorsoftwareisheadquarteredinBurlington,MA.Formoreinformationabouthowexpressorissimplifyingdataintegration,pleasevisitourwebsiteatwww.expressor-software.comorcontactusdirectlyat+1(781)505-4190x222.