Teradata Migration

11
© Bolder Technology, Inc. 2011 Page 1 of 11 A BTI Case Study Data Warehouse Migration Strategies: OracletoTeradata Experiences April 2011 The white paper examines the strategies and issues for migrating data warehouses from Oracle platforms to the Teradata platform. It discusses the motivations and approaches to data warehouse migrations, gives examples of experiences, and concludes with benefits realized, lessons learned, and a synthesis of key points. The research was conducted using anonymous interviews with persons who had direct experiences migrating data warehouses from Oracle to Teradata® Database. Although focusing on specific vendors, this white paper surfaces the generic issues that are useful to IT professionals involved with migrating a data warehouse between database platforms. Motivations for DW Migrations ............................................... 2 Approaches to DW Migrations ................................................ 3 Migration Project Experiences ................................................. 5 Resources ................................................................................. 7 Migration Project Lessons Learned .......................................... 8 Synthesis .................................................................................. 9 About Bolder Technology, Inc. ............................................... 11 About the Sponsor ................................................................. 11

Transcript of Teradata Migration

©  Bolder  Technology,  Inc.  2011     Page  1  of  11  

    A  BTI  Case  Study  

Data  Warehouse  Migration  Strategies:  Oracle-­‐to-­‐Teradata  Experiences  

April  2011  

The  white  paper  examines  the  strategies  and  issues  for  migrating  data  warehouses  from  Oracle  platforms  to  the  Teradata  platform.  It  discusses  the  motivations  and  approaches  to  data  warehouse  migrations,  gives  examples  of  experiences,  and  concludes  with  benefits  realized,  lessons  learned,  and  a  synthesis  of  key  points.  The  research  was  conducted  using  anonymous  interviews  with  persons  who  had  direct  experiences  migrating  data  warehouses  from  Oracle  to  Teradata®  Database.    Although  focusing  on  specific  vendors,  this  white  paper  surfaces  the  generic  issues  that  are  useful  to  IT  professionals  involved  with  migrating  a  data  warehouse  between  database  platforms.    

Motivations  for  DW  Migrations...............................................2  Approaches  to  DW  Migrations ................................................3  Migration  Project  Experiences .................................................5  Resources.................................................................................7  Migration  Project  Lessons  Learned..........................................8  Synthesis ..................................................................................9  About  Bolder  Technology,  Inc. ...............................................11  About  the  Sponsor .................................................................11  

   

©  Bolder  Technology,  Inc.  2011     Page  2  of  11  

 The  VP  of  Finance  looked  a  bit  confused.  She  had  carefully  listened  to  a  well-­‐rehearsed  presentation  by  the  CIO  about  the  migration  of  three  financial  databases  into  a  single  financial  data  warehouse.  The  justification  was  the  cost  savings  from  avoiding  major  future  enhancements  to  support  future  business  needs.  After  a  long  pause,  she  asked,  “The  new  and  old  databases  are  based  on  the  SQL  standard.  Why  does  the  project  cost  so  much  and  take  so  long?”  She  seemed  more  puzzled  than  upset.    

Have  you  faced  a  similar  situation?    

This  is  a  common  situation  for  many  corporations  today.  Data  warehouse  (DW)  migration  projects  are  a  high-­‐priority  item  on  most  IT  agendas.  Yet,  they  are  poorly  understood  by  many  IT  professionals  and  especially  by  most  business  executives.  The  project  seems  deceptively  simple,  since  one  would  imagine…  Just  move  data  from  one  SQL  standardized  database  to  another.  This  project  would  seem  to  be  a  simple  ‘fork-­‐lift’  project  where  we  pick  the  data  up,  move  it,  and  set  it  down.    

The  real  situation  is  more  complex,  both  from  a  business  and  technical  perspective.  Yet,  understanding  about  DW  migration  projects  can  substantially  reduce  the  expected  cost  and  duration.    

The  practice  of  data  warehousing  is  now  more  than  twenty  years  old.  The  first  and  even  second  generations  of  data  warehouses  are  in  need  of  major  renovations.  Most  corporations  grow  incrementally,  and  a  DW  is  a  central  component  of  those  large  organizations.  Each  year  the  data  warehouse  changes  a  little  bit  to  accommodate  growth.  Over  time,  these  incremental  changes  result  in  a  messy,  fragmented  system  with  high  maintenance  costs  and  limited  growth  potential.  Further,  many  corporations  justify  development  of  their  data  warehousing  on  a  case-­‐by-­‐case  basis,  which  usually  creates  many  separate  data  marts,  each  of  which  is  focused  on  its  particular  business  issue  with  its  specific  set  of  data.    

The  underlying  technology  for  data  warehousing  has  dramatically  evolved  over  the  past  decade  so  that  capabilities  and  costs  that  were  not  feasible  in  the  past  are  now  quite  adequate  and  acceptable.  In  particular,  the  shift  from  large  single-­‐processor  architecture  to  Massively  Parallel  Processing  (MPP)  architecture  like  Teradata’s  has  revolutionized  data  warehousing.    

The  challenge  is  to  decide  when  and  how  to  evolve  your  DW  systems  and  to  do  so  based  on  clear  business  objectives.  This  white  paper  will  discuss  the  general  motivations  and  approaches  for  DW  migrations,  but  focuses  specifically  on  migrating  from  Oracle  platforms  to  the  Teradata  platform.  The  white  paper  concludes  with  benefits  realized,  lessons  learned,  and  a  synthesis  of  the  key  points.    

Motivations  for  DW  Migrations  The  motivations  for  most  DW  migration  projects  are  a  mixture  of  business  and  technology  reasons.    

Cost    

Cost,  both  current  and  future,  drives  most  DW  migrations,  often  in  combination  with  the  other  factors  we  will  discuss  later.  Because  of  technology  innovations,  current  cost  could  be  dramatically  reduced,  especially  if  many  small  data  warehouses  are  consolidated  into  a  single  data  warehouse.  Current  technology  can  often  support  such  consolidation  at  a  reduced  cost,  even  though  this  was  not  possible  just  a  few  years  ago.    

Cost  for  a  data  warehouse  ranges  from  hardware/software  purchases/subscriptions  to  maintenance/support,  both  for  the  initial  development  and  then  for  maintenance  and  enhancements  required  over  the  coming  years.  When  managing  a  data  warehouse,  a  company  not  only  monitors  current  costs  but  also  projects  its  future  costs.  These  future  costs  are  often  a  key  motivation  for  DW  migrations  since  the  projections  show  steady  (perhaps  exponential)  increases  as  key  business  processes  continue  to  grow.  Also,  new  business  initiatives  or  corporate  acquisitions  and/or  mergers  often  and  suddenly  multiply  future  data  requirements,  both  in  volume  and  velocity.    

©  Bolder  Technology,  Inc.  2011     Page  3  of  11  

Integrated  Corporate  View  

Another  reason  for  migrating  is  the  business  value  of  having  an  integrated  view  of  the  business.  The  complexity  of  any  global  company  is  mind-­‐boggling,  resulting  in  decision  paralysis  (or  decision  stupidity).  The  enterprise  data  warehouse  is  the  only  place  where  an  integrated  business  view  is  possible.  In  particular,  critical  business  opportunities  (and  business  problems)  are  often  hidden  in  the  “cracks”  between  functional  units.  Understanding  cross-­‐functional  dynamics  becomes  critical  to  growing  the  business  in  a  changing  marketplace.    

Scalability  and  Flexibility  

From  the  technology  perspective,  factors  such  as  scalability,  flexibility,  and  stability,  complement  the  business  reasons.  Companies  in  fast-­‐paced  industries  will  often  stress  scalability  to  support  information  requirements  of  possible  mergers  and  acquisitions.  Further,  these  companies  also  stress  flexibility  to  support  information  requirements  for  unanticipated  business  initiatives.    

Stability  and  Reliability    

Large  global  companies  will  stress  stability  of  their  data  warehouse  since  any  disruption  of  services  by  the  data  warehouse  will  reduce  business  efficiency  and  result  in  monetary  losses  to  the  business.  

Any  combination  of  the  above  reasons  will  justify  a  DW  migration  project,  provided  that  the  new  platform  will  reduce  costs  and  improve  the  ability  of  the  business  to  cope  with  its  future  business  problems.  

Approaches  to  DW  Migrations  This  section  is  an  overview  of  the  various  types,  scoping,  and  tasks  to  approach  a  DW  migration.    

DW  Migration  Types  

In  general,  there  are  three  approaches  to  DW  migrations:  Fork-­‐Lift,  Redesign,  and  Evolutionary.  

First,  the  Fork-­‐Lift  (or  Lift  and  Shift)  approach  is  a  DW  migration  that  has  a  simple,  one-­‐to-­‐one  mapping  of  the  old  DW  design  into  the  new  DW.  If  possible,  names  and  attributes  for  all  database  tables,  columns,  and  other  database  objects  are  preserved  to  minimize  changes  to  applications  that  update  or  query  the  data.  The  Fork-­‐Lift  approach  is  the  easiest  and  quickest  way  to  a  DW  migration.  The  Fork-­‐Lift  approach  does  not  provide  as  much  business  value  as  other  approaches  since  the  business  functionality  of  the  data  warehouse  remains  unchanged.  Further,  many  DW  migrations  involve  the  transition  from  a  Symmetric  Multi-­‐Processing  (SMP)  platform  to  a  MPP  platform,  thus  requiring  database  changes  even  if  the  logical  design  is  not  changed.  The  goal  of  the  Fork-­‐Lift  approach  is  to  achieve  value  quickly  from  performance  and  consolidation.  

Second,  the  Redesign  approach  is  a  DW  migration  that  modifies  the  logical  design  in  the  old  DW  to  be  consistent  with  the  new  DW.  The  DW  redesign  usually  consists  of  consolidating  several  data  marts  with  dimensional  star  schema  into  a  quasi-­‐enterprise  normalized  3NF  schema.  Data  from  the  old  DW  is  extracted  and  extensively  transformed  to  load  properly  into  the  new  database  structures.  The  effort  required  for  this  type  of  DW  migration  is  proportional  to  the  degree  of  information  integration  designed  into  the  new  DW.  The  goal  of  the  Redesign  approach  is  to  achieve  value  from  a  more  integrated  view  of  the  business,  thereby  allowing  the  company  to  pursue  evolving  business  opportunities.    

Third,  the  Evolutionary  approach  is  a  DW  migration  that  incorporates  the  previous  two  types  and  incrementally  evolves  several  old  data  warehouses  into  a  new  data  warehouse.  Although  this  type  of  DW  migration  may  span  many  months,  the  effort  is  actually  a  sequence  of  smaller  projects  that  initially  perform  Fork-­‐Lift  of  the  existing  data  to  the  new  DW  platform  and  then  gradually  emphasize  DW  redesign  and  integration  of  that  data.  One  technique  is  to  identify  duplicate  data  in  the  old  DW  systems  and  consolidate  this  data  in  the  early  stages  of  the  effort.  This  duplicated  data  is  usually  key  reference  data,  such  as  geographic  regions  and  tax  tables,  which  should  be  common  across  the  company.  The  goal  would  be  to  achieve  value  quickly  with  better  performance  and  then  with  enhanced  business  functionality.    

©  Bolder  Technology,  Inc.  2011     Page  4  of  11  

For  larger  DW  migrations,  the  evolutionary  approach  is  more  appropriate,  especially  when  consolidating  multiple  data  warehouses.  In  other  words,  the  goal  of  these  larger  DW  migration  projects  is  not  the  destination;  it  is  a  journey  toward  a  data  warehouse  that  is  more  responsive  to  business  needs.    

Further,  an  experienced  migration  specialist  observed,  “Each  DW  migration  is  unique,  requiring  each  company  to  carefully  examine  their  goals  and  plan  their  migration  accordingly.”1  

Migration  Project  Scoping  

DW  migration  projects  span  the  extremes  in  cost  and  duration.  Good  planning  by  people  who  have  the  right  experience  and  skills  can  reduce  cost  and  duration  by  factors  of  ten  or  more.  The  companies  researched  for  this  report  shared  their  DW  migration  experiences  spanning  small  projects  using  from  three  to  five  persons  for  three  months  to  large  projects  using  a  hundred  persons  for  many  months.  Larger  projects  were  actually  managed  as  multiple  smaller  projects  (nine  months  maximum)  that  overlapped  in  their  sequencing.  More  typical  projects  were  in  the  range  of  four  to  six  months.    

Project  costs  involved  the  usual  employee  salaries,  but  also  off-­‐shore  operations,  professional  services,  hardware  purchases,  software  tool  purchases,  and  special  training  programs.    

Figure  1  shows  a  typical  project  timeline  from  the  initial  project  planning  through  production  and  decommissioning  old  systems.    

 

Figure  1  -­‐  Typical  Project  Timeline2  

 

Migration  Project  Tasks  

The  major  tasks  (as  starred  in  Figure  1)  consist  of  the  following  activities:  

First,  the  Database  Conversion  and  Data  Migration  task  consists  of  converting  the  Oracle  database  objects  (table/column  structures,  indexes,  partitions,  etc.)  into  compatible  objects  on  the  Teradata  platform  and  then  loading  the  revised  data  into  the  Teradata  tables.    

Second,  the  ETL  Process  Conversion  task  consists  of  analyzing  and  rewriting  the  scripts  that  primarily  drive  the  data  flows  into  the  new  data  warehouse.  Within  these  scripts  are  various  SQL  statements  that  retrieve  and  transform  

©  Bolder  Technology,  Inc.  2011     Page  5  of  11  

data  values  from  source  systems  and  then  load  into  specific  columns  of  tables  in  the  new  data  warehouse.  The  conversion  effort  focuses  on  analyzing  each  SQL  statement  and  assessing  whether  it  will  execute  properly  and  efficiently  in  the  new  MPP  environment.    

Third,  the  Application  and  BI  Conversion  task  consists  of  analyzing  and  rewriting  the  SQL  statements  embedded  in  either  application  programming  code  (like  COBOL)  or  BI  query  packages  (like  SAP  Business  Objects).  In  some  ways,  this  task  is  similar  to  the  ETL  Process  Conversion  task  mentioned  above.  However,  a  migration  specialist3  noted  that  if  the  program  uses  a  row-­‐based  (row-­‐at-­‐a-­‐time)  logic  (i.e.,  read-­‐process-­‐write  each  row),  the  program  cannot  be  decomposed  for  parallel  execution  and  thus  take  advantage  of  the  efficiencies  of  MPP  architecture.  This  logic  must  be  changed  to  a  set-­‐based  (set-­‐at-­‐a-­‐time)  logic,  which  can  be  difficult.  This  is  true  of  any  migration  to  MPP  platforms.  

Most  BI  query/reporting  packages  (like  MicroStrategy)  are  designed  with  set-­‐based  processing,  rather  than  row-­‐at-­‐a-­‐time.  Hence,  these  queries  and  reports  usually  convert  quickly  with  little  effort.  However,  older  PL/SQL  scripts  used  by  the  Oracle  databases  often  use  row-­‐based  processing  and  should  be  converted,  with  some  effort,  to  set-­‐based  processing.        

In  the  coming  sections,  we  will  discuss  the  experiences  with  DW  migration,  along  with  the  resources  available  for  conducting  DW  migrations.  In  particular,  Teradata  has  developed  tools  that  automate  many  of  the  migration  tasks,  especially  the  conversion  of  row-­‐based  to  set-­‐based  logic,  as  described  in  a  later  section.    

Migration  Project  Experiences  This  section  sketches  the  situations  with  several  recent  experiences  with  DW  migrations  from  Oracle  to  Teradata  Database.    

Large  Manufacturing  Firm  

Responsiveness  and  scalability  are  vital  because  their  business  is  growing  in  size  and  in  complexity.  The  company  needed  a  data  warehouse  platform  that  could  grow  along  with  them.  Many  of  their  researchers  were  creating  hundreds  of  sophisticated  algorithms  to  explore  new  approaches  to  discovering  defects.  The  Teradata  engine  allowed  the  execution  of  those  algorithms  on  massive  amounts  of  data.    

A  senior  manager  explained  their  situation  as  follows:    

“Where  Teradata  shines  is  the  speed  of  ad-­‐hoc  analyses.  We  need  quick  answers  to  the  ‘what  if’  questions.  With  Teradata,  we  were  able  to  pull  bad  product  off  of  pallets  without  quarantining  the  entire  shipment.  This  quickly  produced  kudos  with  our  customers  as  well  as  the  internal  staff.    

The  story  of  a  recent  acquisition  of  the  Teradata  system  is  amazing  and  instructive.  When  daily  data  loads  to  the  Oracle  data  warehouse  began  requiring  18  hours,  the  development  manager  and  his  team  knew  that  the  old  Oracle  system  was  “running  up  against  the  wall”  and  needed  to  be  replaced.  In  January  of  2008,  the  team  started  procurement  procedures  for  a  new  data  warehouse  platform.  They  selected  the  Teradata  Data  Warehouse  Appliance  2500.  When  asked  about  the  reasons  for  their  choice,  the  development  manager  stated,  “No  one  else  came  close  to  the  price/performance  or  to  the  maturity.”    

The  purchase  was  approved  in  June  of  2008,  and  the  Teradata  system  was  delivered  one  week  later.  The  development  manager  described  that  hectic  time  saying,  “The  box  was  on  site  by  Friday;  all  the  pieces  were  in  place  by  Thursday,  and  in  production  by  the  following  Tuesday.”  The  conversion  team  consisted  of  the  manager  and  two  database  administrators.  The  team  was  pleasantly  surprised  that  it  took  less  than  three  weeks  from  purchase  to  production.  It's  a  testimony  to  the  quality  of  the  DBAs  and  the  system  they  had  architected  that  hundreds  of  conversion  steps  and  challenges  were  handled  literally  all  in  one  week.  The  old  system  was  phased  out  by  mid-­‐July.  

©  Bolder  Technology,  Inc.  2011     Page  6  of  11  

Large  Aerospace  Firm  

A  DW  migration  specialist4  sketched  the  situation  with  an  aerospace  company  that  is  migrating  about  30  financial  data  warehouses  from  Oracle  to  the  Teradata  platform.  Because  of  poor  integration  among  these  data  warehouses,  the  corporation  had  continuing  problems  with  rolling  up  corporate  financials.    

The  goal  of  the  DW  migration  project  was  not  to  create  a  single  consolidated  data  warehouse.  Rather  the  goal  was  to  bring  together  all  the  data  onto  one  platform  and  “de-­‐dup”  (eliminate  any  obvious  duplication  in)  the  data.  This  de-­‐dup  process  concentrated  on  those  tables  that  were  “reference  data”  for  detailed  data  and  should  be  common  across  the  corporation.  An  example  was  the  common  labor  rates  charged  by  various  projects.    

The  initial  DW  migration  project  focused  on  the  largest  database  because  its  data  warehouse  was  more  likely  “to  break”  on  Oracle  under  increasing  loads.  This  database  is  fairly  large,  with  about  two  terabytes  of  raw  user  data  in  more  than  10,000  tables.  The  migration  project  focused  on  about  5,000  tables,  1,000  Cognos  cubes,  and  500  ETL  jobs  (with  10,000  steps).    

The  project  required  six  months  for  this  database.  The  rest  of  the  databases  are  estimated  to  require  less  time,  estimated  at  90  to  120  days.  The  sequencing  of  these  databases  will  be  based  on  business  priorities.    

As  a  best  practice,  it  was  recommended  that  projects  should  not  last  longer  than  nine  months.  If  the  project  is  longer,  it  should  be  decomposed  into  smaller  projects,  each  with  its  specific  business  objective.    

He  also  stressed  the  importance  of  the  initial  “data  audit”  to  determine  what  data  is  actually  being  used  and  from  where  the  data  is  being  sourced.  Verifying  the  completeness  and  accuracy  of  this  information  is  complex  and  time  consuming.  A  DW  migration  project  will  typically  spend  more  than  half  of  the  time  doing  a  data  audit  before  any  data  conversion  or  coding  is  performed.    

Large  Healthcare  Company  

The  infrastructure  was  entirely  composed  of  SMP  platforms  running  Oracle  under  UNIX  with  EMC  storage  units.  This  technology  has  served  well  for  almost  a  decade,  but  the  company  concluded  that  they  were  unable  to  scale  and  satisfy  the  anticipated  ten-­‐fold  increase  in  data  volume  and  workloads.  Further,  the  company  had  outsourced  its  infrastructure  support,  implying  a  reduced  level  of  expertise  to  tune  and  optimize  multi-­‐tiered  architecture  composed  of  Oracle,  Sun,  EMC,  etc.  The  conclusion  of  their  IT  architects  was  the  need  for  an  MPP  database  platform.  

The  lead  development  manager  described  the  limits  of  their  current  Oracle  solutions:  

“The  complexity  of  using  Oracle-­‐based  solutions  was  increasing  as  the  data  volumes  were  growing.  The  company  needed  a  new  database  platform  that  would  scale.  Various  vendors  were  releasing  data  warehouse  appliance  products  that  appeared  to  offer  good  cost  performance  advantages.  Inspired  by  the  emergence  of  data  warehouse  appliances  from  Netezza  and  others,  the  company  began  surveying  the  market  for  MPP  database  technology  for  building  data  warehouses.”  

Their  IT  experts  reviewed  their  options  and  concluded  that  shared-­‐nothing  MPP  platforms  allowed  the  architectural  integration  of  hardware,  operating  system,  database  management,  and  storage  onto  one  system  that  could  be  optimized  for  their  specific  workloads.  The  MPP  technology  was  proven  to  support  ad-­‐hoc  queries  and  complex  analytics  against  multiple  terabytes  of  data,  a  task  that  was  very  uncertain  using  their  current  SMP  platforms.    

Shared-­‐nothing  MPP  platforms  were  also  designed  for  high  availability  with  component  redundancy  so  that  there    was  no  single  point-­‐of-­‐failure.  If  one  component  failed,  another  would  take  its  place.  The  disk  drives  were  able  to  be  “hot  swappable”  meaning  that  if  a  drive  failed,  other  drives  would  immediately  take  over  its  function.  And,  a  new  disk  drive  could  be  physically  swapped  with  the  failed  one,  without  causing  the  entire  system  to  be  shut  down.    

Finally,  the  most  important  characteristic  of  a  MPP  platform  was  its  ability  to  scale  in  its  capacity  to  store  data  and  process  queries.  There  was  plenty  of  evidence  from  prior  customer  experiences  that  if  a  company  has  a  two-­‐node  MPP  platform,  they  could  buy  two  additional  nodes  and  double  their  capacity  without  having  to  add  technical  support  staff.  

©  Bolder  Technology,  Inc.  2011     Page  7  of  11  

Large  Beverage  Company5  

Because  of  a  recent  merger,  this  company  decided  on  an  integrated  reporting  system  based  on  Teradata  technology.  Their  objective  was  to  deliver  business  value  quickly  by  creating  an  integrated  view  of  the  merged  business.  Starting  with  the  design  of  an  integrated  data  model,  the  company  is  consolidating  their  source  systems  onto  the  Teradata  Database.    

The  company  inherited  about  15  data  marts  on  the  Oracle  platform  and  needed  to  convert  them  to  the  Teradata  platform.  Within  six  months,  they  acquired  the  Teradata  platform  and  had  the  first  (and  largest)  data  mart  migrated  and  operating.  This  data  mart  consisted  of  one  billion  rows  that  required  one  terabyte  of  storage.  The  remaining  data  marts  are  estimated  to  contribute  another  terabyte  to  the  new  data  warehouse.  The  company  used  off-­‐shore  services  for  the  ETL  development,  which  involved  three  lead  persons  at  their  company  working  with  ten  off-­‐shore  developers.    

One  observation  was  that  working  with  MPP  platforms  like  Teradata  Database  required  a  different  mindset  for  designing  and  maintaining  DW  systems.  For  example,  a  DBA  with  Oracle  needs  to  choose  which  index  to  build  while  a  DBA  with  Teradata  Database  needs  to  monitor  the  index  to  determine  whether  or  not  it  is  used  properly.    

Resources  There  are  several  tools  that  can  assist  with  a  DW  migration.  A  new  offering  from  Teradata  spans  most  of  the  DW  migration  tasks.  Teradata  Migration  Accelerator  is  an  integrated  toolset  for  doing  various  DW  migrations  to  a  Teradata  platform.  The  initial  focus  of  this  tool  is  DW  migrations  from  Oracle  platforms,  although  additional  adapters  are  planned  for  migrations  from  other  platforms.    

Teradata  Migration  Accelerator  also  does  data  extraction  from  the  Oracle  database,  automatic  target  table  creation  in  Teradata  Database  based  on  the  tables  in  Oracle  and  data  load  and  auto  data  type  conversion.  In  particular,  the  tool  translates  various  Oracle  database  code  into  compatible  code  for  the  Teradata  Database,  such  as:  

• Oracle  SQL  and  PL/SQL  script  into  ANSI  SQL  • Oracle  SQLPlus  scripts  to  BTEQ  scripts  • Mixtures  of  SQL  DDL+DML  and  PL/SQL  into  ANSI  SQL  • Oracle  built-­‐in  SQL  functions  to  Teradata  functions  • Oracle  cursor-­‐based  processing  into  more  efficient  set-­‐based  processing  

Reflecting  on  his  experience  with  developing  migration  tools,  a  migration  tool  developer  remarked,  “In  the  past,  various  tools  were  written  by  field  engineers  to  migrate  data  warehouses  from  Oracle  platforms  to  the  Teradata  platform.  These  tools  performed  50  percent  to  70  percent  of  the  work,  leaving  the  remainder  to  manual  effort.  They  were  written  in  various  languages,  making  maintenance  and  enhancements  difficult.  These  tools  were  used  by  Teradata  professional  services  consultants  since  they  were  too  fragile  for  use  by  most  customers.”6    

Teradata  Migration  Accelerator  uses  Java  as  its  development  language,  which  allows  quick  extensibility  of  its  functions.  Features  like  security,  licensing,  and  other  administrative  functions  were  added,  along  with  collaboration  tools  for  multi-­‐person  projects.  A  metadata  object  inventory  can  track  the  conversion  of  thousands  of  tables.  Further,  Teradata  Migration  Accelerator  can  handle  several  thousand  ETL  routines,  which  may  be  required  to  maintain  a  thousand  tables.    

Finally,  there  are  several  Teradata  partner  tools  that  perform  specialized  conversion  tasks.  Wisdomforce  Fast  Reader  does  a  block-­‐level  extraction  from  Oracle  and  pushes  the  blocks  directly  into  Teradata  Database.  This  processing  is  very  fast  data  load,  especially  for  database  systems  that  are  CPU  bounded.  ZOHO’s  Swiss  SQL  does  on-­‐the-­‐fly  conversion  of  Oracle  SQL  into  ANSI  SQL.  Change  the  app  to  the  Swiss  SQL  API.  Not  PL/SQL  and  not  cursor-­‐based  logic  conversion.      

©  Bolder  Technology,  Inc.  2011     Page  8  of  11  

The  key  advantage  of  using  Teradata  Migration  Accelerator  and  these  other  tools  is  the  significant  reduction  of  cost  for  DW  migrations,  in  terms  of  duration,  skills  required,  and  person-­‐days.  An  example  given  was  a  large  12-­‐month  project  that  was  scaled  down  to  three  months  at  one-­‐third  the  cost,  through  the  use  of  these  tools.  

Migration  Project  Lessons  Learned  Here  are  some  lessons  that  the  companies  described  in  this  white  paper  learned  during  various  DW  migration  projects.  

Stay  Focused  on  These  Key  Success  Factors  

A  DW  migration  specialist  stated  that  a  successful  DW  migration  was  simple  if  you…    

a) Plan  everything  out.  b) Make  a  complete  inventory…of  everything  (tables,  ETL  scripts,  indexes,  etc.).  c) Track  your  progress.  

Clearly  State  the  Business  Value  of  the  Data  Warehouse  

The  beverage  company  noted  that  the  Teradata  system  is  a  large,  visible  cost  item,  as  compared  to  many  smaller  data  marts,  which  tend  to  be  embedded  in  various  budgets.  Although  the  total  cost  for  the  smaller  systems  may  be  greater,  the  Teradata  system  requires  a  clear  statement  of  business  value.  Further,  the  project  needs  a  business  sponsor  early  in  the  process.  

Leverage  the  Opportunity  

A  DW  migration  specialist  phrased  the  benefits  of  the  migration  project  differently.  Before  a  DW  migration,  the  typical  company  has  “hit  a  wall”  and  cannot  see  other  ways  of  using  the  data.  After  the  DW  migration,  the  company  is  at  “the  next  plateau”  and  can  see  many  more  uses  for  the  data.  However,  he  cautioned  companies  not  to  migrate  for  the  sake  of  migrating.  There  must  be  a  clear  business  justification.    

Invest  in  Data  Integration  

Facing  a  recent  corporate  merger,  the  beverage  company  recommended  that  their  focus  of  DW  migration  should  be  on  data  integration  so  that  the  corporation  can  understand  “how  well  they  are  doing.”    Further,  this  migration  must  be  implemented  with  a  timeframe  that  delivers  business  value  quickly.    

Although  data  integration  during  the  DW  migration  has  huge  business  value,  it  is  tough  to  do!  The  large  healthcare  company  chose  a  stepwise  approach  that  balanced  payoffs  in  business  value  with  constraints  in  technical  resources.  The  approach  is  to  consolidate  each  domain  into  a  common  environment,  then  rationalize  each  domain,  and  finally  integrate  the  common  elements  into  a  single  enterprise-­‐wide  view  of  the  business.  This  is  a  long-­‐term  incremental  strategy  that  has  a  high  probability  of  success  and  has  flexibility  to  respond  to  unexpected  industry  changes.  

Adopt  an  Incremental  Strategy  

The  beverage  company  strongly  recommended  avoiding  the  “Big  Bang”  strategy.  A  data  warehouse  should  be  evolved  incrementally.  For  instance,  they  are  trying  to  add  a  new  subject  area  every  year.  

Transition  to  MPP  Requires  New  Thinking  

Several  interviewees  mentioned  that  the  transition  to  an  MPP  data  warehouse,  such  as  a  Teradata  system,  requires  new  thinking  about  its  design  and  operation.  For  example,  it  is  more  important  to  properly  design  the  primary  key  distribution  to  spread  data  uniformly  than  to  create  various  indexes.  There  are  new  concepts  that  you  

©  Bolder  Technology,  Inc.  2011     Page  9  of  11  

must  understand  to  customize  the  MPP  data  warehouse  to  your  unique  situation.  Further,  database  administrators  may  have  to  unlearn  practices  that  are  unnecessary,  such  as  disk/space  management.  

The  large  healthcare  company  mentioned,  “We  had  bumps.  And,  it  took  time  to  get  the  team  up-­‐to-­‐speed.  We  under-­‐estimated  the  time  required.”  As  an  analogy,  DW  migration  is  like  moving  furniture  from  one  house  to  another  –  some  things  will  fit  nicely,  but  some  things  must  be  discarded,  and  other  things  must  be  added  to  exploit  the  new  space.  

Avoid  Doing  Business  and  Technology  Changes  Concurrently  

Any  project  that  makes  major  business  and  technology  changes  at  the  same  time  is  a  risky  project.  But,  incurring  this  risk  is  sometimes  necessary  for  the  business.    Further,  the  project  at  the  large  healthcare  company  with  its  dual  business/technology  changes  was  a  key  catalyst  for  the  paradigm  shift  in  architecting  data  warehouse  environments  for  the  large  healthcare  company.  

Fragile  Support  for  Oracle  Data  Warehouse  

For  the  beverage  company,  one  of  the  reasons  for  moving  from  an  Oracle  platform  was,  “We  had  difficulty  finding  the  right  Oracle  consultant  to  advise  us  about  specific  data  warehousing  issues.  Oracle  has  lots  of  experts  in  lots  of  areas,  but  the  advice  we  got  varied  widely  about  data  warehousing.  On  the  other  hand,  Teradata  is  a  data  warehousing  company  and  could  supply  us  with  the  proper  expertise.”7  

Invest  in  Professional  Services  from  the  Vendor  

The  beverage  company  concluded  that  Teradata  Professional  Services  consultants  assisted  them  through  the  initial  critical  period  by  providing  specific  skills  and  knowledge  about  the  Teradata  platform.  They  felt  that  these  consultants  taught  them  to  fish,  rather  than  fishing  for  them.    

Beware  of  Cursor-­‐Based  Processing  

A  DW  migration  specialist  noted  that  there  is  a  huge  difference  in  database  processing  between  cursor-­‐based  and  set-­‐based  processing.  Performance  can  be  one  hundred  times  faster  with  set-­‐based  processing.  He  gave  an  example  of  converting  Oracle  PL/SQL  code  to  a  Teradata  Stored  Procedure  to  process  a  one  million-­‐row  table  with  various  lookups  to  other  tables.  The  Oracle  processing  was  taking  one  hour  and  four  minutes.  With  a  half  day  of  conversion  effort,  Teradata  reduced  processing  time  from  64  minutes  to  45  seconds!  Although  this  example  is  extreme,  most  conversions  from  cursor-­‐based  to  set-­‐based  improved  execution  times  by  at  least  ten-­‐fold.  

Synthesis  In  summary,  here  are  the  themes  critical  to  a  successful  data  warehouse  migration.  

Performance  as  Motivation  

The  companies  interviewed  cited  performance  problems  as  one  of  the  driving  issues  to  migrate  from  an  older  SMP  data  warehousing  platform  to  a  newer  MPP  platform.  The  performance  problems  stem  from  both  ETL  load  processing  and  from  report  generation.  The  new  platform  (on  Teradata)  was  significantly  better  for  all  the  companies.  There  were  no  negative  comments  about  Teradata  performance.    

Complexity  

Complexity  of  database  administration  was  a  major  factor  for  companies  migrating  from  Oracle.  The  Oracle  database  is  perceived  to  be  difficult  to  manage  and  required  specialized  skills.  Teradata  was  perceived  to  be  simpler  to  manage  as  a  system  and  to  develop  newer,  more  comprehensive  databases.  

Sense  of  Urgency  

There  was  a  sense  of  urgency  in  most  companies  that  this  migration  must  be  accomplished  quickly.  The  problems  had  been  building  over  several  years.  Management  finally  understood  the  gravity  of  their  situation  and  concluded  

©  Bolder  Technology,  Inc.  2011     Page  10  of  11  

that  some  action  had  to  be  taken.  Over  the  years,  data  warehousing  had  matured  as  a  critical  tool  of  business  management,  the  absence  of  which  was  now  recognized  as  a  threat  to  the  company.    

Strategy  for  DW  Migration  

The  effort,  time,  and  complexity  of  a  DW  migration  project  were  under-­‐estimated  by  most  companies.  There  was  an  over-­‐reliance  on  the  simple  ‘fork-­‐lift’  strategy  that  appeared  an  easy  and  quick  solution.  Moving  data  from  one  disk  to  another  is  easy.  Making  sense  of  that  data,  and  integrating  it  for  effective  decision  support,  is  difficult  and  takes  time.  Companies  should  spend  more  effort  in  understanding,  categorizing,  and  defining  the  overall  strategy  for  DW  migration.  Companies  should  enter  into  a  DW  migration  project  with  ‘open  eyes,’  rather  than  being  surprised  by  the  complexity  of  enterprise  data  integration  (which  is  really  at  the  heart  of  a  corporate  DW  migration  strategy).    

Leverage  the  Opportunity  

All  the  companies  interviewed  recognize  that  their  new  data  warehouses  have  substantially  more  capability  than  their  old  ones.  Sadly,  only  a  few  companies  recognize  that  this  was  an  opportunity  to  challenge  key  business  assumptions  and  to  revolutionize  their  business  processes.  In  other  words,  a  DW  migration  project  opens  a  door  for  business  innovation.  For  some  companies,  this  opportunity  was  a  surprising  consequence.  Do  not  be  surprised.  Be  prepared  to  leverage  the  opportunity.  

 

This  paper  presents  the  diverse  issues  that  will  assist  in  a  successful  migration  of  your  data  warehouse.  DW  migration  is  a  major  IT  investment  for  most  corporations,  both  in  cost  reductions  and  in  business  enhancements.  

©  Bolder  Technology,  Inc.  2011     Page  11  of  11  

 

Appreciation  is  given  to  Teradata  Corporation  for  its  sponsorship  of  this  research  into  data  warehouse  migration  strategies  and  to  the  people  who  took  time  to  share  their  experiences  about  these  exciting  developments.  

 

About  Bolder  Technology,  Inc.  Bolder  Technology  is  a  twenty-­‐year-­‐old  consultancy  focused  on  Business  Intelligence  and  Data  Warehousing.  The  founder  and  president  is  Dr.  Richard  Hackathorn,  who  has  more  than  thirty  years  of  experience  in  the  Information  Technology  industry  as  a  well-­‐known  industry  analyst,  technology  innovator,  and  international  educator.  He  has  pioneered  many  innovations  in  database  management,  decision  support,  client-­‐server  computing,  database  connectivity,  associative  link  analysis,  data  warehousing,  and  web  farming.  

Richard  was  a  member  of  Codd  &  Date  Associates  and  Database  Associates,  early  pioneers  in  relational  database  management  systems.  In  1982,  he  founded  MicroDecisionware,  Inc.  (MDI),  an  early  vendor  of  database  connectivity  products,  growing  the  company  to  180  employees.  The  company  was  acquired  by  Sybase,  now  part  of  SAP,  in  1994.  He  is  a  member  of  the  IBM  Gold  Consultants  and  the  Boulder  BI  Brain  Trust.  He  has  written  three  books  and  has  been  a  professor  at  the  Wharton  School  and  the  University  of  Colorado.    

About  the  Sponsor  Teradata  is  the  world's  largest  company  solely  focused  on  data  warehousing  and  integrated  marketing  management  through  database  software,  enterprise  data  warehousing,  data  warehouse  appliances,  and  analytics.    Teradata  provides  the  best  database  for  analytics  with  the  architectural  flexibility  to  address  any  technology  and  business  need  for  companies  of  all  sizes.    Supported  by  active  technology  for  unmatched  performance  and  scalability,  Teradata’s  experienced  professionals  and  analytic  solutions  empower  leaders  and  innovators  to  create  visibility,  cutting  through  the  complexities  of  business  to  make  smarter,  faster  decisions.    Simply  put,  Teradata  solutions  give  companies  the  agility  to  outperform  and  outmaneuver  for  the  competitive  edge.  

References  1     Anonymous  interview  with  first  migration  specialist  on  February  22,  2011.  2    Oracle-­‐to-­‐Teradata  Warehouse  Migration  Program:  Methods  and  Tools,  2011.  3    Anonymous  interview  with  second  migration  specialist  on  March  18,  2011.  4    Anonymous  interview  with  first  migration  specialist  on  February  22,  2011.  5    Anonymous  interview  with  large  beverage  company  on  March  23,  2011.  6    Anonymous  interview  with  migration  tools  developer  on  February  23,  2011.  7    Anonymous  interview  with  large  beverage  company  on  March  23,  2011.  

 

 Teradata  and  the  Teradata  logo  are  registered  trademarks  of  Teradata  Corporation  and/or  its  affiliates  in  the  U.S.  and  worldwide.  

EB-­‐6368   >  0511