Data Migration In An Agile Open Source World
-
Upload
craig-smith -
Category
Technology
-
view
5.427 -
download
1
description
Transcript of Data Migration In An Agile Open Source World
Data Migrationin an Agile Open Source World
Craig SmithSuncorp
Welcome…
Image http://www.neopoleon.com/blog/images/excel/7.jpg
According to Wikipedia:
Data migration is the process of transferring data between ... computer systems ... is usually performed programmatically to achieve an automated migration ... required when organizations ... change ... or upgrade to new systems, or when systems merge...
Image: http://i.ehow.com/images/GlobalPhoto/Articles/2065416/dictionary_Full.jpg
“In” End Point
Extract
Transform
Load
“Out” End Point
ExtractTransform
Load(ETL)
Pattern
(Enterprise)Application Integration
Patterns
File transfer
Shared database
Remote Procedure Invocation
Messaging
Image http://www.eaipatterns.com/eaipatterns.html
Data Migration
+ Application Integration
=Similar
ApproachImage http://kansolutions.net/yahoo_site_admin/assets/images/integration.302161703_std.jpg
Image http://blog.theworkinggroup.ca/wp-content/uploads/2009/04/agile-manifesto1.gif
Agile Manifesto
Image http://blogs.conchango.com/Admin/ImageGallery/blogs.conchango.com/Colin.Bird/Scrum%20Overview%20Diagram.png
Scrum Practices
Image Beck, Kent – Extreme Programming Explained: Embrace Change
XP Practices
Source http://www.assembla.com/wiki/show/burro/Development_Principles
Development PracticesFixing a
broken build is always the
highest priority
100% Code
Coverage
Commit Often
If In Doubt, Spike It
Out (On A Branch)
All Prod Code
Paired (Or Reviewed)
No Code Ownership
Test Code Is
Prod Code
Fast (<3 mins) Builds
Changing The Following Things
Is A Team Decision...
Code Is A Liability
Molecular Over
Atomic Tests
Use It Or Lose
It
Nulls Are Evil
Statics Are Evil
Use Delegation
Over Inheritance
Create Small
Classes And Methods
Use Outlandish
Class Names
Break It Down... IterativelyImage http://iqizone.com/images/paper%20pile.jpg
Image http://static.soxfirst.com/soxfirst.com/imgname--derivatives_time_bomb_for_big_banks---50226711--bomb.jpg
Agile Builds Confidence
Continuous (Inte)(Mi)gration
Image http://i.ehow.com/images/GlobalPhoto/Articles/2289906/P1120995RPM_Full.jpg
Throttle Data
Prioritisation & CategorisationImage http://www.mikkeman.nl/blog/wp-content/uploads/2009/10/priority-300x137.jpg http://ralphlosey.files.wordpress.com/2009/10/playing-cards.jpg
Image http://silverspaceship.com/static/shot_1.png
Volatility Categorisation
Analysis = BDUF NDUF SDUFImage http://www.domainlanguage.com/_library/images_people_working/designing_at_whiteboard.jpg
Image http://silverspaceship.com/static/shot_1.png
Implement “Big Bang”
Implement “Master + Deltas”Image http://sapplanning.files.wordpress.com/2008/07/integrate-technology.jpg
Image http://www.spycomponents.com/images/xml_at_work.gif
Data in non-binary format > diff
Master Database DumpsImage http://images.yogee.com.au/cat-dump-truck-609-2.jpg
Image http://www.life123.com/bm.pix/read-balance-sheet.s600x600.jpg
Activity Migration
Image http://www.life123.com/bm.pix/read-balance-sheet.s600x600.jpg
Reconciliation
Open Source Continuous Integration / Build Tools
Continuous Integration Servers Hudson: https://hudson.dev.java.net
Cruise Control:http://cruisecontrol.sourceforge.net
Build Tools Ant: http://ant.apache.org
Maven: http://maven.apache.org
Gradle: http://www.gradle.org
SCons: http://www.scons.org
Phing: http://phing.info
Rake: http://rake.rubyforge.org/
Open Source Developer ToolsIDE’s (multiple language support) Eclipse: http://www.eclipse.org
NetBeans: http://www.netbeans.org
RadRails: http://www.aptana.com/rails
Database Clients DBVisualizer: http://www.dbvis.com/
SQuirreL: http://squirrel-sql.sourceforge.net
Web Service / SOAP Tools soapUI: http://www.soapui.org
Version Control Systems Subversion: http://subversion.tigris.org
Git: http://git-scm.com
Database Versioning Liquibase: http://www.liquibase.org/
Open Source Testing ToolsTest Specification – xUnit Tools JUnit: http://www.junit.org
PHPUnit: http://www.phpunit.de/
Test Specification Tools – Expressive/BDD FitNesse: http://fitnesse.org
Concordion: http://www.concordion.org
Cucumber: http://cukes.info
easyb: http://www.easyb.org
Test Execution / Record / Log / UI Selenium: http://seleniumhq.org
Watir: http://wtr.rubyforge.org
Performance Test JMeter: http://jakarta.apache.org/jmeter/
Grinder: http://grinder.sourceforge.net/
Open Source ETL/ESB Tools
ETL Donkey:
http://www.assembla.com/wiki/show/burro
Pentaho (Kettle): http://kettle.pentaho.org/
Talend: http://www.talend.com
Clover: http://www.cloveretl.com/
Scriptella: http://scriptella.javaforge.com/
ESB Mule: http://www.mulesoft.org
Apache ServiceMix:http://servicemix.apache.org
Apache Camel: http://camel.apache.org/
Questions ??Craig Smithhttp://www.suncorp.com.auhttp://[email protected]@smithcdau
Suncorp is one of Australia and New Zealand's largest diversified financial services providers, supplying banking,
insurance and wealth management products to around 7 million customers through well-established and recognised brands such as AAMI, Australian Pensioners Insurance Agency, Shannons, Vero, Asteron and Tyndall, as well as Suncorp and GIO. Today,
Suncorp is Australia's fifth largest bank and second largest domestic general insurance group, with over 16,000 staff. Suncorp has representation in 450 offices, branches and
agencies throughout Australia and New Zealand.