Post on 31-Jan-2018
1
2
How CME Group Used Informatica to Merge Three Major Exchanges While
Reducing Support and Processing Times
Hadi Abdul @Hadi_Sid
Lead Systems Analyst
Amanda Morris @ProManda
Lead Software Engineer
CME Group @CMEGroup
3
Agenda
• About CME Group
• Business Challenge
• Solution Case Studies
• Benefits and Use of Informatica
• Future Roadmap
• Lesson Learned
• Q & A
4
• The world's leading and most diverse derivatives marketplace
• Result of mergers between three major exchanges
• 2007 merger of Chicago Mercantile Exchange and Chicago Board of Trade
• 2008 acquisition of New York Mercantile Exchange
• Headquarters in Chicago – offices around the world
• Partner Exchanges – DME, Korean, and Green
5
CME Group Trading Floor
Also…Electronic Trading and
Privately Negotiated Trades
6
CME Group Mergers and Acquisitions
2007
2008
2010
2011
Chicago Mercantile Exch merges with Chicago Board of Trade
New York Mercantile Exch and Commodity Exch Acquisition
Dow Jones Acquisition
Partner Exchanges – Dubai and Korea
• Combined organization became known as CME Group
• Integration completed in 2008
• Now four exchanges operating under CME Group
• Integration completed in 2009
• 90% investment in Dow Jones Indexes
• Including the Dow Jones Industrial Average
• Continuing to seek global opportunities
7
CME Group Futures & Options Products
• Crude Oil (WTI) , Natural Gas, Heating Oil Energy
• Gold, Silver, Copper Metals
• Corn, Wheat, Soybeans, Cattle Agriculture
• Eurodollar, Treasury Notes and Bonds Interest Rate
• S&P 500, Dow Jones Industrial Average Equity Indexes
• G10 and Emerging Market Currency Pairs FX
• Interest Rate Swaps, Credit Default Swaps OTC
• Home Price Index Futures Real Estate
• Hurricane, Snowfall, Rainfall Weather
8
Business Challenge
• Data from 3 different exchanges needed to be in
a single place in a single format
• Complicated by complex code base and
divergent systems
• Aging Technologies needs to be replaced
• Mainframe decommission deadline
• Volume increased exponentially, SLAs became
harder and harder to meet
• Number of trades doubled from 1 to 2 Billion over 4 years
9
CME Group Data Processing Facts
• Data in many formats • Oracle, SQL Server, DB2, flat files, XML, FIXML,
Spreadsheets
• Over 300 distinct production Oracle Schemas
• Large data sets • 15 TB data warehouse
• Globex Audit Data tops at about 500+ million records a day
• Millions of trades and quotes a day, copies everywhere
• Diverse technology • Java, .NET, shell scripts, perl, PL/SQL, COBOL, DB2,
other ETL tools
10
Pre-Integration Data Flow
DB2 Chicago
Mercantile
Exchange
Chicago Board
Of Trade
New York
Mercantile
Exchange
Oracle
Mainframe ETL
Flat Files
Java
Oracle ETL
Oracle/Excel
Streetbook/
Bridgeback
11
Post-Integration Conceptual View
Market
Public Data
Pre Clearing
Trade Data
Post Clearing
Trade Data
Reference
Data
ERP
Other Data
Extract Control
Source Data
Cleansing
Data Integrity
Reference Data
Management
Business Rules
Error Handling
Market
Data Liquidity
Volume Order
Quantity
Order
Execution
Trade
Revenue
Market
Maker Regulatory
Other
Internal Reports
Ad Hoc Requests
External Reports
Dashboards
Business Objects
Source Data ETL Architecture Data Warehouse Applications
Client Applications
12
EFFICIENCIES GAINED IN POST TRADE REPORTING
Case Study 1
13
Stats
Data Warehouse
Report Engine
CME Group and Partner Exchanges
Reports
Prices
Other Customers
Product Volume/OI Settlements
Data Cloud
Forward Contracts
Post Trade Reporting Informatica Utilization
14
Users and Reports
• Users
• Over 350+ internal users
• Reports, Dash Boards
• 3000+ hits per day for trade data reports
• Traders, brokers, investors, and universities.
• Reports
• Business Objects, Qlikview, and Crystal Reports
• Over 300+ reports for external customers
• Over 1000+ internal reports
15
CME Group - Published Data
Crude Oil and Gold Futures Contracts
16
Efficiencies Gained
• Mainframe ETL Processing Time Lines
• ETL Processes Completed around 11 PM
• Reports were being generated at 1:30 AM
• Informatica ETL Processing Time Lines
• ETL processing completes at 8:30 PM
• Reports Generation completes at 9:30 PM
• Trade Data Load
• Mainframe ETL: 55 Minutes
• Informatica ETL: 22 Minutes
• Gain of 60%
17
Batch Processing Efficiencies Gained
• Less jobs running due to simplified processes
• SLAs consistently met
• Improved Performance
• Reuse versus customization
• Ease of Maintenance
• Much Shorter Learning Curve
18
Informatica Processing Benefits
• Ease of Troubleshooting
• Java error vs. Informatica error
• In the middle of the night, which would you rather dig through?
• Standardized error handling
• Programmers often “forget” to include error handling or miss possible unexpected exceptions
• Standardized monitoring
• All processes monitored side by side in standard format
19
Informatica Project Development Benefits
• Safely retire legacy applications
• Resources can easily retrain to Informatica
• Keep people with valuable business knowledge
• Even as systems are decommissioned
• Share and reuse of code for faster delivery
• Faster development cycle
• Minimize business outages
20
SIMPLIFYING REGULATORY BATCH PROCESSING
Case Study 2
21
Past: Regulatory Trade Processing
Flat Files
DB2
Oracle
PL/SQL
SQL
Loader
Mainframe
Flat File
Interface
Java
“Splitter”
COBOL
Flat Files
External
Tables
Java
Processing
Informatica
Distributed
22
Complex Code
• Overly complex code used to move data
repeatedly
• Mainframe decommissioning – opportunity for simplifying?
• Java programs that were essentially ETL
• PL/SQL packages, perl scripts, shell scripts, etc.
• Lots of duplicate, obsolete, and unnecessary
coding
• Hard to support
23
Some people see a problem and think “I
know, I’ll use Java!”
Now they have a ProblemFactory.
@Lonnen Chris Lonnen Twitter
24
Informatica to Simplify
• Take a hard look at your code base
• Do you really need a custom app and framework for what is essentially an ETL process?
• Push back from “coders”
• Gut reaction is to code and go with what they know
• Need a strong champion of Informatica
• Proving that it CAN be done in Informatica
• “Why not?” vs “Why?”
25
Simplified: Regulatory Trade Processing
Oracle
Regulatory Data
Warehouse
Compliance Distributed
Systems
Oracle Informatica
Informatica
Informatica
Java
Processing PL/SQL
Staging Load Processing and
Final Table Load
26
Simplified Process
• 80% + of data movement and processing
performed via Informatica
• Every new process starts with, “Can we do this in Informatica?”
• Still using PL/SQL and Java
• Compelling arguments for usage
• Iterative functionality
• Complex business logic
• Cursor-driven logic required
• Multi-threading
27
Future Roadmap
• Upgrade to PowerCenter 9.1
• Upgrade to Exadata
• Add Informatica Source Control
• Enforce development standards
• Create in-house user group and share resources
and knowledge
• Continue to look for opportunities to use
Informatica to simplify processes
28
Exadata Performance Improvement
Records
Baseline
Time
Oracle
Exadata
Time
Savings /
N(X) Faster
60 GB
Flat File
Load 365 Million 2:00:00 0:14:00 1:49:30
Staging
Load 9 Million 0:16:00 0:15:00 1:00
Query 1 103,147 0:13:55 0:00:27 40
Query 2 2,605 0:18:15 0:00:06 183
29
Uses of Informatica at CME Group
• Recurrent reports
• Data loads
• Data updates
• Automated testing (QA)
• Ad hoc extracts
• Historical data loads from legacy systems
• Archiving data off database to files
30
Lesson Learned
• No SQL overrides
• Hides code and table usage
• Lose benefits of Informatica
• Define Standards Upfront
• We defined, but didn’t enforce – enforce!
• Code Reviews
• To enforce those standards!
• Often overlooked because it isn’t “real” code
31
Thank You