Moving a 100TB Application from Oracle to Microsoft SQL Server
description
Transcript of Moving a 100TB Application from Oracle to Microsoft SQL Server
![Page 1: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/1.jpg)
Moving a 100TB Application from Oracle to Microsoft SQL Server
Praveen SrivatsaDirector, Asthrasoft ConsultingMicrosoft Regional Director and MVP
DBI311
![Page 2: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/2.jpg)
Agenda
NMS (Large Load) Case Study Comparison between Oracle and SQL implementationScaling Points and Optimizations
Async operations with Service BrokerDistributing data with Table partitionsData CompressionOther optimization options
![Page 3: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/3.jpg)
Case StudyExisting NMS application
The NMS application collects, logs and analyzes the logs of various digital equipment in the networkUsed for mobile towers, wi-max networks, wifi-networks and data centersIt’s agents collects logs 24x7 from all equipment and pumps the same to a central store.The logs are analyzed for security breaches or attacks as well as to monitor the health of the various equipment.
![Page 4: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/4.jpg)
Existing NMS application
LogManager
1. Collect the data
2. Analyze the data
3. Log and Report
Agent
Agent
![Page 5: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/5.jpg)
Existing NMS application
LogManager
Agent
Agent
0.5 KB/record
2500 records/ agent
10 agents
= 25K r/s(12. 5 MB/s)
25K r/s == 12.5 MB/s
1.5M r/min == 750 MB/min
90M r/hr == 45 GB/hr
2.16B r/day == 1.08 TB/day
Reporting on 100 days of data
![Page 6: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/6.jpg)
Case StudyCurrent architecture with Oracle
RAW TABLES
Reporting Dashboard1. Agents collect data
and write to CSV
2. Oracle BULK Loads data
3. Data is indexed and analyzed
4. Materialized Views are created for reporting
![Page 7: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/7.jpg)
Case StudyServer Configurations
App Server2 Quad Core (8 Cores)32 GB RAM1TB HDDRedhat LINUXPython based App
DB Server2 Quad Core (8 Cores)32 GB RAM1 TB HDDSolaris 10Oracle R2 10G
200 TB External storage on RAID 5 (120 TB usable)
![Page 8: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/8.jpg)
Case StudyKey Challenges
Bulk Data Import is very fast, but the data is not ready for analysis or reporting till indexes/materialized views are builtWith the data inserts going on, the indexing and analysis is really slow – so differed to end of the day when a new table is created (EOD : 13 hrs)Data querying for reporting now has to be done across multiple tables (upto 100 tables with a UNION) adding a lot of overheadLAG from data collection to reporting was anywhere from 13 hrs to 37 hrs
![Page 9: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/9.jpg)
Case StudyMoving to SQL Server
SERVICE BROKER QUEUES
Compressed Tables
Partitioned Tables
WCF
Reporting Dashboard
![Page 10: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/10.jpg)
Case StudyMoving to SQL Server
Collection with data co-relation33 records/sec(2.85 M/day)
WCF
Reporting Dashboard
![Page 11: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/11.jpg)
Case StudyMoving to SQL Server
SERVICE BROKER QUEUES
Collection with differed co-relationBatch of 100 recs25 batches/sec/Q (2500/s)X 10 Queues (25000/s)
2.16 B/dayWCF
Reporting Dashboard
![Page 12: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/12.jpg)
Case StudyMoving to SQL Server
SERVICE BROKER QUEUES
Partitioned Tables
Parallelized data co-relation5 batches (100r)/sec = 500r/sX 5 Readers = 2500 r/sec/QX 10 Q = 25000 r/s
2.16 B/day
WCF
Reporting Dashboard
![Page 13: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/13.jpg)
Case StudyMoving to SQL Server
SERVICE BROKER QUEUES
Compressed Tables
Partitioned TablesRaw data saving due
to compression28%
WCF
Reporting Dashboard
![Page 14: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/14.jpg)
Case StudyServer Configurations
App Server2 Quad Core (8 Cores)32 GB RAM500GB HDDWindows Server 200R2WCF Based CollectorPython App for reporting
DB Server2 Quad Core (8 Cores)64 GB RAM500GB HDDWindows Server 2008R2SQL Server 2008R2
200 TB External storage on RAID 5 (120 TB usable) on Windows Storage Server
![Page 15: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/15.jpg)
Case StudyKey Benefits with SQL Server
Data transfer was continuous and could be parallelized using more queuesQuerying and analysis did not slow down the data collectionMore complex analysis and reporting could be added onto the logsLag between collection and reporting came down to 1hr 12 mins
![Page 16: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/16.jpg)
Case StudyAdditional Benefits with SQL Server
Historical data was compressed, reporting data was uncompressed, but partitionedDB Snapshots every hour provided a shorter window of analysis and reportingCLR Procedure provided advanced rules operations on the data packets
![Page 17: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/17.jpg)
demo
NMS Walkthrough
End to End walkthroughof the simulated app
![Page 18: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/18.jpg)
Scale PointsService Broker Advantages
Is a queued operations like MSMQ or MQSeriesScales for rapid insertsCan create multiple parallel queuesCan use activation to process the queueGives a guaranteed onetime only deliveryCan transmit data across DB instances
![Page 19: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/19.jpg)
OptimizingService Broker
Configure multiple packets per conversationBulk load multiple records into a single messageTransform and bulk insert from SSB XML to tableOptimize packet size of SSB Queue
![Page 20: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/20.jpg)
demo
Service Broker Demo
Optimizing SSB performance
![Page 21: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/21.jpg)
Scale PointsTable Partitions and Compressions
PartitionSeparates logical units of data without changing codeAllows fast archiving with the SWITCH Partition operationAllows partition level management rather than at table
CompressionCompressed 20-35% dataLoading data, backups, traversal over network all benefittedPage-Row level configurations possible
![Page 22: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/22.jpg)
OptimizingTable Partitions and Compressions
PartitionCreate a logical partition switch to parallelize insertsAlign partitions to number of disks and procsAlign indexes to the table partitions
CompressionEnable compression at a page levelPlan table row size in alignment with the page size
![Page 23: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/23.jpg)
demo
Table Partition Demo
Inserting into multiple partitions in parallel
![Page 24: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/24.jpg)
Scale PointsOther tweaks
DB SnapshotsGives a point-in-time read-only view of the data
XML operationsFast and easy XML Bulk operations on SSB Queue data
CLR ProceduresCompiled procedures for co-relation and alerts
![Page 25: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/25.jpg)
Summary
Key benefits of moving to SQL Server from the current Oracle architecture included
A shorter lag for critical reporting (from 13-37hrs to about 75 min)
A scale-out model allowing for even greater scalability
An integrated set of technologies (relational database, queuing technologies, compression options, reporting solutions) to work with
![Page 26: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/26.jpg)
Takeaways
SQL Server leverages a scale out modelIt does this by using parallel operations at multiple levels
SQL Service Broker provides a robust queuing mechanismAsync queues provides a horizontal scale operations even across multiple databases
Table Partitions allows tables to scale to billions of recordsEffective segregation allows single tables to handle large volumes of data effectively
Leveraging SQL Server featuresDB Snapshots provides a point in view of the dataCompression allows us optimize our data storage
![Page 27: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/27.jpg)
Resources
Architecting Service Broker Applicationshttp://technet.microsoft.com/en-us/library/aa964144.aspx
Partitioning Tables and Indexeshttp://msdn.microsoft.com/en-us/library/ms345146
My website, blog and EMailweb : http://www.asthrasoft.comblog : http://studidesk.asthrasoft.com/srivatsapraveenemail : [email protected]
![Page 28: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/28.jpg)
Related Content
DBI307 | Automating database migration to Microsoft SQL Server
DBI315 | Microsoft SQL Server in virtualization and private cloud
DBI330 | Can your BI Solutions scale?
![Page 29: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/29.jpg)
Database Platform (DAT) Resources
Try the new SQL Server Mission Critical BareMetal Hand’s on-Labs
Visit the updated website for SQL Server® Code Name “Denali” on www.microsoft.com/sqlserver and sign to be notified when the next CTP is availableFollow the @SQLServer Twitter account to watch for updates
Visit the SQL Server Product Demo Stations in the DBI Track section of the Expo/TLC Hall. Bring your questions, ideas and conversations!
• Microsoft® SQL Server® Security & Management • Microsoft® SQL Server® Optimization and Scalability• Microsoft® SQL Server® Programmability • Microsoft® SQL Server® Data Warehousing• Microsoft® SQL Server® Mission Critical • Microsoft® SQL Server® Data Integration
![Page 30: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/30.jpg)
Resources
www.microsoft.com/teched
Sessions On-Demand & Community Microsoft Certification & Training Resources
Resources for IT Professionals Resources for Developers
www.microsoft.com/learning
http://microsoft.com/technet http://microsoft.com/msdn
Learning
http://northamerica.msteched.com
Connect. Share. Discuss.
![Page 31: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/31.jpg)
Complete an evaluation on CommNet and enter to win!
![Page 32: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/32.jpg)
Scan the Tag to evaluate this session now on myTech•Ed Mobile
![Page 33: Moving a 100TB Application from Oracle to Microsoft SQL Server](https://reader035.fdocuments.us/reader035/viewer/2022062501/5681664b550346895dd9c441/html5/thumbnails/33.jpg)