CERN IT-DB Services

CERN IT-DB Services: Deployment, Status and Outlook Luca Canali, CERNGaia DB Workshop, Versoix, March 15th, 2011

1OutlineReview of DB Services for Physics at CERN in 2010AvailabilityIncidentsNotable activitiesInfrastructure activities, projects, planned changesOutlook and service evolution in 2011

2CERN IT-DB Services - Luca CanaliCERN and LHC LHC data correspond to about 20 million CDs each year!

Concorde(15 Km)Balloon(30 Km)CD stack with1 year LHC data!(~ 20 Km)Mt. Blanc(4.8 Km)RDBMS play a key role for the analysis of LHC data3CERN European Organization for Nuclear Research located at Swiss/French borderLHC Large Hadron Collider The most powerfull particle accelerater in the world launched in 2008CERN IT-DB Services - Luca CanaliPhysics and DatabasesRelational DBs play today a key role for LHC Physics data processingonline acquisition, offline production, data (re)processing, data distribution, analysis SCADA, conditions, geometry, alignment, calibration, file bookkeeping, file transfers, etc.. Grid Infrastructure and Operation servicesMonitoring, Dashboards, User-role management, ..Data Management Services File catalogues, file transfers and storage management, Metadata and transaction processing for custom tape-based storage system of physics dataAccelerator logging and monitoring systems

4CERN IT-DB Services - Luca CanaliOriginally deployed at CERN for the construction of LEP, relational databases now play a key role in the experiments' production chains, from online acquisition through to offline production, data distribution, reprocessing and analysis

Listed among the critical services for the LHC experiment 4CERN IT-DB Services - Luca Canali5

Databases at CERN5

CERN Databases in NumbersCERN databases servicesGlobal users community of several thousand users~100 Oracle RAC database clusters (2 6 nodes)Currently over 3000 disk spindles providing more than ~3PB raw disk space (NAS and SAN)Some notable DBs at CERNExperiments databases 14 production databasesCurrently between 1 and 12 TB in sizeExpected growth between 1 and 10 TB / yearLHC accelerator logging database (ACCLOG) ~50 TBExpected growth up to 30 TB / year... Several more DBs on the range 1-2 TB

6CERN IT-DB Services - Luca CanaliUpdates on LHCSuccessful re-start of LHC operation in 20102011 run started mid Feb., beam energy of 3.5 TeV Work going on with the acceleration to increase luminosity (and rate of data collection)

7CERN IT-DB Services - Luca Canali

CERN IT-DB Services - Luca CanaliStatus of the DB Servicesfor Physics 8CERN IT-DB Services - Luca CanaliService NumbersInfrastructure for Physics DB Services~115 quadcore machines ~2500 disks on FC infrastructure9 major production RAC databases. In addition:Standby systemsArchive DBsIntegration systems and test systemsSystems for testing streams and 11.2

99CERN IT-DB Services - Luca CanaliServices and Customers Offline DB Service of LHC experiments and WLCGOnline DB ServiceReplication from online to offlineReplication from offline to Tier1sNon-LHCbiggest user in this category is COMPASS and other smaller experiments

10CERN IT-DB Services - Luca CanaliDBA Support24x7 support for online and offline DBsFormalized with a CERN piquet8 DBAs on the piquetTemporary reduced personnel in Q3 and Q4:Note on replication from offline to Tier1s is best effort, no SMS alert (only email alert)on-call DBA checks email 3 times per day

11CERN IT-DB Services - Luca CanaliService AvailabilityFocus on providing stable DB servicesMinimize changes to services and provide smooth running as much as possibleChanges grouped during technical stops4 days of stop every ~5 weeksSecurity patches, reorg of tablesMajor changes pushed to end-of-the-year technical stop (~2 months of stop)

Service availability:Note these are averages across all production servicesOffline Service availability: 99.96%Online Service availability: 99.62%

12CERN IT-DB Services - Luca CanaliNotable incidents in 2010 1/2Non-rollingness of April PatchSecurity and recommended patch bundle for April 2010 (aka PSU 10.2.0.4.4)Contains patches marked as rollingPassed tests and integrationTwo issues show up when applied in productionNon rolling on clusters of 3 or more nodes with loadOn DBs with cool workload Symptoms: after ora-7445 and spikes of load appearOra-7445Reproduced on test and patch available from OracleThanks to persistency team for helpNon-rollingness Reproduced at CERNRelated to ASM

1313CERN IT-DB Services - Luca CanaliNotable incidents in 2010 2/2Two issues of unscheduled power cut at LHCB online pit~5 hours first occurrence (9/8)~2 hours for second occurrence (22/8) In first incident DB became corruptedStorage corruptionLost write caused by missing BBUs on storage after previous maintenanceRestore attempted from compressed backup, too time consumingFinally switchover to standby performedSee also further comments on testing standby switchover in this presentationAnother instance of corrupted DB after power cut18-12-2010, archive DB for Atlas corruptedRecovery from tape: about 2 days

14CERN IT-DB Services - Luca CanaliNotable recurring issuesStreamsSeveral incidentsDifferent parts of replication affectedOften blocks generated by users workload and operationsHigh loads and node rebootsSporadic but recurrent issuesInstabilities caused by loadRun-away queriesLarge memory consumption makes machine swap and become unresponsiveExecution plan instabilities make for sudden spikes of loadOverall application-related. Addressed by DBAs together with developers

15CERN IT-DB Services - Luca CanaliActivities and Projects in 2010

16CERN IT-DB Services - Luca CanaliService Evolution Replaced ~40% of HWNew machines are dual quadcores (Nehalem-EP)Old generation was based on single-core PentiumsNew storage arrays use 2TB SATA disksReplaced disks of 250GB New HW used for standby and integration DBsNew HW (RAC8+RAC9): 44 servers and 71 storage arrays (12 bay)Old HW (RAC3+RAC4): 60 servers and 60 storage arrays (8 bay)

17CERN IT-DB Services - Luca CanaliConsolidation of Standby DBsNew HW installed for standby DBsQuadcore servers and high-capacity disksThis has increased resources on standby DBsProvided good compromise cost/performance in case of switchover operation (i.e. standby becomes primary)Installed in Safehost (outside CERN campus)Reduce risk in case of disaster recoveryUsed for stand by DBs when primary in CERN IT

Primary DBStandby DBRedo Stream18CERN IT-DB Services - Luca CanaliOracle EvolutionEvaluation of 11.2 features. Notably:Evaluation of Oracle replication evolution:Streams 11g, Goldengate, Active Dataguard Evolution of clusterware and RACEvolution of storageASM, ACFS, direct NFSSQL plan management for plan stabilityAdvanced compressionWork in collaboration with Oracle (Openlab)

19CERN IT-DB Services - Luca Canali10.2.0.5 Upgrade - EvaluationEvaluation of possible upgrade scenarios11.2.0.2, vs 10,2.0.5, vs staying 10.2.0.411g has several new featuresAlthough extensive testing is needed11.2.0.2 patch set came out in September and with several changes from 11.2.0.110.2.0.4 will go out of patch support in April 201110.2.0.5 supported till 201310.2.0.x requires extended support contract from end July 2011Decision taken in Q3 2010 to upgrade to 10.2.0.5 (following successful validation)

20CERN IT-DB Services - Luca Canali10.2.0.5 Upgrade - ReviewTesting activitySeveral key applications testedNo major issues foundVery difficult to organize a real world testsUpgrade of production during January 2011Technical stop for the experimentsMostly a smooth changeSome minor issues found only when switching to productionA few workaround and patches add to be added

21

CERN IT-DB Services - Luca CanaliActivities on BackupBackups to tape using 10gbps have been successfully testedSpeed up to 250 MBPS per RMAN channelFirst phase of production implementationDestination TSM at 10gbpsSource multiple RAC nodes at 1gbps Typically 3 nodesIn progress (~30% of DBs by Q1 2011)Other activitiesMoving backup management to a unified tool inside the groupUnified tool for routine test of DB recoveries from tape

22

CERN IT-DB Services - Luca CanaliActivities on MonitoringImprovements to custom streams monitoringAdded Tier1 weekly reportsMaintenance and improvements to streammonDML activity per schema, PGA memory usage OEM 11gCurrently deployed at CERNSeveral issues needed troubleshootingNotably a memory leak triggered by browser Internal activities on monitoringWe are unifying monitoring infrastructure across DB group

23

CERN IT-DB Services - Luca CanaliActivities on Data LifecycleGoal: avoid that DB growth impact manageability and performanceActivity launched in 2008Partitioning and data movement main toolsCompression used tooIn 2010 more applications modified to allow partitioningData start to be moved to archive DBsJoint work DB group and experiments/development

24

CERN IT-DB Services - Luca CanaliActivities on SecurityInternal application developedTo track access to DBsMining audit dataAllows to spot unusual access patternsCan be source of info for defining white listsFirewall active on DB serversFurther discussion on activating white lists going on

25CERN IT-DB Services - Luca CanaliActivities for Online SetupsALICE, LHCB and CMS onlineInstallations of the DBs at the experiments pitsHW is managed by experimentsHW warranty expiring Replacement under wayIT discussed with experiments on HW replacementGoal of having similar HW at the pit as in IT to reduce maintenance effort and complexityDeployment of new HW expected in Q1 2011

26CERN IT-DB Services - Luca CanaliStandby TestsStandby DBs and switchover operationTested and documentedIdeally a test switch should be performed on all production DBsActivity needed to validate the disaster recovery infrastructureDuring technical stop in Q1 2011Scheduled test of Atlas online standby DBDowntime ~.5h to switch to standby and ~.5h to switch back

27CERN IT-DB Services - Luca CanaliOutlook and Activities for 201128CERN IT-DB Services - Luca CanaliMajor Scheduled ChangesUpgrade to 11gR2Upgrade of Oracle to 11gR2 in Q1 2012 (11.2.0.3?)Replacement of 2/3 of production HWNew servers and storageCombined changeSwing upgrade: upgrade of standby built on new HWProduction DBs and constraints from experimentsMaintenance window limited to extended technical stop for many DBs, i.e. Q1 2012

29CERN IT-DB Services - Luca CanaliSoftware changes preparationTesting of existing applications on 11gR2In collaboration with the experimentsOur experience is that some issue are only seen under loadLoad-based testing necessaryInvestigating Oracle RAT?Other software changesInvestigating RHEL6rpm-based installation (integrated with CERN OS installation, i.e. quattor)Overall goal of unifying procedures across IT-DB group

30CERN IT-DB Services - Luca Canali11gR2 FeaturesWe will further testing and prepare to deploy new 11gR2 features of interestVery interested inActive Data GuardOther notable features of interest for us:Improvements to streamsImprovements to ASM and NFSImprovements to clusterwareSQL plan managementInterval partitioning

31CERN IT-DB Services - Luca CanaliActivities on ArchitectureNew HW acquisitionIn 2011 a large group of production machines goes out of warrantyHW renewal and occasion to profit from more recent HW for performance and capacitySeveral technologies have been evaluatedUse of SSD for caching10gbps Ethernet for interconnect8gbps Fiber Channel storage accessNAS at 10gbps with flash cacheEvaluation of Oracle VM

32

RAC DB 2

Current Model33RAC DB 1RAC DB 2FC SwitchesDual-socket quad-core DELL blade servers, 24GB memory, Intel Xeon Nehalem; 2.27GHzDual power supplies, mirrored local disks, redundant 1GigE, dual HBAs, RAID 1+0 like with ASM and JBOD

CERN IT-DB Services - Luca Canali

RAC DB 1Ethernet SwichesNAS and NetApp at CERNEvolution of technology is very interestingPerformance and capacity10 GigE connectivitySSD cache to boost IOPS (PAM modules)Allow large DBs with 2TB SATA disksMaintenance and reliabilityFilesystem Snapshots to be used as backup against logical corruptionMature OS and filesystem for stabilityRedundant controllers for transparent rolling maintenanceSupport from a major storage vendorExperience at CERN

34CERN IT-DB Services - Luca CanaliSolid State StorageA revolution in storageMany physics applications spend significant time on random IO readSSD for large increase in IOPS and reduced latencyAreas of interest Flash-based cache in NAS controllersDB Flash cache feature in 11gR2To be further investigatedEntire DB on SSD?For the moment on hold because of cost for multi-TB DBsArea in evolutionat present a multi-TB DB on SSD is very expensive

35CERN IT-DB Services - Luca CanaliServer SpecsNew HW acquisition allows us to profit of technology evolutionLatest CPUsAlthough probably 4-cores still best choice for licensing reasonsMore RAM Enlarge Oracle cache to reduce random IOFor example servers with 48 GB of RAMFaster interconnect:10GigEFor storage accessBackup to tape

36CERN IT-DB Services - Luca CanaliDatabase technologyFuture needs regarding DB servicesReview for medium to long term in collaboration with physics experiments and users community at CERN NoSQL DBsPreliminary talks and interest from the experiments

37CERN IT-DB Services - Luca CanaliConclusionsFocus on stability of DB services in 2010Following several years of preparationInfrastructure activities on improving backups, archive, application testing, HW testingUpgrade to 10.2.0.5 performed before 2011 runContinuity of DB operations in 2011Priority on running smooth services during data acquisitionPreparation for 11gR2 upgradeNew HW acquisition, evolution of storage and serversInvestigations of new technologies HW and SW

38CERN IT-DB Services - Luca CanaliAcknowledgments CERN IT-DB group and in particular:Zbigniew Baranowski, Marcin Blaszczyk, Eva Dafonte, Kate Dziedziniewicz, Przemyslaw Radowiecki, Jacek Wojcieszuk, Dawid Wojcik.

More info:http://cern.ch/it-dep/dbhttp://cern.ch/canali

39CERN IT-DB Services - Luca Canali

39