How Cisco IT Migrated from HP Computing...
-
Upload
hoangduong -
Category
Documents
-
view
219 -
download
3
Transcript of How Cisco IT Migrated from HP Computing...
© 2012 Cisco and/or its affiliates. All rights reserved.Presentation_ID Cisco Public
How Cisco IT Migrated from HP
Superdome to the Cisco Unified
Computing System
Jag Kahlon, IT Architect, Cisco IT
Sean McKeown, Technology Solutions Architect, Cisco DC Sales
2
© 2012 Cisco and/or its affiliates. All rights reserved.Presentation_ID Cisco Public 4
42?a)The answer to the Ultimate Question of Life, the
Universe, and Everything
b)The ASCII value of the asterisk (*) character
c) The size in TB of Cisco IT’s E-Business Suite mission-
critical database
d)The value in billions $USD of business supported by the
above database
© 2012 Cisco and/or its affiliates. All rights reserved.Presentation_ID Cisco Public
Jag Kahlon Technology Solutions Architect for Cisco Data
Center sales
Former IT Director, Data Center Operations @Oracle
15 years of IT experience, 8 in management for Oracle DC operations
Have worn many operational hats:
Server Management
Storage Management
Backup and Recovery
Change Management
Major Incident and Problem Management
Living & Education: Reside in Colorado with wife and two children, originally from Washington, DC; MS in astrophysics from Northwestern University, BS in physics & theology from Georgetown University
• Architect for Cisco IT
• 18 years of IT experience
• Experience in architecture and design on large cross functional projects
• Currently providing technical leadership on the HP Superdome to UCS transition within Cisco IT
• Living & Education: Reside in Fremont, CA with wife and two children. Have worked on both coasts for various large organizations within IT. Originally from India, BS in Electrical Engineering from Punjab University
Sean McKeown
© 2012 Cisco and/or its affiliates. All rights reserved.Presentation_ID Cisco Public
Topics
Cisco IT overview
HPUX PA-RISC and Itanium lights out
Cisco Unified Computing System
Methodology: planning, strategy and execution
Case studies and lessons learned
© 2012 Cisco and/or its affiliates. All rights reserved.Presentation_ID Cisco Public
Cisco IT Overview
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
locations incountries
offices
data centers and server rooms
employees
of data center space
of UPS power to raised floors
servers virtualized in new DCs, overall
Virtualization goal =
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
* Count covers all lifecycles and all stages of blade commissioning. Data as of October 27, 2011
Allen, TX (200)
Green Park,
UK (40) Amsterdam
(275)
Bangalore
(447)
Carrolton, TX (296)
Irvine, CA (72) Lawrenceville, GA (35)
Mountain View, CA (64)
Richardson, TX (2230)
RTP, NC (424)San Jose, CA (1486)
< 100 UCS Blades
200-500 UCS Blades
> 1400 UCS Blades
6400+ Total Cisco UCS Blades Deployed Globally*
© 2012 Cisco and/or its affiliates. All rights reserved.Presentation_ID Cisco Public
HPUX Lights Out
© 2012 Cisco and/or its affiliates. All rights reserved.Presentation_ID Cisco Public
Cisco HPUX Lights Out
Before
• Approximately 200 HP Superdome and mid-range HP-UX servers in the U.S. and Europe
• Hosting Oracle Real Application Clusters (RAC) and Oracle eBusiness Suite.
• Of 29 production Superdomes, 17 were PA-RISC and 12 Itanium
After
• Goal: None
• Current status: only a handful of legacy environments awaiting decommission
Objective: Remove proprietary platforms running HP-UX from the enterprise and migrate applications to UCS
© 2012 Cisco and/or its affiliates. All rights reserved.Presentation_ID Cisco Public
Cost Avoidance
Cisco IT saved 85% in CAPEX with the Quote to Cash migration
HPUX Roadmap
PA-RISC EOL’d
Itantiumdevelopment uncertain
Software support dwindling
Standardization
Single platform: x86 on UCS
Two OS’s: Linux and Windows
Consolidation
Fewer, modern data centers
HP to UCS Migration – Business drivers
© 2012 Cisco and/or its affiliates. All rights reserved.Presentation_ID Cisco Public
HP to UCS Migration – Technology drivers
Manageability
Management and support increasingly difficult
Cisco IT can rapidly provision a replacement by applying a Cisco UCS Manager service profile to a different blade
Scalability
Cisco IT was hitting their Superdomes’ limits for vertical scalability for key applications
Horizontal scale-out deemed too costly and painful
Environmental
Cisco UCS occupied one-sixth the data center space of the HP systems that it replaced, consumed 65 percent less power, and used 40 percent fewer cables
UCS
Enterprise level performance and scalability
Industry standard x86 + Linux architecture
© 2012 Cisco and/or its affiliates. All rights reserved.Presentation_ID Cisco Public
Cisco Unified Computing System
© 2012 Cisco and/or its affiliates. All rights reserved.Presentation_ID Cisco Public
Cisco Unified Computing System
A Single System That Unifies Compute: Industry standard x86
Network: Wire once for SAN, NAS, iSCSI
Virtualization: Control, scale, performance
Low latency 10GbE network ideal for RAC
Embedded Management Service profiles enable swappable compute
Dynamic resource provisioning
Ability to integrate with broad partner ecosystem
Energy and Cost Efficient Fewer servers, switches, adapters, cables
Lower power and cooling requirements
© 2012 Cisco and/or its affiliates. All rights reserved.Presentation_ID Cisco Public
Cisco Unified Computing SystemInnovation unites computing, networking, virtualization, and storage access
Cisco UCS Traditional Blade Server
vs.
Traditional
Blade Server
Offerings
© 2012 Cisco and/or its affiliates. All rights reserved.Presentation_ID Cisco Public
Installing Typical Servers
LAN
SAN
•RAID settings
•Disk scrub actions
•Number of vHBAs
•HBA WWN assignments
•FC Boot Parameters
•HBA firmware
•FC Fabric assignments for HBAs
•QoS settings
•Border port assignment per vNIC
•NIC Transmit/Receive Rate
Limiting
•VLAN assignments for NICs
•VLAN tagging config for NICs
•Number of vNICs
•PXE settings
•NIC firmware
•Advanced feature settings
•Remote KVM IP settings
•Call Home behavior
•Remote KVM firmware
•Server UUID
•Serial over LAN settings
•Boot order
•IPMI settings
•BIOS scrub actions
•BIOS firmware
•BIOS Settings
© 2012 Cisco and/or its affiliates. All rights reserved.Presentation_ID Cisco Public
Simplify the Data Center: UCS Service Profiles
LAN
SAN
Serv
ice
Pro
file
© 2012 Cisco and/or its affiliates. All rights reserved.Presentation_ID Cisco Public
Service Profiles Hardware State Abstraction – Oracle Examples
RAC node 2
HTTP Server ERP
RAC node 1 RAC node 4RAC node 3
ERP
Benefits
• Enables consistent setup, reduces service calls and downtime
• Add new instance in minutes, not hours or days
• Immediately test if workload is CPU or memory constrained
• Automatically load profile when new server added to system
• No LAN, SAN Zoning or any reconfiguration
© 2012 Cisco and/or its affiliates. All rights reserved.Presentation_ID Cisco Public
Blades and Rack-Mount Servers Managed by
UCS Manager
© 2012 Cisco and/or its affiliates. All rights reserved.Presentation_ID Cisco Public
UCS Solution Focus
Compute
Network
Virtualization
Storage
Operating Systems
Applications
Information
Cisco UCS B-Series
Cisco UCS Manager
Cisco Nexus® Family
Switches
NetApp FAS
10 GE and FCoE
Complete Bundle
VBLOCK FLEXPOD Standard Configurations
Infrastructure
OS/ Hypervisor
VDI Databases Enterprise Apps Business Analytics & DW RISC Migration
Unified Computing
HANA &
BWA
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
Migrating Oracle from HPUX to UCS:
Methodology and Successes
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
Dual Core 90nm
Quad Core 65nm
2005 2005 2006 2007 2008 2010 2011
Eight Core 45nm
Single Core 90nm
Ten Core 32nm
Xeon 7040 Six Core
45nm
Xeon 7500
Xeon 7400
Xeon 3.66GHz
Dual Core 65nm
Xeon E7-4800
Xeon 7300Xeon 7100
Source: Intel internal OLTP database workload performance estimates as of 15 April 2011. Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.
2000% Performance Increase in CPU power since 2005
Cost
UCS servers are significantly cheaper
Less maintenance and support cost
Reliability
Clustering and virtualization provide better reliability
Vertically scaled nodes increase chances of failure
Independence
X86 platform and Linux are not tied to a single vendor
Performance
x86 chipsets have surpassed proprietary performance
levels, especially for database work
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
Large ERP Environments
Legacy systems reaching technology end of life and scalability limits
Increasing maintenance and support costs for older platforms
Proprietary (e.g. HP-UX, VFS) vs. “open” (Linux)
Data Center migration: Technology/architecture “rethink” opportunity
Oracle de-support for Itanium
Cisco UCS provides better performance at considerably lower cost
Large ERP DB Size Migration Date
Cisco Customer Care Center Service Sales 40 TB Live March 4th (Q3FY12)
The Configuration & Ordering Experience 3 TB Done - Q1FY12
E-Customer – Customer Master 8 TB Done - Q3FY11
Quote and order entry (QTC) 6 TB/4 TB Done – Q3FY11
Goal to Commissions 4 TB Done – Q1FY11
Plan to Build ,Manufacturing & Logistics. 1 TB Done = Q1FY11
Business Intelligence systems 28 TB/ 3 TB Done – Q2FY12
Partner Ordering Tool 2.5 TB Q4FY12
Note: Cisco follows FY – (Aug – Jul)
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
Cisco IT – Large ERP Systems analysis
Based on Cisco IT internal benchmarking
Value Add for IT
•Reduction in cost
•Better scalability
•Accelerate move off HP RISC platform30%
growth
UCS 4 node RAC 8 cores
SJPROD PEAK Req
QTCPRD PEAK Req
ECUST PEAK Req
C3 PEAK Requirements – Three SD 128 cores (40% - 30% - 60%)
CORE Peak Req
ODS Peak Req
G2C Peak Req
BVPROD Peak Req
SDMPRD Peak Req
ICW PEAK Req
1 node RAC HP IA Superdome 64 core
2 node RAC HP IA Superdome 32 Core
2 node RAC HP IA Superdome 16 Core
UCS 1 node 32 cores
UCS 8 node RAC 32 cores
3 node RAC HP IA Superdome 64 core/128 Cores/128 cores
CFN PEAK Req
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
Source System (HP Superdome) Target System (Cisco UCS B200)
1. 2 Node HP Itanium Superdome SD64B, Oracle RAC cluster with 32 CPU Core * 64 GB Memory each
2. E-Business version 12.0.4 using Oracle Incentive Compensation (OIC) module
3. Oracle database version 10.2.0.4
4. Tier-II DC
1. 4 Node Cisco UCS B200 M1, Oracle RAC cluster with 8 CPU Core* 48 GB Memory each
2. E-Business version 12.0.4 using Oracle Incentive Compensation (OIC) module
3. Oracle database version 11.2.0.1
4. Tier-III DC
G2C Migration Details• Goals to Commission (G2C) application supports
Cisco Sales compensation platform• No additional certification was required. UCS
hardware is already certified for target Oracle E-Business Suite applications
• No application re-architecture was required• One 10th the tile space, less than one third the
power utilization and less than half the data cables
HP Superdome to Cisco UCS Migration
Used Oracle DG for DC Migration
0.00
5.00
10.00
15.00
20.00
25.00
# of DC Tile Space Power Utilization
(KW) Number of Data Cables
4.00
13
24
0.40
4
10
Source HP Superdomes (2 qty)
Target Cisco UCS Servers (4 qty)
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
Quote to Cash Migration Benefits
100% of Cisco revenue flows through the Quote to Cash systems
UCS occupies one-sixth the data center space that replaced legacy
Consumes less than 65% power and uses 40% fewer cables
Legacy Current (Cisco UCS)
Production Server
Configuration
2 HP-UX Superdomes
(San Jose)
2 Cisco UCS B440 M1 Blade Servers
(Richardson))
Operating System HP-UX Version 11.11 Red Hat Enterprise Linux Version 5.5
CPU Cores 80 64
Memory 160 GB 512 GB
Performance Capacity for current workload Capacity for double the current workload
Database Oracle RAC 9i Oracle RAC 10g
Oracle E-Biz 11.5.9 11.5.9
Used Oracle DG for DC Migration
0
5
10
15
20
25
DC Rack spaces
Power Utilization (KW)
Number of Data Cables*
2 HPUX Superdomes plus peripheral 2 Cisco UCS Server B440
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
Cisco IT Superdome to UCS Instances and Cost
Cisco UCS – The Right Solution For Oracle
Capital Cost
reduction of
more than
75%
Saving
$167,200/yr
Of SW
maintenance
cost
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
User Experience Before and After
0
50
100
150
200
250
300
350
400
Cisco Invoice Interface -
US
Autoinvoice Import
Program
General Ledger
Transfer Program
Revenue Recognition
Order Import Aging - 7 Buckets - By
Account Report
Customer Interface -
US
Customer Interface -
JPN
Cisco Holds -Gross Dollars Report
Cisco Open RMA Report
Cisco Back Log Report
Cisco Backlog-AR
Report
63
326
168 172
103
19
268
50
108 107
60 62
HP Superdome (Feb 2011): Maximum Runtime (minutes) Cisco Unified Computing System (Mar 2011): Maximum Runtime (minutes) Gain
0
20
40
60
80
100
120
Aged Account Summary (Search by customer)
Aged Account Summary (Search by Bill To)
SOM Scrubber (Hold Criteria for AMS)
Change Order Cross Scrubber (By SO Order)
Change Order Scrubber (SO Search)
Order Inquiry (Search by SO number)
Unallocated Report –Download Report
26
6
19
67
20
52
13
Time on HP Superdome (Seconds) Time on Cisco Unified Computing System (Seconds) Percent Improvement
Batch
OLTP
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
Migrating a 40 TB Oracle RAC E-Business
Database from HP to UCS
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
1/3 data center space usedSupports Cisco Service Sales, Advance
Service Mgmt, Technical Service Delivery47 kW
power saved
23% of Cisco revenue ($10B annually) 60% reduction in CAPEX/OPEX
per annum
C3 Migration Benefits
Source (Legacy) Target (Cisco UCS)3 Legacy RISC-based IA Superdomes 12 Cisco UCS B440 blade servers
Production Server Configuration
HP-UX v11.23 Red Hat Enterprise Linux v5.5Operating System
384 384CPU Cores
1536 GB 3072 GBMemory
Capacity: current workload Capacity: 1.5 times the current workloadPerformance
3 node RAC Oracle 10g (CRS 10g) 12 node RAC Oracle 10g (CRS 11gR2)Database
N+1 ArchitectureLegacy: Cisco UCS B440:
15 4vs.
74 KW 27 KWvs.
241 100vs.
27% less Tile space
36% less Power
41% less cables
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 33
Migrate C3 Platform from San Jose to Richardson
Richardson, Texas
C3 Platform
Technical ServicesService Delivery and
Parts Fulfillment
Advanced ServicesManagement
of Services Projects Engagement
Service Sales20% of Cisco Revenue
BusinessValue
Improved User
ExperienceIncreased performance
Cisco on CiscoShowcase UCS &
Cisco Data Center
Capabilities
AgilityReduce time-to-market for
new business capabilities
ResiliencyEliminate Cisco Services
single source of failure
Application PortfolioRationalized
• Reduce complexity• Improve supportability• EOL unneeded items
Cost Optimization & Risk Avoidance
• Servers at CPU capacity• Support and license
fees are costly
San Jose, California
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
Legacy C3 Environment
• Highly active, revenue impacting OLTP database• Based on EBS 11i/10g ~ 200+ bolt-on applications• 3 RAC Node HP IA64 Superdomes • 128 Cores each @ 1.5 GHz, 512 GB RAM, EMC SAN• ~ 36 TB, expected 40 TB soon• Redo of 3 TB/day; peaks 4 to TB• ~ 8000+ sessions from Conc. managers and JVM connections• Reached hard limit on vertical scalability!
Considerable Cost to increase capacity
or move to next-gen Superdomes
with
Multiple life cycle env
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
8 Node UCS RAC
Required:
8 node certification Services Setup, RAT Testing
Capacity Analysis
Split Option ALL UCS
Required:
Business Buy in for Split Option
Services Setup, Capacity Analysis
Split Option UCS and HPUX in RCDN
Required:
Business Buy in for Split Option
Services Setup, Capacity Analysis
HPUX in RCDN as is with UCS provided for App failover
Required:
Migration and planning for Application failover
If Not Possible
If Not Possible
If Not Possible
1
2
3
4
Option DC OPS DBA APP
1
2
3
4
Option of last resort
2,206
4,626
378
0
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
4,500
5,000
AWR TPS Interconnect (MBPS)
1 Node
2 Nodes
Based on 0.8 RAC factor for 2nd node and 0.7 factor for subsequent nodesTwo Node TPS actual tests
Based on Cisco IT internal testing for heavy Database loads
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
C3 Migration Analysis
Oracle 10g with 11iLinux RHEL 5.5Oracle ASMServices based configurationApplications segregation with data model
Oracle 10g with 11iHPUX 11.21Raw VolumesServices based configurationApplications segregation with data model
Itanium 128 core node 1
Itanium 128 core Node 2
UCS 32 core core 3 node
UCS 32 core core 3 node
• High Operational impact of Scaling horizontally with Superdomes – limit 128 cores•Considerable capital cost to migrate to next generation•Very high support cost•High downtime to replace parts or upgrade
UCS 32 core core 2 node
Itanium 128 core Node 3
36 TB Data
Over 10000 TPS
Peak IO 5 GB/s
Less that 10 ms
average response
time
Application Profiling
App segregation and Load Balancing
Physical Design Verification
Load testing and validation
RATto test
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
(Known): Capture metrics at system (host, database) level.
(Known): Capture metrics at database session level by process.
Need a bridge between system and processes running within the database. Different types of metrics; TPS, CPU, IO vs. Logical IO, Physical Reads, Physical Writes
(Derived): Group processes by application (logical grouping); typically based on business process
alignment and data affinity.
Application grouping is bridge. Logical IO most closely relates to TPS. Generate ratio of a processes Logical IO out of total Logical IO for all processes. Trend metrics over a period of time.
How to connect our processes and systems within a large system so that we can distribute and scale? We leverage a blend of known facts and derived ratios.
Application Profiling
App segregation and Load Balancing
Physical Design Verification
Load testing and validation
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
C3 groups
Application Group TPS Node
EBS Concurrent Managers
Contract Management 1050 3
E-Biz and Misc CM Jobs 781 4
Quoting & Ordering 1565 5
Service Request, Service Supply Chain, Projects 1194 6
E-Biz Online (Forms, OA Web, Utility, Admin) 835 1, 2
Infra Processes, Adhoc Requests, Misc 758 2, 1
Online Technical Services (Service Request, Service Supply
Chain, Projects)785 7, 8, 9
Online Service Sales (Contract Management, Quoting
&Ordering, Config)1329
10, 11,
12
Total 8297 12
Application Profiling
App segregation and Load Balancing
Physical Design Verification
Load testing and validation
Application Group TPS Node
E-Biz Online (Forms, OA Web, Utility, Admin) 835 1
Online Technical Services (Service Request, Service Supply
Chain, Projects)785 1
EBS Concurrent Managers (Critical Technical Services Jobs) 430 1
EBS Concurrent Manager
Online (non-critical) + Batch + Failover
(Upstream/Downstream)1390 2
E-Biz and Misc CM Jobs 155 2
Infra Processes, Adhoc Requests, Misc 758 (1, 2, 3)
Online Service Sales (Contract Management, Quoting &
Ordering, Config)1329 3
EBS Concurrent Managers (Critical Service Sales Jobs) 2615 3
Total 8297 3
Before After
Superdome to UCS Application Mapping
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
Node2 Node4 Node1 Node3 Node7 Node5
CM1 CM2OTH
Order MgmtService Request
Projects
CCS-MSC
Service Agreements
Quoting
Node6 Node9 Node11 Node8 Node10 Node12
E-business Apps Forms/Java ( All)TPS: 835.12IO: 802.35 MBIOPS: 51KData: All
EBIZDB links, Infra/ Desktop clients, 3rd party, batch jobs etc.TPS: 757.45IO: 822.6 MBIOPS: 51KData: Other/NA
Conc. MgrGroup 1TPS: 899IO: 517.95 MBIOPS: 32KData: SA/IB
CM3 CM4
Conc. MgrGroup 2TPS: 681IO: 146.7 MBIOPS: 9KData: SA/IB + Quote
Conc. MgrGroup 3TPS: 1257IO: 422.55 MBIOPS: 27KData: Quote
Conc. MgrGroup 4TPS: 973IO: 761.85 MBIOPS: 48KData: All Remaining
Service RequestTPS: 552.1IO: 404.55 MBIOPS: 13K
Order ManagementTPS: 201.46IO: 121.95 MBIOPS: 17K
ProjectsTPS: 19.82IO: 12.15 MBIOPS: 4K
Service Agreement /Install BaseTPS: 459.42IO: 164.25 MBIOPS: 5K
QuotingTPS: 869.24IO: 317.7 MBIOPS: 10K
Miscellaneousonline
cci/x, perlonline
Misc DataTPS: 10.77 IO: 5.4 MB
IOPS: 15K
!ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1!
ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1!
ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1!
ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1!
ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1
!ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1
!ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1!
ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1!
ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1!
ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1!
ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1!
ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
Current C3 Environment – (N+1) with 50% more capacity
• Highly active, revenue impacting OLTP database• Based on EBS 11i/10g ~ 200+ bolt-on applications• 12 RAC Node Cisco UCS• 32 Cores each Xeon 7500, 256 GB RAM, EMC SAN• ~ 40 TB DB size• Redo of 5 TB/day• ~ 8000+ sessions from Conc. managers and JVM connections• Significantly increased horizontal and vertical scalability
Minimal Cost to increase capacity or
move to next gen UCS
with
Multiple life cycle env
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
SAN DW1
SAN DW2
SAN GW1
SAN GW2
SAN GW3
SAN GW4
SAN SW1
SAN SW2
SAN SW3
SAN SW4
San PC: 64 Gbps
San PC: 40 Gbps
Data Frame
Frame
Other
Apps
Frame
Set B
Frame
Redo Mirror
12 x DB
nodes
S L O T
1
S L O T
5
S L O T
3
S L O T
7
S L O T
2
SLOT6
S L O T
4
S L O T
8
!
U C S 5 1 0 8
O K F A I L O K F A I L O K F A I LO K F A I L
CO NS OLE
Cisco UCS 6140X P
PS
2P
S1
STAT
L1 MGM T 0
L2 MGM T 1
4 8 12 16
3 7 11 15
2 6 10 14
1 5 9 13
2 0 24 28 32
19 23 27 31
1 8 22 26 30
17 21 25 29
36 40
35 39
34 38
33 37
1 00-24 0V~9 .2 A
50 -6 0 H z
SLO
T2S
LOT3
1 /2/4 G F I BRE CHANNE L
N 1 0 - E 0 0 8 01 2 3 4 5 6 7 8
1 /2/4 G F I BRE CHANNE L
N 1 0 - E 0 0 8 01 2 3 4 5 6 7 8
CO NS OLE
Cisco UCS 6140X P
PS
2PS
1
STAT
L1 MGM T 0
L2 MGM T 1
4 8 12 16
3 7 11 15
2 6 10 14
1 5 9 13
2 0 24 28 32
19 23 27 31
1 8 22 26 30
17 21 25 29
36 40
35 39
34 38
33 37
1 00-24 0V~9 .2 A
50 -6 0 H z
SLO
T2S
LOT3
1 /2/4 G F I BRE CHANNE L
N 1 0 - E 0 0 8 01 2 3 4 5 6 7 8
1 /2/4 G F I BRE CHANNE L
N 1 0 - E 0 0 8 01 2 3 4 5 6 7 8
FI-A
FI-B
8 x vHBA
1&3&5&7
8 x vHBA
2&4&6&8
EMC
SET A
EMC
EMC
EMC
EMC
EMC
Redo Frame
Vsan 2
Vsan 1
FA to MDS 192 GbpsMinimal Cost to increase capacity
or move to next gen UCS
UCS throughput:FC (Storage): 128 GbpsEth (Network): 160 GbpsPer BladeFC: 128/12=10.66 GbpsEth: 160/12= 13.33 Gbps
If there is a Fabric Interconnect outage solution will still meet requirements
MDS 9513 – 13 modules (24, 48,..) Edge Core topologyData Frame VmaxCache – 512 GB Mirrored
RA
C
Priv
ate
netw
orkP
ublic
netw
ork
12 UCS B440Cores – 384Xeon CPU – 2.2 GHzFE Node – VMCM FE Node – B250(4)
0
2
4
6
8
10
12
C3 Data Frame Requirements today
C3 Data Frame requiremnts with 30% growth
Frame Capability tested (R/O)
Frame capability R/W (EMC input)
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
UCS Projections (based on RAT Testing and Production Stats)
Growth % 12.00% 46.1% 6% 5% 5% 3% -5% 4% 6% 6% 6% 5% 5% 5%
RAT CPU utilization Dec
9th data
Other system cpu load added
Projected cpuutilization Q4FY11
Q1FY12 Q2FY12 Q3FY12 Q4FY12 Q1FY13 Q2FY13 Q3FY13 Q4FY13 Q1FY14 Q2FY14 Q3FY14 Q4FY14
12 node Production Configuration
25.0% 37.0% 54.1% 57.4% 60.4% 63.7% 65.5% 62.4% 64.9% 68.6% 72.5% 76.6% 80.0% 83.6% 87.4%
Projected Peak utilization(15-20% net safety margin is reflected in the numbers)
0.00
10.00
20.00
30.00
40.00
50.00
60.00
2:5
1:0
2
2:5
3:0
2
2:5
5:0
2
2:5
7:0
2
2:5
9:0
2
3:0
1:0
2
3:0
3:0
2
3:0
5:0
2
3:0
7:0
2
3:0
9:0
2
3:1
1:0
2
3:1
3:0
2
3:1
5:0
2
3:1
7:0
2
3:1
9:0
2
3:2
1:0
2
3:2
3:0
2
3:2
5:0
2
3:2
7:0
2
3:2
9:0
2
3:3
1:0
2
3:3
3:0
2
3:3
5:0
2
3:3
7:0
2
3:3
9:0
2
3:4
1:0
2
3:4
3:0
2
3:4
5:0
2
3:4
7:0
2
3:4
9:0
2
3:5
1:0
2
3:5
3:0
2
3:5
5:0
2
3:5
7:0
2
3:5
9:0
2
4:0
1:0
2
4:0
3:0
2
4:0
5:0
2
4:0
7:0
2
4:0
9:0
2
4:1
1:0
2
4:1
3:0
2
4:1
5:0
2
4:1
7:0
2
4:1
9:0
2
4:2
1:0
2
4:2
3:0
2
4:2
5:0
2
4:2
7:0
2
4:2
9:0
2
4:3
1:0
2
4:3
3:0
2
4:3
5:0
2
4:3
7:0
2
4:3
9:0
2
4:4
1:0
2
4:4
3:0
2
4:4
5:0
2
4:4
7:0
2
4:4
9:0
2
4:5
1:0
2
4:5
3:0
2
4:5
5:0
2
4:5
7:0
2
4:5
9:0
2
5:0
1:0
2
5:0
3:0
2
5:0
5:0
2
5:0
7:0
2
5:0
9:0
2
5:1
1:0
2
5:1
3:0
2
5:1
5:0
2
5:1
7:0
2
5:1
9:0
2
5:2
1:0
2
5:2
3:0
2
5:2
5:0
2
5:2
7:0
2
5:2
9:0
2
EBS - N1 - Average 3.63
EBS,Misc-N2 - Average 7.11
Others,Misc-N3 - Average 4.4
CM1 - N4 - Average 9.29
CM2 - N5 - Average 6.08
CM3-N6- Average3.07
CSSD - N7 - Average 8.11
CSSD - N8 - Average 7.95
CSSD - N9 - Average 7.83
BASE - N10 - Average 17.13
BASE - N11 - Average 16.05
BASE - N12 - Average 16.98
RAT Testing from Dec 9th data
Actual utilization :- 40% for CM nodes, 20% for others
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
Application Performance
0
5
10
15
20
25
Search Service
Request by User Profile
Search Service
Request by XML
Search Service
Request by Manager
Search Service
Request by Customer
Search Service Request
Search Service
Request by User Profile
Search Service
Request by Contracts
HP Production SJ to SJ UCS Production RCDN to RCDN
0
2
4
6
8
10
12
14
Adding Install Site
on to a Quote
Create 2tier SMS3 Quote
Generate Quote
Moving Items
Install Sites (Move
Products)
Renewal Quote
Creation
Contract Number Search
Serial Number Search
Bill To Id Search -X-
Track
Quote Summary
Page
Serial Number Search -CSCC Page
Create New Quote
Contract Number
Search in Advance Search Page
HP Production SJ to SJ UCS Production RCDN to RCDN
1.62
1.37
1.2
1.25
1.3
1.35
1.4
1.45
1.5
1.55
1.6
1.65
HP UCS
Average Run Time (minutes) per exeution of Conc
programs running >=1 mt but < 30 mts 39.59
30.88
0
5
10
15
20
25
30
35
40
45
HP UCS
Average Run Time (minutes) per exeution of Conc
programs running > 30 mts
Co
ncurre
nt P
rogra
ms E
xecutio
n
Co
mpa
rison
Onlin
e tra
nsactio
ns E
xecu
tion
Com
paris
on
Sec
Sec
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
Instance Crash issue before and after
Node2
Node4
Node1 Node3 Node7 Node5
CM1 CM2OTH
CCS-MSC
Service Agreements Quoting
Node8Node10
Node12
EBIZ CM3 CM4
Miscellaneousonline
!ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1!
ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!A
03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1!
ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1!
ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1!
ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1
!ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1
!ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1!
ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1!
ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1!
ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1!
ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1!
ResetConsole
UCS B440 M1
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
A03-0300
GA
2
10k S
AS
300G
B
!
!
SLOT
2SLOT
1
Order Mgmt
Service Request Projects
Node6 Node9Node11
EBIZ,OTS, CM partial, MISC
CM most, MISC
Service Mgmt, CM partial, MISC
Instance Crash on any node• Significant impact to business• Manual failover with degradation• Downtime for business• Considerable effort from Infra
Instance Crash on any node• No impact to business• Automated failover with no degradation• No Downtime for business• Minimal effort from Infra
N+1 Architecture
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
Copy Export RMAN read RMAN write
�
4 TB / Hr�
�
4.5 TB / Hr�
�
40 TB
San Jose Data Center
Node1 Node2 Node3
Richardson Data Center
Storage
Frames
Richardson Data Center
Standby
Instance
Rman Write
Rman Read
Copy
ActivityTime Taken in
Hours
EMC recoverpoint sync up from San Jose
to Texas DC and validation of database0.5
DB backup before and verification on HP
server1
Pre-migration activities like re-sync domain
indexes, purge OKC tables, generate and
prepare XXTS scripts
1.5
Copy data files to Cisco UCS 8
RMAN conversion 1
Import metadata 5
DB Auditing 2
Validation and auditing
Auto configuration on DB nodes 3
Auto configuration on 11 FE nodes 10
Total time taken 32
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
Earlier we had multiple issues almost every day with C3 and there were two dedicated DBA
Now we are down to one DBA and have given him a other databases to manage as things are so quiet
Do we need to be worried about our job security
Coffee room discussion
© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public
Horizontal Vertical
Proposed C3 UCS Configuration allows for adding additional 12 nodes dynamically
FY09 - UCS 8 Core
FY10 - UCS 16 Core
FY11 - UCS 32 Core
FY12 UCS 40 Core plus
Additional cores for performance improvement
Increased processing power
Scalable architecture to meet C3 current and future needs
C3 configUCS B440
Solution design throughput:At 100% availability FC (Storage): 128 GbpsEth (Network): 160 Gbps
Two FI IO (6140): Per BladeFC: 128/12=10.66 GbpsEth: 160/12= 13.33 Gbps
At 50% FI availabilityThroughputFC (Storage): 64 GbpsEth (Network): 80 Gbps
Per BladeFC: 64/12= 5.33 GbpsEth: 80/12= 6.66 Gbps
Cisco IT has put a $42 B plus business on UCS