BayCare Health System Designs Next Generation Data Center August 8, 2007 Doug Lauterbach Data Center...

BayCare Health SystemDesigns

Next Generation Data CenterAugust 8, 2007

Doug LauterbachData Center Director, Bay Care Health System

Matt Schuster

Principal Consultant, BT INS

In today’s presentation…

• About BayCare• Challenges that necessitated a new data center• Planning the new facility• A look at the new data center• Your questions

About BayCare

• 9 hospitals in Tampa Bay area• Largest full service community-based health

system in the region• 17,000 team members• 2,707 Beds • 355,523 ER visits • 6.3 million lab tests

Challenges that necessitated a new facility

Business needs driving IT

• In the process of a 7 year electronic medical record project (moving towards paperless)

• Constant growth of digital imaging• Increasing compliance regulations - JCAHO

concerns about information management and availability

• Increased clinical reliance on IT systems requires high availability for quality patient care– Pharmacy– Lab– Radiology– Food and Nutrition– Bed tracking– Scheduling

Business Drivers of the new Facility

• Best Practices• Business Continuity• Business Expansion• Capacity Planning or Performance Analysis• Change in Application Architecture, Providers or Vendors• Compliance• Disaster Recovery• Expense Management• Increased Capacity• Operational Optimization• Regulatory• Reliability• Risk Management and Mitigation• Technology Growth or Consolidation• Technology Refresh

Existing facility challenge: space

• Short on space– 8 separate spaces for IT equipment– 4,700 sq. ft data center split into 4 spaces

• 30% of data center space used for circulation

• Large command center using a large portion of raised floor space

• Building’s physical resiliency not up to hurricane standards

• Design aspect to 40 watts/sf• Shallow cabinets limit the ability to expand and

distribute power and low-voltage cabling• Structure cabling unable to support:

– Blades– VoIP– Anticipated SAN growth

Existing facility challenge: power load

• Design didn’t support heat and power load• Downtime related to power design and capacity

– circuit capacity lacking– Distribution: not dual bus all the way to

equipment• Lack of available power threatening to limit new

technology adoption• No standard power configurations• No means or method to manage/measure power

at the equipment racks

Existing facility challenge: cooling

• Cooling was inefficient– Not using hot-aisle/cold-aisle design– Uneven temperatures through space– Hot spots creeping up

• Equipment lifespan likely shortened due to heat• 13-year old design, not optimized for blade

configurations• Not easily expandable• Low 12” raised floor and under-floor low-voltage

cabling restricting air supply• No available physical space for additional cooling• Legacy CRACs were not communicating to

efficiently cool the space.

Existing facility challenge: visibility & control

• Monitoring – currently focuses on circuits that are tripped (reactive), not designed for proactive data

• Temperature monitoring handled at the return air of the precision air conditioner (not looking at hotspots)

• Little remote control for servers• Lights-out management (iLO)• Existing Liebert SiteScan equipment was not

managing non-Liebert equipment• No temperature sensors at equipment location or

around the room• No branch circuit monitoring• 100% of lighting on 24x7

Existing facility challenge: managing change

• Provisioning is not standardized, automated or managed

• System deployments are based on:– looking through racks– personal knowledge– gut feeling– where uneducated users want placement

• Not managing capacity of:– space– circuits– power– weight– heat loads– network switch ports

Planning the New Facility

Objectives

• Provision for planned growth – addition of 100 servers per year

• Gain complete visibility and control and improve administration

• Improve system availability through full redundancy of the power and cooling systems

• Improve fire detection and suppression• Implement hot-aisle/cold-aisle design for maximum

capacity and energy efficiency• Improve planning and provisioning• Improve cable management• Increased Mechanical and Electrical efficiencies• Reduced operational costs• Operate as a nearly lights-out facility• Remote access into all systems

Building a Team

• Doug LauterbachImaging Services Director (serving as Data Center Project Director), BayCare Health System

• Chris Jenkins, Director Technology, BayCare Health System

• Matt Schuster, Principal Consultant, BT INS• Thomas Gooch, Local Liebert Representative,

Emerson Network Power• Paul Carastro, Carastro & Assoc, MEP Engineers• Mike Montecalvo, General Contractor, Solutions,

Inc.• Paul Schnitzlein, Architect, Harvard Jolly

Documenting Existing Equipment

• Document existing devices, power supplies and port counts per device at the rack level

• Determine/Graph device averages per rack to determine cabinet profiles

• Each piece of equipment assigned a number and quantities and loads were considered. Classifications included: – Legacy– Low– Medium– High– Blade– IBM RISC– Network

Planning Process: Modeling

• Mapped watts per square foot via CFD several times, tweaking each time to determine placement of precision air conditioners

• Room designed for 800-1,000 kW; CFD proves capacity

• 24-inch raised floor, multiple precision air conditioning units

• White space to grow into: 40 to 50 percent allows for doubling of capacity

• Rerun CFD every three to six months as changes are required using actual values collected from thermistors, branch circuit monitoring and other tools

Existing facility: Servers per Cabinet

Planning Process: Profile Planning

Cabinet Profiles

Best Practice: Technology Infrastructure

• 44U, 45” deep cabinets to allow for distribution of power and low-voltage cabling

• Metered Power Strips– Every cabinet has at least two strips, one for

each bus– Some cabinets with additional strip behind a

static transfer switch– Each strip has colored tape to quickly

determine which bus• KVM over IP in each cabinet

– Centralized– Decentralized

• iLO switch in each cabinet ($38/Ethernet port)

Best Practice: Power System Design

Liebert Precision Power Center

225 kVA480 Volt Input

120/208 Ouptut

Liebert 610 UPS 225 KVA

480v Input480v Output

BUS A

Dual Power Supply Servers

Automatic Transfer Switch

# 1RSC

Liebert 610 UPS 225 KVA

480v Input480/ Output

BUS B

70 kW A/C

BUSA

70 kW A/C

70 kW A/C

70 kW A/C

Main Electrical

distribution panel


# 3Building

800 A EmergencyDistribution

Panel480v

New caterpillarGenerator1000 KW


#2SPAWAR

Liebert Precision Power Center

225 kVA480 Volt Input

120/208 Ouptut

Dual BusRemote

Distribution Cabinet

BUSB

BUSA

Dual BusRemote

Distribution Cabinet

BUSB

Dual Power Supply Servers

Best Practice: Dual-Bus EPO Systems

Best Practice: Fire Suppression

• Very Early Smoke Detection Apparatus (VESDA)• Clean Agent Suppression system

– People-friendly– Environmentally friendly– Lower quantity of agent vs. FM-200– Over 3,000 pounds of FE-25 agent

• Four zones• Over 150+ detection points (above and below

floor)• Sprinklers were required, therefore pre-action with

double-interlock was installed

Best Practice: Remote Management

• Integrated Lights Out – allows remote control of servers from anywhere in the world

• People-free facility benefits:– Removed NOC Command Center from raised

floor– Minimizes human error disruptions– Allows managers to monitor and control

environment remotely

Best Practice: Proactive Monitoring

• Followed some of ASHRAE’s TC9.9 recommendations for monitoring temperature in racks/cabinets

• Temperature and Relative Humidity• Branch Circuit Monitoring• SiteScan I/O-32 and additional units• Leak Detection• VESDA• Fire Suppression• Fire Detection• Liebert Environmental Control Systems ICOMs

Control Network

Best Practice: Labeling

• Mechanical equipment• Electrical equipment• Electrical receptacles• Grid Coordinate• Racks and Cabinets• Structured Cabling

Tools: Rackwise/DCM

• Used tool to document existing inventories• Tracking relocation from one facility to another• Manage asset location by rack unit• Provisioning limitations per cabinet• Reporting capabilities

– Space and Assets– Power and Cooling– Cost and Capacity Planning– Rack Assembly with Cabling– Service & Warranty Contracts– Department & Customer Information

• Long term objective: to integrate a change/configuration management tool to track and manage assets

Tools: Rackwise/DCM

Tools: Computational Fluid Dynamics

• Allowed modeling the complete room for the initial 6 CRACs and full deployment of 12


• Initial assessment indicates potential problems related to placement of precision cooling units…

Low Pressure Zone

CRACs

1 2

34 5 6


• CFD Program allows for virtual change - validating proposed solution prior to installation.

CRAC Moved To New Location

Low Pressure Zone Removed


• Follow-on Modeling Provides Proof of Operational Success Assuming Full Load – 735 kW.

Clouds Per Fahrenheit Degree


• Modeling allows for proof of design concept at full load – prior to implementation

Pathlines from CRACs

Temperature (oF) in 1 foot increments

CRAC Network Communications

New Data Center – Construction

Current Data Center – Raised Floor

New Data Center – Raised Floor

Current Data Center – Operations

New Data Center – Operations

Current Data Center – Generator

Current Data Center – Network Rack

Current Data Center – Server Cabinets

Current Data Center – UPS Room

New Data Center – UPS Room

Current Data Center – Yard

New Data Center – Yard Construction

New Data Center – Yard Outside

Critical Success Factors

• Teamwork between BayCare, INS and Emerson Network Power/Liebert enabled success

• Maximization of modeling tools allowed a solid design, even before the first nail was hammered

• Have done virtualization study – partially executing recommendations ahead of time to make move easier

• Can’t convert everything to dual corded, so a Liebert STS was installed to provide redundancy

• Space for a second generator is provisioned and an ATS is in place to accommodate expansion

• Move in progress!

Questions and Answers

BayCare Health System Designs Next Generation Data Center August 8, 2007 Doug Lauterbach Data Center...

Documents

Transcript of BayCare Health System Designs Next Generation Data Center August 8, 2007 Doug Lauterbach Data Center...