Future of Cooling High Density Future of Cooling High Density Equipment Equipment
Steve MadaraSteve MadaraVice President and General Manager Vice President and General Manager Liebert Precision Cooling BusinessLiebert Precision Cooling BusinessEmerson Network Power Emerson Network Power
2
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
AgendaAgendaAgendaIssues facing the Data Center– Managing the explosive growth– Power consumption getting the attention of Governments
Developing the right cooling strategy for today and into the futureExamples of how the right cooling strategy can lower the Total Cost of Ownership
Myth – High Density computing is more costly
3
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Managing High Density Servers Managing High Density Servers Managing High Density Servers
200028 x 2U Servers2kW Heat Load
200242 x 1U Servers6kWw Heat Load
20066 Blade Centers24kW Heat Load
2009Rabid Blades
40kW Heat Load
Heading Toward 50kW
200028 x 2U Servers2kW Heat Load
200242 x 1U Servers6kWw Heat Load
20066 Blade Centers24kW Heat Load
2009Rabid Blades
40kW Heat Load
Heading Toward 50kW
Rack density trend when fully populated with the
newest server technology
How most sites are dealing with server density
4
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Progression to High DensityProgression to High DensityProgression to High Density
1 kW
2 kW
5 kW
10 kW
15 kW
20 kW
Rack kW
Issues facing the IT ManagerGetting Air out of the RacksHot air mixing with the inlet of other racksDiversity of loads in the Data CenterNot aware that more Fans create heatFlexibility of “On Demand Cooling”
The average server replacement cycle is 3-4 years
5
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
DCUG Survey Results – What are the Biggest Issues facing the IT ManagerDCUG Survey Results DCUG Survey Results –– What are the What are the Biggest Issues facing the IT ManagerBiggest Issues facing the IT Manager
22%
19%18%
8%
7%7%
5%
4%3%
2%2%
1%
1%0%
0% 10% 20% 30% 40% 50%
Heat Density (Cooling)Space Constraints/Grow th
Pow er DensityAdequate Monitoring Capabilities
Availability (Uptime)Technology Changes / Change
Energy Costs / Equipment Eff iciencyOther
Security (Physical or Virtual)Data Center Consolidations
Data StorageHardw are Reliability
Regulatory ComplianceStaff ing/Training Limitations
6
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Air Movement
12%
Electricity Transformer/
UPS10%
Lighting, etc.3%
Cooling25%
IT Equipment50%
Source: EYP Mission Critical Facilities Inc., New York
Cooling Presents an OpportunityFor Energy SavingsCooling Presents an OpportunityCooling Presents an OpportunityFor Energy SavingsFor Energy Savings
Data Center Power Draws
CoolingAbout 37%-
45%
Sources of Energy WasteFans / Blowers running on redundant unitsLack of air containment (cable openings, room leakage)Unnecessary cooling unit cycling on and offLack of humidification control between unitsMixing of Hot and Cold air lowering the effectiveness of the cooling unitExcess Fan energy that turns into heat
7
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
US EPA Report Data Center Efficiency Public Law 109-431US EPA Report Data Center Efficiency US EPA Report Data Center Efficiency Public Law 109Public Law 109--431431
Energy consumed by servers & data centers has doubled in last 6 years and is expected to double again in next 5 years to > 100 Billion kWHState-of-the-Art technologies and best management practices could reduce electricity use by up to 55%Recommendations include:– Standardized performance measurements for data centers and
equipment– Incentive programs– Research and development– Industry collaboration and partnerships
www.energystar.gov/ia/partners/prod_development/downloads/EPA_Datacenter_Report_Congress_Final1.pdf
8
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Understanding The BasicsUnderstanding The BasicsUnderstanding The BasicsHeat generated is directly related to the server power (100%)As server power increases (kW), the airflow (CFM) through the rack increases proportionallyRaised floor tiles are limited in airflow (about 500-1000 CFM)Higher entering air temperatures on a cooling unit will provide more capacity and increased efficiencyHigher density servers will have a greater range of temperatures leaving the rack over time (larger swings of server load)Fan horsepower to move the air is significant and all the power turns into heat (100 kw Cooling unit uses typically a 10 HP motor that generates 8.5kW of heat)
9
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Having The Right High Density Cooling Strategy Delivers. . .Having The Right High Density Cooling Having The Right High Density Cooling Strategy Delivers. . .Strategy Delivers. . .
Cooling for High Density racks– Increase servers per rack– Increase number of racks per room
Energy Efficiency– Lower Operating Costs– More power allocated for IT/Server loads
Get more out of your existing facility
10
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
0 5 10 15 20 25 30 35 40kW / Rack
Computer Room /
Data Center
TRADITIONALFloor-Mount
SUPPLEMENTAL COOLING
+
Traditional Floor-Mount through the first 100-150 w/sqft(or 4-5 kW per rack) and Supplemental Cooling above
that level
Planning for High Density requires A Systems Approach to the Cooling SystemPlanning for High Density requires A Planning for High Density requires A Systems Approach to the Cooling SystemSystems Approach to the Cooling System
Controls Humidity and FiltrationCreates the base airflow distribution
Provides high density sensible cooling at the source
11
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Critical Requirements Of The Base Cooling Load EquipmentCritical Requirements Of The Base Critical Requirements Of The Base Cooling Load EquipmentCooling Load Equipment
Cooling units with Variable cooling capacity – DX – Compressors such as the Digital Scroll– VFDs on the fans
Controls so that units can work as a team– Eliminate dehumidification/humidification fighting– Balance the load– Optimize the cooling performance
High efficiency condensers“Green Refrigerant” products
12
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Moving the HeatMoving the HeatMoving the Heat
Areas for ImprovementReduced server fan powerHigher temperatures over cooling coils (room temperature / less mixing of hot and cold air)Reduced resistances lower fan power
Room
Rack
Chip
Memory Other componentsNetwork DevicesPower Supplies
Servers
Heat Sink
CRAH
Chiller Heat Rejection
13
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Cooling Solutions to Meet the Higher Density RequirementsCooling Solutions to Meet the Cooling Solutions to Meet the Higher Density RequirementsHigher Density Requirements
Requires moving the cooling closer to the heat source to more precisely cool the specific load and not waste energy “brute force” cooling the whole room.Cooling coils may be in multiple locations– External to the Rack – Overhead, Back, Side– In the Rack – Under, Side– Part of the Server
Requires a fluid to be delivered to the cooling coils to transport the heat out of the Data Center– Chilled Water (CW)– Refrigerant (pumped low pressure)
14
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Liebert XD & Base Cooling
CoolingUnit
Traditional Cooling OnlyFan Power- 8.5kW per 100 kW of Cooling
Average entering air temperature of 80-84°F
Fan Power- 3.5 kW per 100 kW of Cooling (XD @ 2 kW per 100kW)
Average entering air temperature of 96-98°F
Blower Resistances
Energy Efficiency Benefits for Cooling Closer to the LoadEnergy Efficiency Benefits for Cooling Energy Efficiency Benefits for Cooling Closer to the LoadCloser to the Load
65% less fan powerGreater cooling coil effectiveness
15
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Liquid Cooling Configurations –with Building Chilled WaterLiquid Cooling Configurations Liquid Cooling Configurations ––with Building Chilled Waterwith Building Chilled WaterBuilding Chiller XDWP
XDP
CW
RefrigerantCDU
HeatExchanger
Tank
Pump Pump
Valve
2ndLoop
Building Chilled Water
HeatExchanger
Tank
Pump Pump
Valve
2ndLoop
Building Chilled Water
16
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Open and Closed ArchitecturesOpen and Closed ArchitecturesOpen and Closed Architectures
Open and Closed Architecture Systems as defined by ASHRAE– The open architecture systems utilize cooling coils
near the heat load either inside or outside the server rack and utilize the room air volume as a thermal storage to ride through short power outages.
– The closed architecture fully encloses the rack with the cooling coil inside. Other provisions are required for power loss ride through.
17
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Secondary Fluid ComparisonsSecondary Fluid ComparisonsSecondary Fluid ComparisonsPumped Refrigerant SystemAdvantages– No Water in the DC or
electrical hazards– Micro-channel coil efficiency
(+50%), lower air side pressure drop => lower operating costs
– Smaller piping requirements– Cooling Modules can be
located anywhere– Scalable Capacity (2-3x to CW )
Disadvantages– Small room scalability – Higher Fluid Cost
Chilled Water Based SystemAdvantages– Lowest Fluid Cost– No limitation to room size
Disadvantages– Electrical Hazard– Lower Operating Efficiency– May require fluid treatment to
prevent fouling– Limited overhead cooling
options
18
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Flexibility - Extends Your Existing Infrastructure InvestmentFlexibility Flexibility -- Extends Your Existing Extends Your Existing Infrastructure InvestmentInfrastructure Investment
Hot Aisle – Cold Aisle ConfigurationHot Spots, Zones and Hot Rooms
Liebert XDO
On demand, plug-and-play flexibility to add additional capacityCooling at the source of heat with advanced compact heat exchangersMultiple module configurations to meet any Data Center layout Works with any brand of racksCooling fluid is a gas (no water)Self-regulating capacity100% Sensible cooling
Liebert XDV
Liebert XDH
19
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Plug and Play Capacity on Demand
20
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Liebert XD Energy Efficiency BenefitsLiebert XD Energy Efficiency BenefitsLiebert XD Energy Efficiency BenefitsCooling closer to the source – Dramatically less fan power required to move the air– Higher air temperature entering the coil results in
increase performance
Coil Technology– Microchannel coils – most efficient coil surface
Sensible cooling– All cooling module operate the coil at 5 degree F
above the dew point in the room– Does not unnecessarily dehumidify requiring additional
humidification (value of 7% in efficiency)
21
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
The Traditional Way (120 racks @ 8kw/rack)The Traditional Way The Traditional Way (120 racks @ 8kw/rack)(120 racks @ 8kw/rack)(12) 30 ton CW Air Handlers
– 10 operational for the load
– 2 stand by
Floor space – 4256 sqft
Requires a raised floor of 48”
PDU PDU PDU PDU PDU PDU
PDU PDUPDU PDU PDU PDU PDU PDU
PDU PDU PDU PDU PDU PDU
PDU PDU PDU PDU PDU PDU
AC AC AC AC
ACACACAC
ACACACAC
22
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Liebert XD Solution (120 racks @ 8kw/rack)Liebert XD Solution Liebert XD Solution (120 racks @ 8kw/rack)(120 racks @ 8kw/rack)
(4) 20 ton CW Air Handlers– 3 operational for the load
– 1 stand by
(6) XDP with (96) XDV– 5/80 operational– 1/16 redundant
Floor Space – 3640 sqft
PDU PDU PDU PDU PDU PDU
PDU PDUPDU PDU PDU PDU PDU PDU
PDU PDU PDU PDU PDU PDU
PDU PDU PDU PDU PDU PDU
AC
AC
AC
AC
XD
XD
XD
XD
XD
XD
23
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
120 Racks8 kW per Rack
Floor Mount AH
Liebert XD & Floor Mount
Rack Loads 960.0 960.0Fans(BHP) 101.7 44.6Room Latent 5.1 5.1Excess Latent 137.3 29.8PDU 28.8 28.8People 1.5 1.5Build Env 7.9 7.9Lights 5.6 5.6
Total (kw) 1247.9 1083.3
120 Racks8 kW per Rack
Liebert XD benefits
Total Room Load CalculationsTotal Room Load CalculationsTotal Room Load Calculations
– Less Fan power
– 100% sensible
Smaller load to size the Chiller
24
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Summary of EquipmentSummary of EquipmentSummary of EquipmentLiebert XD
(vs traditional method)
– 20% less Chiller plant
– 15% less floor space
– Scalable Platform
Floor Mount AH Liebert XD & Floor Mount
Chillers (3) 250 ton (3) 200 tonCW Pumps (3) 25 HP (3) 20 HP
Floor Mount Units (12) 30 ton (4) 20 tonLiebert XD (96) XDV
Floor Space (sqft) 4256 3640Raised Floor Height (in) 48 24
Scalable Design 8kw 20kw
120 Racks8 kW per Rack
25
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Annual Energy Consumption (kW)Annual Energy Consumption (kW)Annual Energy Consumption (kW)
Cooling closer to the source is more efficient.
120 Racks8 kW per Rack
Floor Mount AH
Liebert XD & Floor Mount
Prec Air Units 101.7 25.4XDV 0.0 19.2XDP Pumps 0.0 6.3CW Pumps 42.4 33.9Chiller 195.1 169.4Tower Fans 15.3 13.2Condenser Pumps 15.3 13.2Rehumidification 51.5 11.2
Total Kw 421.3 291.9
Operating Costs (@$.08/kw) 295,231$ 204,571$ Delta (90,660)$
-31%
120 Racks8 kW per Rack
26
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Capital Costs – Total ImpactCapital Costs Capital Costs –– Total ImpactTotal ImpactResults
Traditional – 4256 sqft
Liebert XD – 3640 sqft (-616 ft²)Example at $200 / ft²
Industry range $250-1000 / ft²
120 Racks8 kW per Rack
Floor Mount AH
Liebert XD & Floor Mount
Chiller 187,500$ 150,000$ Cooling Units 135,360$ 258,015$ Installation 322,860$ 452,265$ Floor Space -$ (123,200)$ Total (E,I,S) 645,720$ 737,080$
Delta 91,360$
Operating Savings (90,660)$ Payback (yr) 1.0
120 Racks8 kW per Rack
27
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Liebert XD Technology - Fluid Cooling at the Source Drives Down Operating CostsLiebert XD Technology Liebert XD Technology -- Fluid Cooling at Fluid Cooling at the Source Drives Down Operating Coststhe Source Drives Down Operating Costs
Chiller Capacity
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
Traditional CW CRAC CW Enclosed Rack Refrigerant Modules
kW C
hille
r Cap
acity
per
kW
of S
ensi
ble
Hea
t Loa
d Latent LoadFan LoadSensible Load
20% less capacity of the support equipment– Chiller (s)– Cooling Tower / Condensers– Chiller water circulating
pumps– Emergency Generators– Electrical Switch Gear
Annual Power Consumption
0.000
0.050
0.100
0.150
0.200
0.250
0.300
0.350
0.400
0.450
Traditional CW CRAC CW Enclosed Rack Refrigerant Modules
kW p
ower
to c
ool 1
kW
of s
ensi
ble
heat
Fan
Pump (CDU)
Pump (CW)
Chiller
30% Lower
28
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Liebert XDFull Range of OpportunitiesLiebert XDLiebert XDFull Range of OpportunitiesFull Range of Opportunities
XDPXDCXDV10
XDH20/32
or
Base Infrastructure
(160 kw)
Future pumping units of larger
capacity
Standard Cooling Modules 10-35kw +++
XDO20
Embedded Cooling
(microchannelintercoolers)
Tested 35-60kw
Developing up to 100kw
Embedded & Chip Cooling
(microchannelintercoolers and
Cooligy chip cooling)Tested 50 kw (100%
redundant)
Capable over 100kw
29
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
30%0.31
0.2
0.42
0.65
0.36
0
0.1
0.2
0.3
0.4
0.5
0.6
0.719
90
2000
2005
2006
/200
7
2004
/200
6
2006
/200
7
2010
Futu
re
Additional System Opportunities for Improved Cooling Efficiencies Additional System Opportunities for Additional System Opportunities for Improved Cooling Efficiencies Improved Cooling Efficiencies
Traditional Cooling Liebert XD Opportunities
XD M
odul
es
(10
–35
kW
)
Egen
era
(10
–20
kW
)
Embe
dded
Coo
ling
(35
–60
kW
)
Com
pone
ntC
oolin
g(>
50 k
W)
Encl
osed
Rac
k
kw fo
r coo
ling
per k
w o
f ser
ver h
eat l
oad
Data Center Best Practices
Energy Savings driven by reduction in Fan Power (cooling system and server) plus Heat
Transfer efficiency
45%O
ptim
al
30
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Cooling Process Throughout The Range Of Rack LoadsCooling Process Throughout The Range Cooling Process Throughout The Range Of Rack LoadsOf Rack Loads
Cooling Process Avai
labi
lity
Flex
ibilit
y
TCO
Avai
labi
lity
Flex
ibilit
y
TCO
Avai
labi
lity
Flex
ibilit
y
TCO
Avai
labi
lity
Flex
ibilit
y
TCO
Avai
labi
lity
Flex
ibilit
y
TCO
Avai
labi
lity
Flex
ibilit
y
TCO
Avai
labi
lity
Flex
ibilit
y
TCO
Avai
labi
lity
Flex
ibilit
y
TCO
Avai
labi
lity
Flex
ibilit
y
TCO
Chip/Component Cooling
Embedded Rack - Refrigerant
Rack Mounted - Refrigerant
Rack Mounted - CW
Traditional CRAH
Rack Load (kW)>0-5 >5-10 >10-15 >15-20 >50>20-25 >25-30 >30-35 >35-50
31
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Improving The Total Cost Of Ownership with the Liebert XD Cooling SystemsImproving The Total Cost Of Ownership Improving The Total Cost Of Ownership with the Liebert XD Cooling Systemswith the Liebert XD Cooling Systems
HowCooling closer to the source of the heat makes the heat exchangers more efficient (higher entering air temperatures)Lower total Fan HP
Sensible cooling eliminates wasted energy dehumidify unnecessarily and then having to re-humidifyLess Chiller or DX infrastructure required
Overhead cooling modules require no additional floor spaceCooling solutions that meets the requirements to fill racks of high density serversInfrastructure for modules today and for future server / rack designs
Results in …More cooling capacity for energy consumed
Less Power (energy consumed)Less Power (energy consumed)
Less Power and Capital EquipmentLess Floor Space consumedLess Floor Space consumedExtends your capital life
32
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Air Movement
12%
Electricity Transformer/
UPS10%
Lighting, etc.3%
Cooling25%
IT Equipment50%
Lower Power For Cooling Provides More Power For IT EquipmentLower Power For Cooling Provides Lower Power For Cooling Provides More Power For IT EquipmentMore Power For IT Equipment
Data Center Power Draws
59%12%
26%
For the same building power, you can allocate more power to the IT Equipment (18%)
33
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Take AwaysTake AwaysTake AwaysCooling solutions for higher density will need to move closer to the load and will require a reliable fluid delivery means– Cost effective cooling solutions exist that can be
employed today that meet future needs– Allow racks to be fully populated
Cooling at the source of the heat load will actually lower your incremental energy consumption– Less power for the cooling system provides more
power for the IT equipment
Results in more available Floor Space and Growth capability for the IT equipment
34
2007 IBM Power and Cooling Technology Symposium © Copyright 2007 Liebert Corporation. All rights reserved.
Thank YouContact:
Top Related