VIACOM Data Center Optimization Project

VIACOMData Center Optimization Project

Executive BriefingMay 7, 2010John Wallerich,

Dell Global Services2

Agenda• Review Project Requirements and Objectives• Dell’s Gated Project Methodology• ASIS State Assessment• TOBE State Analyses• APC InRow Cooling Analysis• Implementation Cost Estimates


Project Objectives

IMPROVE

Cooling Infrastructure EffectivenessData Center CapacityIT Equipment TemperaturesCost of Operations

TOOLSComputation Fluid Dynamics (CFD)3-D Graphical Software to model airflow and thermal dynamics


Cost Improvement Categories• Improvements to room density and OPEXo Cool load using less cold air volumeo Reduce Bypass air o Reduce equipment Inlet air Delta-T o Raise Supply Air Temperatureso Reduce Fan Energyo Reduce DX Compressor loads

• Extend useful life of this data center and avoid new build


Services Engagement MethodologyAn Interface to Investment Decision-Making

An efficient “gated” engagement approach designed to arrive at solutions with reduced cost and improved efficiency


Glossary of TermsBLANKING PANELA plastic or metal plate fitted to the front of a rack where no IT equipment is present. BYPASSThis refers to the percentage of cold air that is generated by the cooling system, yet for one reason or another never passes through IT equipment. CACComputer Room Air-Conditioner. This is a generic term that includes all of the cooling devices in the Viacom data center. CFDComputational Fluid Dynamics. This is the technical term for the software tool that we use to model both airflow movement and thermal dynamics for both the under-floor cold air supply plenum, the computer room itself, and if used, the hot air return plenum (above the false ceiling). CFMCubic Feet per Minute. This is a measurement of air volume. For this study, this term is used to describe the amount of air processed by CACs as well as the volume of air flowing into the computer room through a perforated tile. RETURN AIRThis is a term that refers to the air that enters to CAC cooling units to be cooled.

HOT AIR RETURN PLENUMThis represents an area above the false ceiling when it is used as part of an airflow management system. COLD AIR SUPPLY PLENUMThis refers to the area under the raised floor where cold air is injected by the CAC units. The cold air is distributed into the room via perforated tiles.


Solution Options Summary

Solution Option #1: Deploy Hot Air Return Plenum

Solution Option #2: Install Airflow Management

Solution Option #3: Deploy Containment Strategy

Special Study: APC In-Row Containment Design

Dell Global Services

CFD Modeling

Computational Fluid Dynamics allows for the analysis of airflow and thermal dynamics with the detail and accuracy required for fine tuning of the data center cooling infrastructure8


Low Pressure resulting

from high tile count

Low Pressure resulting from high velocity

Low Pressure resulting from high velocity

Under-Floor PRESSURE DynamicsHigh

Pressure

High Pressure


ASIS: Thermal Slice at 6ft Elevation

Exhaust air contaminatio

n


n


n


ASIS: Max Inlet Temps and CAC % USEDASHRAE Temperature Guidelines


ASIS: Cooling Challenges

Hot air contamination

SAN RoomExhaust air is flowing over tops of racks contaminating surrounding racks.

AS400’sRacks are facing same direction so

exhaust air contaminates rows behind it with increasingly warmer

air.


ASIS: RETURN AIR STREAMS

CACs

3 CACs in center of room are drawing server exhaust from servers that are closer to other CACs


ASIS: AS400 AIRFLOW CONTAMINATIONExhaust air from the front rows is directed at air inlets of rows behind them. This increases the inlet air temp and puts the back rows at risk of over heating.


ASIS: BEST PRACTICES OBSERVED• Use of Blanking Panels good Best Practice• Use of hot-aisle/cold-aisle allows for enhancements• Plumbing and wire trays are well implemented• Lack of combustible materials• Lead-Acid batteries not on raised floor• Site is clean and well maintained• Cable cut-outs and floor leakage is minimal


ASIS: Design Weaknesses • AS400 racks are not optimized (facing same direction) • SAN room suffers from very low UF pressure• Room cooling is greater than needed to compensate for

weak airflow management design• CACs are of 2 different designs for high availability, yet

redundancy is no longer an option• CACs are aging and lack VFDs and multi-stage

compressors• Hot server exhaust contaminates server inlets as it flows

towards CACs – no airflow controls• Room capacity is limited by cooling effectiveness


Solution Option #1: DesignBasic Airflow Control: Return PlenumBy connecting the return ducts of CAC units to the

area above the false ceiling, and installing return air grills above hot aisles and other heat sources, the area above the false ceiling becomes a hot air return plenum.

Hot air is “pulled” from the room and returned to the CAC unit to be re-cooled.

This is a foundational requirement to installing more advanced airflow control measures and yields a number of benefits.


Solution Option #1: DesignHot Air Return Plenum

CAC Duct Extensions

Hot server exhaust air is removed from the room through ceiling

grills and directed back to CACs at higher

temperatures

CAC ducts connected to false ceiling turns ceiling void into a hot air return plenum


Solution Option #1Operational Statistics

Number of Cabinets 365 379

Total Cooling Airflow 199,538 CFM 187,138 CFM

Total Equipment Demand 142,069 CFM 142,069 CFM

Cooling Capacity 1213kW 1213kW

Cooling Capacity in Use 947kW 947kW

Maximum Inlet Temp 93F 87F

Number of Active ACUs 18 18

ACU Power Consumption 107.6kW 102kW

Note: Number of cabinets increased as some new racks were added to establish row symmetry (racks in purple)

Description ASIS SO#1


Solution Option #1: BenefitsImpact of Enhanced Heat Removal

• Hot exhaust air mixing with cool room air is reduced

• Return air is warmer: CAC can reject more heat

• Reduced contamination of nearby server inlets

• Supply air temperature increases from 58F to 64F

• Room can sustain increased IT power loads

• Can cool using reduced supply air


Solution Option #2: DesignESD Airflow Control CurtainsSolution Option #2 introduces Electro-Static Discharge curtains to provide enhanced isolation of hot and cold air, and to direct hot exhaust air into the hot air return plenum.

These curtains are located between the tops of racks and the false ceiling and help to isolate hot and cold air masses

ESD curtains have been adopted as a low-cost yet highly effective mechanism for simple airflow management.


Solution Option #2: DesignInstall ESD Airflow Control Curtains

ESD Curtains

Note: false ceiling not shown


Solution Option #2: DesignInstall ESD Airflow Control Curtains

Ceiling Grill

ESD Curtains

Hot Aisle

ESD curtains prevent hot exhaust air from flowing over the tops of racks into cold aisles where it would contaminate server inlets


Solution Option #2 DesignESD Curtains and Airflow Management

False Ceiling is not shown, but hot air removal and effect on cold aisles is apparent as a result of deployment of ESD curtains


Solution Option #2 BenefitsAmbient temperatures more stableA thermal slice at a 6ft elevation now shows dramatic improvements as a result of ESD airflow control curtains. The result is further improvements in effectiveness of cold supply air and reduction of hot air contamination


Solution Option #2 BenefitsCooling Effectiveness Improved

• Improved isolation of hot and cold air• CAC return air is warmer due to reduced mixing• Supply Temperature is increased from 60F to 68F• Rack Delta-T is reduced• Volume of supply air is reduced due to higher

effectiveness









ACU Power Consumption 107.6kW 97.5kW




Solution Option #3: DesignContainment StrategyContainment represents the most advanced airflow

management system due to the creation of a closed airflow loop. This solution builds on Solution Options 1 and 2, and provides the ultimate solution in terms of achieving the stated project objectives.

The concept behind containment is to completely enclose either a hot or cold aisle. Cold aisle containment prevents cold air exiting perforated tiles from going anywhere but through equipment. But the problem then becomes the effect of hot air entering the room.

Hot aisle containment encloses the hot aisle and provides a number of major benefits.


• Creates slight negative pressure in hot aisle which “pulls” air through the air inlets

• Pressure difference eliminates seeping of hot air through the rack into the air inlets

• Entire room becomes cold air plenum: the volume of air supplied at a given perforated tile is no longer critical

• Allows for racks in areas where there is low UF pressure to still get enough air as long as total supply is > total demand.

• Rack Delta-T is very low allowing for increase in supply temps

• Consumes 30% less air than Solution Option #1• Supply Temperature raised to 74F

Solution Option #3: BenefitsHot Aisle Containment Benefits


Solution Option #3: DesignContainment Strategies: CAC

Example of a Cold Aisle Containment Strategy. Notice the return air ceiling grills above the hot aisles

PROS:Reduces require air volume by 20%Isolation of hot exhaust airDoes not require ceiling plenum

CONS:Hot air enters roomVery little excess cold air capacity


Solution Option #3: DesignContainment Strategies: HAC

Example of a Hot Aisle Containment Strategy.

PROS:Reduces require air volume by 30%Isolation of hot exhaust airPulls air through servers

CONS:Requires hot air plenumCreates warm work area


Solution Option #3: DesignCombined Containment Model

Example of a Contained and non-contained areas


Solution Option #3: BenefitContainment Benefits: CAC vs. HACCold aisle Containment:

Reduces bypass air by enclosing cold air supplyEliminates hot air exhaust contaminationDoes not require hot air plenumReduces total supply air by 20% over SO#1

Hot Aisle Containment:Maximizes air bypass reductionHigher degree of airflow control (pull vs. push)Removes all hot air from roomReduces total supply air by 30% over SO#1Increases rack densityExtremely even Delta-T across rack face


Solution Option #3: BenefitIncreased Supply Air Temperature

Raising supply temps reduces costs of cooling, and requires less cold air volume.

Ambient room temp varies only slightly and is a comfortable 75F.

Heat rejection is 96% done by the CAC


Solution Option #3: BenefitStable Equipment Supply TemperaturesA combined

containment strategy has resulted in a significant increase in supply temperature and reduction of total cold air supply needed. This translates into reduced OPEX while providing a more stable operating environment as well as the ability to increase equipment power loading.


CAC Cluster Creates Low Pressure Zone

CACs

Low Pressure Zone

One factor that is limiting CAC effectiveness is caused by 3 CACs near the middle of the room. These units and their close proximity to each other creates a low pressure zone that pulls exhaust air from










ACU Power Consumption 107.6kW 97kW



Special Design Review APC High Density ClusterHot Aisle Containment: sub-optimum

APCs leaking warm air

APC’s InRow

coolers are an efficient

solution, but their

effectiveness is lost on

hot aisle containmen

t8 APC InRow Coolers with 12-12kW racksContained area is so hot, the coolers are sending 85F air into the room


Solution Design ReviewAPC High Density ClusterCold Aisle Containment

APC units are optimized when in a cold aisle containment design. Total load is 216kW over the hot

aisle maximum load of 144kW

and all equipment is experiencing inlet temps in the mid-70F

range. 6 APC InRow Coolers with 12-16kW and 2-12kW racks. All rack inlets are below 78F and hot air is not affecting local racks.


Solution Option Comparative AnalysisNumber of Cabinets 365 379 379 379

Total Cooling Airflow 199,538 CFM 187,138 CFM 178,138 CFM 165,419 CFM

Total Equip. Demand 142,069 CFM 142,069 CFM 142,069 CFM 142,069 CFM

Cooling Capacity 1213kW 1213kW 1213kW 1213kW

Cooling Capacity in Use 947kW 947kW 947kW 947kW

Maximum Inlet Temp 93F 87F 84F 81F

Number of Active ACUs 18 18 18 17

ACU Power Consump. 107.6kW 102kW 97.5kW 89kW

AVG. Supply Temp. 60 62 63 68

Description ASIS SO#1 SO#2 SO#3


Cost Basis Metrics and Calculations

1kW of IT Electrical Load - Annual cost: $6,000(space, power, cooling)

Annual OPEX expense per 3kW rack $18,000

1F of Set Point increase = 1% energy reduction(CAC compressor efficiency*)

• If VFD’s were deployed, an additional 20% of total CAC energyreduction would be realized as a result of reductions in fan energy.Dependency is that Solution Option #3 would be deployed.


Data Center Design Comparative Models

CURRENT STATE METRICS Number of Racks 365Total Cooling Capacity 1213kWTotal Cooling Load in Use 947kWTotal Connected Power 905kWServer Temperatures 65-95F

CONTAINMENT DESIGN UNDER MAX LOADNumber of Racks 397Total Cooling Capacity 1213kWTotal Cooling Load in Use 1156kWTotal Connected Power 1114kWRoom Temperatures 72-82F

DELTA

+32

+209kW+209kW-9F (max)


FINANCIAL ANALYSIS – Containment CostsCeiling Grill Construction

Grills (1540 sqft @ $1/sqft) = $ 1,540 Containment Curtains and Doors* = $ 78,000

Additional Racks (12) =$ 18,000

TOTAL $ 97,540

*includes CAC ducts to ceiling plenum

Recommended:Replace CAC’s with more efficient units $1,800,000


Containment Cost Reduction EstimatesOPEX Reduction

1. Reduce cooling costs by 6% annually @ $360/rack365 racks X $360 = $131k/yr

2. Equipment replacement (annual) $20k/yr3. Upgraded CAC cooling efficiency gains $48k/yr

TOTAL $199k/yr

CAPEX Reduction1. Increase rack power loads by 30% (avoid new racks) $200k2. Avoid adding new CACs (3 @ $60k) $180k3. Increase cooling system effectiveness (no cooler exp.) $500k

TOTAL $880k OTCCOST AVOIDANCE

4. New Data Center (20 sqft) Estimated at ~$30m


Recommendations1. Deploy Solution Option #3: combined containment strategy. This will reduce OPEX while extending the capacity of the existing Cooling infrastructure and data center.

2. Virtualize as much of the environment as possible. This will have a multiplier effect in terms of power reduction (each W of IT power removed equates to 2-3 Watts of electrical and cooling overhead reduction

3. Shorten the server refresh cycle. Consider a 3-year plan. This will provide benefits both in terms of service levels from faster servers, and will also have a multiplier effect in terms of power savings as a result of more power efficient processors.


Additional Recommendations (1)1. Replace all CACs with more efficient models:

VFD fans (10% reduction in speed = 50% reduction in power)Multi-phase compressors (2 or 3 stage compressors)

2. Relocate new CACs for better distribution and coverage

VFD’s and Multi-phased compressors will help reduce power Consumption for the cooling infrastructure particularly for dataCenters with containment strategies.

Containment allows for the same degree of cooling using less volumeof air. So, VFD’s, totaling over 80kW today could be cut in half.

Multi-phase compressors allow CACs to provide required coolingrunning in a more power efficient mode (stage 1 vs. stage 2)


CAC SET POINTS and MIN SUPPLY TEMPS CAC # ASIS SO#1 SO#2 SO#3

MST SP MST SP MST SP MST SP

CAC-1 60 68 64 88 66 88 62 88

CAC-2 60 68 66 86 66 90 64 90

CAC-3 60 68 62 86 66 88 64 86

CAC-4 60 68 64 86 66 86 60 86

CAC-5 60 68 64 86 66 88 64 88

CAC-6 60 68 64 86 64 86 64 84

CAC-7 60 68 64 88 64 88 66 90

CAC-8 60 70 66 90 64 90 66 88

CAC-9 60 67 64 78 64 84 60 82

CAC-10 60 68 64 86 66 88 60 90

CAC-11 60 72 62 76 64 84 60 76

CAC-12 60 68 64 86 64 90 60 88

CAC-13 60 66 64 88 66 88 62 84

CAC-17 60 68 62 78 62 80 62 78

CAC-18 60 68 64 80 60 76 OFF OFF

CAC-19 60 68 64 84 62 78 62 80

CAC-21 60 65 66 88 64 88 60 84

CAC-22 60 70 64 88 64 86 64 88

AVG 60.0 68.1 64.0 84.9 64.3 85.9 66.25 85.3


Thank You

VIACOM Data Center Optimization Project

Documents

Transcript of VIACOM Data Center Optimization Project