r

High-Fidelity Simulation of aComputer Room

-.0.By Shishir'Pandya

NASA Ames Research Centerfor Hewlett-Packard Labs

Collaborators:Jasim Ahmad, EL.ORET Inc.

William Chan, NASA Ames Research CenterNeal Chaderjian, NASAAmes Research Center

I

%VOutline,

• Introduction to Columbia

• Motivation

9 Methodology

• Results

• Concluding remarks

https://ntrs.nasa.gov/search.jsp?R=20090032250 2018-07-14T15:23:25+00:00Z

Columbia

FI

F,

zs*^tias^i.;

• 10240 CPUs

20 Sgq ACtix superdusters (512 CPUS/duster)

• 2nteC gtanium(b 2 processors (1.S G?-fz)

• 1TB n1nOry/cC14tcr, 44oTB storage-

- Sgq s "N UMAffex architecture (7f finiband )

Top performance to date: 51.E TFlops

inpack benchmark

#2 on the TOPSoo Cist-

wilt in < months full -- operational5 ^ Y P

wwl nas. nasa.gov/About/Projects/Columbia/columbia.htm I

3

Advanced Supercomputing Division

4

1• r

Motivation

• Need to know if cooling is adequate• 7dentify T6g-temperature regions

• Effect of ca6Ce trays in 6or isles

Find dead-zonesF • Areas of stagriatedair

• Short-cycling• CoCd air from f loor tiles returning directCy to coolers

I' • '16ot air from racks returning to racks

' Deadline: 6 weeks

,

LAM

M-ethodology`ter

• Overset mesh approach

• Geometric model1VOfi-Vr^ii j 5Cryt- 6asedmodern,

• `Many structured meshes make up the domain of interest'

• C61'mera Grid i-oA (cGj-)

• PDE solution model• Compressi6Ce ?favier-Stakes equations

• 'fiouudary conditions

• Ov40w2 so6ver

6

Body-fitted grids• COMYCex geometry

f Geometry• %u driCateraCcefrs

Overlapping POW(

Distri, Jb-u

Units (P

17

JT Overset Mesh Approach

Simulation on an overset mesh

i. Generate surface meshes

2. Generate volume meshes

3. Specify initial conditions

4. Apply boundary conditionsf

S. Solve on each volume mesh

6. Interpolate flow data in overlapping regions

7. Repeat 4-6 until converged (steady results)

8

i3 Coolers

1r Cable Trays

Perforated FloorTiles

Mass Storage

Geometric Model

9

PDE Solution Modeln

® Overflow2 (NASA/Army developed)

102 structured overset meshes, 12 miMon points

• ,Soruti6n. of 9-,f r-Stokes! equations

• Compressi6le, viscous f rows

• fow-speed, pre-conditioner for accuracy

it Obtain steady-state

• kesult: f row field/temperature fieldG^t - 01,, io®R I

` r

4 '^ t

QApproach

• Develop geometric model

• 2D sub-floor model

• Develop necessary boundary conditions

• 3D main-floor model_

• Flow visualization

ZD sub-floorCooler exit Pipes Perforated tiles

• Simulation indicates that pies 6Cocking sub floor f l-ow haves

CittCe eect on t<e per orated tfk f row rags

• _Qterature: A fuLf three d^itn-enswnaCsu6floorsimuCation wAy pes and

conduits fount(littCe ITCM, tare variation in Poor ties except near coolers.

Karki, K.C., Radmehr, A., and Patankar, S.V., "Use of Computational Fluid Dynamics forCalculating Flow Rates Through3Perforated Tiles in Raised-Floor Data Centers,"HVAC&R Research Journal, Vol: 9, No. 2, April 2003, pp. 153-16

• REASONABLE ASSUMPTION: THE COOLER FLOW RATE

IS EVENLY DISTRIBUTED AMONG ALL THE

PERFORATED FLOOR TILES

t ^

Boundary Conditions

• Perforated tiles: Temperature, speed known

• Rack intake: Speed based on fan rating

Rack exit: Known temperatur6 increase

• Cooler intake: Known speed based on CFMrating

13

Cool-Air Floor Tile

Disk Rack

Temp of r ^: _^86.3

782

62.1

70:2 Hot Isle

11 54.0 Cold Isle

Surface T(mP i-iern re

I4

Temperature/velocity vectors1'

s_

r ^^^ 1

,J^ is

SA`•'

CPURa

PD-U

Disk Rack P^

It24r1

Temperature/velocitv vectors31 v

16

CPURack

PDU

Cab1P Tr^^r

Flow Down DraftBlockage (Streamline Short Cycling)

'^^' ,^^^'dsL^3t`^E9TF"'€'..'..k:^s'^c^5_"° vs`:•&Z?'F_^.T^^iP^#r^7a'a'?ice€-. .... > :(^.'^`t" _ ^,

is

^e

)w,tion

ers

-0

' Slight Short Cycling Temp °F

At 30,000 CFM/ISU1 78.

78.22

70.262.154.0

Streamline Short Cycling

4 ^^

# « r : `Ericls Tendfloi93 Warmyyyy A;`

84.df!": 3 p } Short cycling

fi

1

19

,._,_

drftell

a^1Sul More Cool Air Less Cool Air` Enters CPU Fan Inlet Enters CPU Fan Inlet

UM

Cooler With Cooler Without

Diverter Diverter

Possible. Modification:Dver ter on Cooler

20

n I •

'

a

^r

Concluding Remarks

°-! Used overset mesh technology to evaluate thefluid/thermal character of a computer room

- • Computed average temperature in room tocertify the adequacy of the coolers

Identified high' temperature regions and deadzones

Improved understand i ng of effect of the cable

,k trays on local temperature

Identified short-cycling

3-• ^estecfyossi6Ce inodifi"cation to reduce short-cyc6incL,

Y _