SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Green Datacenter...

SAN DIEGO SUPERCOMPUTER CENTER at theUNIVERSITY OF CALIFORNIA, SAN DIEGO

Green Datacenter Initiatives at SDSC

Matt CampbellSDSC Data Center Services Manager

[email protected]


SDSC Data Center Overview• 19,000 sq ft• 13 MW of on-site power• 100+ projects operating IT equipment in data center

G105G111

G110G109

G108G107

G106G119

G125G124

G123G122

G121G112

G118G117

G116G115

G114G113

J105

J111

J110

J109

J108

J107

J106

G126G132

G131G130

G129G128

G127

J112

J113

G133G134

G135

O105O111

O110O109

O108O107

O106O121

O119O125

O124O123

O122O112

O118O117

O116O115

O114O113

O126O132

O131O130

O129O128

O127O133

O134O135

R105R111

R110R109

R108R107

R106R121

R119R125

R124R123

R122R112

R118R117

R116R115

R114R113

R126R132

R131R130

R129R128

R127R133

R134R135

J128

J134

J133

J132

J131

J130

J129

J135

J117

J123

J122

J121

J119

J118

J124

J125

Air re

turn

Air re

turn

Air re

turn

Air re

turn

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

G120

O120

R120J120

XWVUTSRQPONMLKJIHGFEDCBA

101102

103104

105106

107108

109110

111112

113114

115116

117118

119120

121122

123124

125126

127128

129130

131132

133134

135136

137138

139140

141142

143144

145146

147

PD

U-U

1

PDU-N3

PD

U-N

2P

DU

-N1

PD

U-U

3P

DU

-U2

VentedTile

VentedTile

VentedTile Vented

TileVented

TileVented

TileVented

TileVented

TileVented

TileVented

TileVented

TileVented

TileVented

TileVented

Tile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

Iso-base area

184 sq. ft.

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

Iso-base area

96 sq. ft.

VentedTile

Iso-base area

129 sq. ft.

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

Iso-base area

30 sq. ft.

Iso-base area

96 sq. ft.

Iso-base area

96 sq. ft.

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

Iso-base area

96 sq. ft.

Power Pillar

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

U2F 40/42&

U1C 22/24

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64

65 66 67 68 69 70

A

B

C

D

E

F

G

H

I

J

K

L

M

N

O

P

Q

R

S

T

U

V

W

X

Y

Z

BB

CC

DD

EE

FF

AA

GG

HH

II

JJ

KK

LL

MM

NN

OO

PP

QQ

RR

B

A

C

D

E

F

H

I

J

G

K

L

M

N

O

P

Q

R

S

T

U

V

W

X

Y

Z

AA

BB

CC

DD

EE

FF

GG

HH

II

JJ

KK

LL

MM

NN

OO

PP

QQ

RR

716968676665646362616059585756555453525150494847464544434241403938373635343332313029282726252423222120191817161514131211109876541 2 3

U2F 10/12 & 14/16

U1C 14/16&

U2H 10/12

Pod A 37/39&

U4B 9/11(with in-rack UPS)

Pod A 14/16&

U2K 26/28U2F 17

U2E 35/37&

U4B

U1C 2/4&

U2H 2/4

U4B 32/34/36 & 38/40/42

U4B 26/28/30 & 37/39/41

U4B

Pod A 64/66, 68/70 & Pod B 64/

66, 68/70

U1C 5

U2I U1C

U2F U2E

PDBMillenniumU4B 1/3/5

PDB

From Millennium

PDB

From Millennium

PDBMillenniumU4A 1/3/5

PDB

From Millennium

U2K 1/3 (9311)U2K 14/16 (3590)

U2K 2/4 & U2F 35/37 (drive racks)

U2K 9/11 (9311)U2K 18/20 (3590)

U2I 35/37 (drive rack)

U2K 5/7 (9311)U2K 6/8 (3590)

U2I 6/8,10/12 (drive racks)

U2K 13/15 (9311)U2K 10/12 (3590)

U2K 17/19 (9311)U2K 22/24 (3590)

U3E-A 6/8 & 10/12U2K 29/31 & U2K 36

U2K 38

Power PillarU4A

Liebert air handler Liebert air handler

Liebert air handler Liebert air handler

Liebert air handlerLiebert air handler

Power PillarPower Pillar

Power PillarDistribution

panel

SATA 3 & 4

SAN 41HPSS

SAN 42HPSS

SAN 43HPSS

SAN 44HPSS

SAN 45HPSS

SAN 46HPSS

SAN 47HPSS

SAN 48HPSS

Distribution panel

SATA 1 & 2

SAN 56HPSS

SAN 55HPSS

SAN 54HPSS

SAN 53HPSS

SAN 52HPSS

SAN 51HPSS

SAN 50HPSS

SAN 49HPSS

U3A 1/3 & 5/7

U2K U2J

U2B

U2A

U1A

U2G (empty)

U3C

U3C-A

U3B

U3B-A

U3E U3D

U3E-A U3D-A

U1C 38/40&

U2J 26/28

U3E-A 2/4, 6/8, 10/12U4A10/12, 34/36, 37/39

U1C 7/9, 35/37U2J 13/15, 17/19, 21/23, 25/

27

U2I 18

U2I 26/28&

U4B 29/31

Pod B&

U1C 10/12

U2F 36/38(both PDU’s feed

from in-rack)

U2I 9/11&

U4B 21/23

U1C 31 & 33& U2E 17& U2I 21

Empty

U2I 39/41

Pod B 5/7&

U2I 38/40/42

Empty

72 73 74 75 76 77 78 79 8070 81

71 72 73 74 75 76 77 78 79 80 81

Power Pillar

U2K 21/23

GPFS-WANserver

GPFS-WANserver

Pod BPod A 26/28, 55/57

& Pod B 63/65

Pod A 6/8&

Pod B 55/57

On-Demand cluster

On-Demand cluster

On-Demand cluster

On-Demand cluster

GPFS-WAN

SAM-QFS

SAM-QFS

GPFS-WAN

GPFS-WAN

Pod A 30/32&

U2I 1/3

Physicsstorage

VentedTile

U2F 29/31&

U4B 17/19

U2F 5/7&

U1C 6/8

240v

240v

240v

240v

240v

240v

240v

240v

240v

240v

240v

240v

240v

240v

240v

240v

240v

240v

240v

240v

240v

240v

New APC

Pod A 21/23 & Pod B 21/23

240v

Pod A 22/24 & Pod B 22/24

240v

Pod A 25/27 & Pod B 25/27

240v 240v

IT-L7

240v

SDSCMars

240v

Nesis and GEON

240v

Pod A 10/12&

Pod B 47/49

240v

Pod A 1/3&

Pod B 29/31

240v

IT-J14HPSS tape movers

240v

6140 disk

240v

Engenio rack

240v

U1B 17/19&

U2H 25/27

240v

XFMRCDSA

XFMRCDSD

XFMRCDSE

XFMRCDSC

XFMRCDSB

SBCDS2

CDSA

CDSB CDSC CDSD CDSE

240v

Pod A 18/20 & Pod B 18/20

240v

Grey 2 Grey 1

240v

U2E 14/16

Power pod A Power pod B

VentedTile

VentedTile

240v

240v

240v

240v

240v

240v

240v 240v

240v

240v

240v

240v

240v

240v

240v

240v

240v

240v

240v

CMS disk

CMS blades

CMS networking

CMS disk Empty

Empty

EmptyEmptyEmpty

VentedTile

VentedTile

VentedTile

240v

240v

240v

240v

240v

240v

240v

240v

240v

U2E 32/34&

U2I 5/7

U2E 31/33&

U2E 7/9&

U2E 36/38&

U2I 14/16

U2F 6/8&

U1C 26/28/30

U2F 32/34&

U2I 13/15

240v

240v

240v

240v

240v

240v

U2F 9/11 & 13/15&

U2I 27/29 & 31/33

240v

New APC

U2F 26

GPFS-WAN

240v

Ne

twor

king

C

ore

U3A

U1C 32

U2E 24

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

240v

240v

240v

Liebert air handler

240v

Iso-base area

261 sq. ft.

Iso-base area

261 sq. ft.

UC colo

UC colo

UC colo

240v

240v

240v

240v

240v

240v

240v

240v

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

Iso-base area

316 sq. ft.

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

Iso-base area

316 sq. ft.

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

Iso-base area

349 sq. ft.

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

Liebert air handler

Iso-base area

360 sq. ft.

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

Iso-base area

360 sq. ft.

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

Iso-base area

360 sq. ft.

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC coloUC colo UC colo UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

Liebert air handlerLiebert air handler Liebert air handler Liebert air handlerLiebert air handler Liebert air handlerLiebert air handlerLiebert air handler

Gray system

Black system distribution cabinet

Blue jump-out

Power PillarU2H U1B

U2D U2C

New APC

New APC

New APC

U2B

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

VentedTile

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

UC colo

Power

Power


Key Points• How do we illustrate energy loss and quantify savings?

• Establishing a common unit of measurement for data centers• What areas can be improved?

• Mechanical• Electrical• IT equipment


Power Usage Effectiveness PUE =

• Industry accepted unit of measurement• 2+ is still the data center industry ‘norm’• SDSC’s datacenter came was audited last year with a resulting

PUE of 1.35 – 1.42.• Systems located at SDSC would save 28-65% on energy alone!

Total Facility PowerIT Equipment Power


Energy Savings Multiply

Courtesy Jack Pouchet, Emerson


Working on Many Fronts

Air handling retrofit with variable frequencydrive fans and modulating chilled water

valves to match cooling to real-time loads.

High-efficiency UPS Retrofit

Cold / Hot Aisle Containment

“Free Cooling”Opportunities

Efficient SystemLifecycles

Distributing Higher-Voltage(277/480V) Power to the

floor to reduce transformerand distribution losses.

Experimenting with DC-OnlyPower Distribution


Data Center Improvements to Lower PUE

• Mechanical Systems• Aisle containment• CRAH VFD retrofits• Blanking panels, floor brushes

• Electrical Systems• High efficiency UPS

• System Lifecycles• Virtualization• Consolidation of ‘ghost’ servers


Cold/Hot Aisle Containment

Cold Aisle Containment• Knuerr Coolflex

• First deployment in the United States• Requires a standardized datacenter• Enclosed cold aisles (70-78F)• Servers exhaust to room as a whole,

which runs hot (90-100F)

Hot Aisle Containment• Essentially a “heat chimney” on top of hot

aisles with doors at each end.• Flexible, can be phased incrementally• Cold room as a whole (70-78F)• Exhaust to enclosed hot aisle (100+F),

ducting directly to cooling systems

• Expect 50-80% better cooling airflow efficiency than traditional hot/cold aisle• Separates cold/hot air, eliminating the need for over-cooling, over-blowing• Allows for high temperature differentials, maximizing efficiency of cooling equipment


Filling In The Gaps

• Brushes to cover tile cutouts• Allows cables to pass through

while blocking air

•Cost effective blanking panels•Minimizes leakage


CRAH VFD Retrofits• Added variable

frequency drives to computer room air handlers• Allows CRAH units to

modulate based on control signal


Electrical Systems• High efficiency UPS units that can save ~500,000

kWh/year • Higher voltage to IT equipment

• Providing 277/480V power on the floor to more efficiently power HPC systems without transformer and distribution losses

• Exploration of DC power for IT equipment


Hardware Lifecycle Efficiency• Hardware retirement

• Equipment refresh more often leverages technology improvements

• Virtualization• Maximize equipment utilization

• Consolidation of ‘ghost’ servers• Underutilized IT equipment• ‘Forgotten’ equipment


QUESTIONS?

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Green Datacenter...

Documents

Transcript of SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Green Datacenter...