SAMPLE ABC College Data Center Assessment & Recommendations

Data Center

Assessment & Recommendations

Quad Hall Anex

Prepared for: ABC College

86 Turner Road

E. Falmouth, MA 02536

Prepared by: Jeffrey Greenwood

Submitted on: May 2, 2015

ABC College Data Center Assessment & Recommendations

Revision: 7 Greenwood PCM - Confidential and Proprietary Property of 75 Do not scan or otherwise copy. Copying is illegal. Do not make or distribute copies for permitting, construction,

collecting bids or estimates

Table of Content

Table of Content ..................................................................................................................................................................... 2

Vision Team ............................................................................................................................................................................. 5

Revision History ...................................................................................................................................................................... 5

Greenwood PCM Physical Infrastructure SDF (Strategic Delivery Framework) Methodology ............................................... 6

Teaming Methodology ............................................................................................................................................................ 7

Executive Summary ................................................................................................................................................................. 7

ABC College Data Center Analysis Summary Chart ................................................................................................................. 8

ABC College Data Center Analysis Summary Chart ............................................................................................................. 9

Facility Site Selection ............................................................................................................................................................ 12

Description ........................................................................................................................................................................ 12

Facility Site Selection Findings .......................................................................................................................................... 12

Facility Site Selection Recommendations ......................................................................................................................... 16

Facility and Operations ......................................................................................................................................................... 17

Description ........................................................................................................................................................................ 17

Facility and Operations Findings ....................................................................................................................................... 17

Facilities & Operations Recommendations ....................................................................................................................... 18

Existing Computer Room and Equipment Placement ....................................................................................................... 20

Data Center As Built Raised Floor Perforations -Visio- ..................................................................................................... 22

Data Center As Built Racks –Visio- .................................................................................................................................... 23

Data Center As Built All Assessed Data –Visio- ................................................................................................................. 24

Power Requirement Methodology & Caveats ...................................................................................................................... 29

Fluke 435 Three-Phase Power Quality Analyzer Features ................................................................................................ 30

Ideal 61-958 Inductive Circuit Tracer Features ................................................................................................................. 30

Data Center Power ................................................................................................................................................................ 31

Description ........................................................................................................................................................................ 31

Data Center Power Finding ............................................................................................................................................... 31

Data Center Power Recommendations ............................................................................................................................ 32

Existing Panel Schedules & Load Statistics ........................................................................................................................ 35

Existing Rack Load Summary ................................................................................................................................................. 39




Data Center Cooling .............................................................................................................................................................. 40

Description ........................................................................................................................................................................ 40

Data Center Cooling Findings ............................................................................................................................................ 40

Data Center Cooling Recommendations ........................................................................................................................... 42

Budgetary Proposal ............................................................................................................................................................... 43

UPS Potential ROI .................................................................................................................................................................. 44

Gartner & Uptime Institute Data Center Tier Classifications ................................................................................................ 45

Fluke Power Quality Report Finding ..................................................................................................................................... 46

Computational Fluid Dynamics Findings ............................................................................................................................... 56

CFD Results As-Built Room Analysis .................................................................................................................................. 56

CFD Solution Design Options ............................................................................................................................................ 64

CFD Findings Down Flow CRAC Solution Analysis ............................................................................................................. 64

CFD Results Close Coupled Cooling Analysis ..................................................................................................................... 69

Onsite Pictures & Reference Material .................................................................................................................................. 71

NEC EPO Requirement ...................................................................................................................................................... 74

Raised Floor Weight Capacities ......................................................................................................................................... 74

Raised Floor Grounding Notes .......................................................................................................................................... 74

Original Generator Sizing Worksheet ............................................................................................................................... 75




May 2, 2015

David Turner

ABC College CIO

Dear David,

I would like to thank you and ABC College for the opportunity to assist in your data center assessment and expansion

feasibility study in the ABC College data center located at 86 Turner Road E. Falmouth, MA.

This deliverable was created specifically for ABC College to provide “as built” documentation to determine the existing

power, cooling and physical space capacity identify deficiencies and options to increase availability.

After a review of our preliminary findings, scaled Visio drawings and project status updates during a March 26, 2015

webinar. Greenwood PCM was asked to increase the initial power & cooling sizing from 40kVA to 80kVA and add

concurrent maintainability components to closely achieve Tier III availability levels.

Greenwood PCM worked with the ABC College IT staff and facilities to confirm the availability of a second utility source

and site for a second generator which could not be confirmed in the allotted timeframe. However, the proposed solution

with redundant UPS’s and CRAC units can be easily retrofitted to make use of these power sources should they become

available in the future without disrupting services.

Greenwood PCM strives for unmatched commitment to customer satisfaction and believes the following findings and

recommendations will provide a framework to enhance the availability levels and growth potential of the data center to

ensure that it matches the business continuance requirements of ABC College.

Sincerely,

Jeff Greenwood

Senior Data Center Design Consultant

Designer – Physical Infrastructure

Notice:




Greenwood PCM does not accept any responsibility or liability in any manner for how this report is used by the Customer.

Vision Team

Name Company/Function Phone E-mail

Jeff Greenwood Greenwood PCM Senior Data

Center Design Consultant 1-508-548-4474 [email protected]

David Turner ABC College CIO 1-508-XXX-XXXX [email protected]

Nick Ayers ABC College 1-508-XXX-XXXX [email protected]

David Drew ABC College

Senior Systems Administrator 1-508-XXX-XXXX [email protected]

Jim Sweeney ABC College 1-508-XXX-XXXX [email protected]

Revision History

Revision Revision Date Name Notes

Revision 0 2/8/2015 Jeff Greenwood PSA signage

Revision 0a 2/28/2015 Jeff Greenwood Onsite visit to start initial as build scaled drawings

Revision 0b 2/29/2015 Jeff Greenwood Onsite continue adding as build scaled content

Revision 0c 3/2/2015 Jeff Greenwood Continue adding content from as built data

Revision 1 3/5/2015 –

3/8/2015 Jeff Greenwood

Continue adding content from as built data, send designs to

vendors for pricing

Revision 2 3/26/2015 Jeff Greenwood

GoToMeeting review current status and preliminary designs.

ABC College & UMass consensus to go with redundant 80kVA

solution instead of 40kVA as originally proposed. (Original

sizing based on current load of 29kVA and downsizing by

moving to cloud computing)

Revision 3 3/27/2015 –


Continue adding content from as built data, send new designs

to vendors for pricing for larger UPS & CRAC’s

Revision 4 4/3/2015 –


Continue adding content based on recent changes and push

vendors for quicker budgetary numbers

Revision 5 4/30/2015 Jeff Greenwood Continue adding content


Finish content, brief grammatical error review, add Jonathan

Caserta CFD report to end of report and email deliverable to

customer


After presentation Nick Ayers request all notices regarding

confidential, proprietary, duplication, redistribution, liability

and potential use to be removed.

mailto:[email protected]












Greenwood PCM Physical Infrastructure SDF (Strategic Delivery Framework) Methodology

Greenwood PCM physical infrastructure team uses a SDF process. Which includes a built in mechanisms to ensure each

phase and milestone is properly documented and communicated to mitigate risk and is delivered OTOB (On Time on

Budget).

Greenwood PCM -

Greenwood PCM




Teaming Methodology

To create the best possible outcome with the lowest exposure to risk, Greenwood PCM fosters a teaming approach

during each phase of the project lifecycle. Teaming means that both ABC College, Greenwood PCM and its partners will

have joint tasks and responsibilities. Teaming also may incorporate contractors such as electricians, plumbers,

mechanical, engineering, carpenters, riggers, movers and other third parties when appropriate.

Each team is comprised of subject matter experts that come together in a synergistic fashion to the benefit of ABC

College. Greenwood PCM and its partners typically leads the team as a subject matter expert in technology, where ABC

College is obviously the expert in matters that pertain to their own services, as well as the way that technology is used to

the benefit its services.

Executive Summary

Greenwood PCM highly recommends that the first and foremost decision that must be made is to verify and document

the Tier level requirement for this facility to confirm the physical infrastructure (Power, Cooling & Physical Infrastructure)

meets or exceeds the service levels needed for this facility. Please take the time to toughly read and understand the

material ion page 45 identified as “Gartner & Uptime Institute Data Center Tier Classifications” including the links provided

to the actual documentation that will allow ABC College to classify their data center requirements and add the necessary

redundancy to meet the appropriate service levels.

Power is an important resource for any operation, but for mission-critical operations power quality is vitally important; it

must be continuous and reliable. Utilities in the United States traditionally provide power with three to four nines of

availability (99.9% = 9 hours’ annual downtime/year, 99.99% = 59 minutes’ annual downtime).

Mission-critical operations must focus on the following two strategies:

1. A good electrical & cooling system design with adequate redundancy and backup is paramount (Ex. by installing

UPS systems, by adding redundant units, dual-feeds, static-switches, backup generators, etc.).

2. Once a data-center is built it must be run, maintained and tested to ensure the reliability targets are met, and that

the design and operational goals are achieved. If the appropriate steps are taken in the design stage, running, testing

and maintaining the center will be more successful. How well the center is documented and maintained is ultimately

the deciding factor in actually achieving uptime goals.

Note:

One of the most important and overlooked items that effect the availability and sustainability of a data center during an

emergency are the People, Processes, and Procedures.




ABC College Data Center Analysis Summary Chart

For the purposes of this data center assessment Greenwood PCM used a Tier l level (Availability of 99.671% with approx.

28.8 hours of annual service disruptions) to grade the power equipment and distribution for the existing data center.

The following chart is a visual summary of the findings from ABC College, additional details of these elements are contained

in other sections of this report. Each area reviewed in the assessment is reported along with any risk identified in the

analysis.

• Items flagged as “High Risk” represent critical items that could result in the loss of data, services, damage to IT

equipment or present a life safety risk.

• “Medium Risk” items, under certain circumstances, could result in data loss, services or damage to IT equipment

but do not present an imminent threat.

• “Low Risk” items present some degree of inefficiency to data center operations but do not pose a risk to data,

services or network equipment.

• “Meets / Exceeds” requirements, represents items that are capable of meeting the power or cooling requirements

needed for this specific project.




ABC College Data Center Analysis Summary Chart

• This table rates the infrastructures capability to sustain and unplanned component failure without disrupting the

critical load.

• Survey was limited to the ABC College data center located at 86 Turner Road E. Falmouth, MA with a focus of

identifying current deficiencies and making recommendation to increase availability. (*Additional out of scope

elements were identified and noted below as a courtesy to ABC College)

N/A

N/S

O/S

R&D

O/S

O/S

O/S

N/S O/S

N/S O/S

N/S O/S

N/S O/S

N/S O/S

N/S O/S

O/S

N/S O/S

Vapor Barrier (Paints & Liners)

Walls, Ceilings, Floor, Doors & Windows (Fire & Wind Rating, Seals & Sweeps)

Ris

k L

ev

el

Medium Risk

High Risk

Not Applicable

Not Surveyed

Out of Project Scope

Additional R&D Recommended

Structural Floor Loading (Data Center & Adjacent Office,confirmed ground level concrete slab)

Meets or Exceeds

Low Risk

Flood Prevention Systems (Flood barriers, elevation, pumps, sewage check valves, floor drains, etc.)

Raised Floor Loading & Replacement Components (Floor physically unstable R&D with manufacture states tile not rated for data

center use and discontinued in 1990's also no bolted stringers make replacement / upgrading difficult)

ADA Accessibility (Openings & Ramps 12:1 Pitch) *Compliant only when elevators functioning

Structural Bracing Systems (High wind storms, hurricane and tornado reinforcement)

Lighting (Minimal 50 foot candles at 3' above finished floor, needs attention)

Seismic Bracing

Flood Prevention Systems Interior & Exterior (Fire resistant building material, fire rated walls & fire stops)

Halon 1301 Fire Supresion Sysyem (EOL)

Raised Floor Plywood Modification and 2x4 Loading Concerns & Combustion Hazard (No combustible material can be used in air

plenum)

Pre-Charge Wet Sprinkler System (Prexisting non-compliant sprinkler head clearnces)

Site Selection Natural Disasters (Hurricane, tornado, snow, lightning, floods, seismic, etc.)

Co

mm

en

ts

Site Selection Man Made Disasters (Airports, nuclear power plants, freeways, stadiums, banks, refineries, pipelines, tank farms,

railroads, etc.,)

Site Selection and Architectural Systems




N/A

N/S

O/S

R&D

Ris

k Le

vel

Medium Risk

High Risk

Not Applicable

Not Surveyed



Meets or Exceeds

Low Risk

Co

mm

en

ts

N/S O/S

O/S

O/S

O/S

Electrical Panel Schedules (None available, tracing services provide by Carousel will include detailed spreadsheet).

Computer Room Cooling AHU 1 & 2 (15 Ton Mammoth & Trane 10 Ton *Only one unit operational)

Electrical Infrastructure Systems

Utility Services (Currently have single utility source, sufficient for Tier I)

Engine Generator Plant (Existing 30kVA natural gas unit sized for life safety only)

Single Line Electrical Drawings (None available, Carousel will provide drawings specific to data center electrical distribution only)

Engine Generator Plant Capacity & Future Growth

UPS Graceful Server & Storage Shutdown (Event where UPS batteries are depleted and generator does not start. Currently no network

communications with UPS or agents installed on servers)

Standardize Redundant Rack PDU's (Convert all 120V distribution to 208V for a 7% energy efficiency gain and greated power density)

Standardize Rack Built (Streamline deployment, serviceability, availability and onsite spares)

Electrical Monitoring, Control & Alerts (Existing UPS's do not have enabled network management cards)

UPS(s) Active SLA w/ 4 Hour Onsite response or onsite spares

UPS(s) Documentation, Testing, Training and Service records.

UPS(s) Maintenance ByPass Panel

UPS(s) EPO (Emergency Power Off) Switch as required by NFPA 70, NEC 645-10 and 645-1

Engine Generator Plant Active SLA

Engine Generator Plant Documentation, Testing, Training and Service records.

Computer Room Cooling AHU 1 & 2 SLA

Computer Room Perforated Tile Air Flow Analysis / CFD Modeling (See final CFD Modeling report, color rating based on these findings)

Computer Room Tile Cut-Outs & Sweeps (See final CFD Modeling report, color rating based on these findings)

Mechanical Infrastructure Systems

UPS(s) Type, Multiple Line Interactive Units (2kVA-5kVA) (Double Conversion Units Recommended in Data Centers)

Transient Voltage Surge Suppression (None noted in electrical distribution area of data center. Pending proposal will include TVSS)

UPS(s) Battery Condition (Life Expectancy 3-5 Years *Multiple units with battery alarms will fail during power outage lasting move

than 2 ms)

Telco Main Ground Bus (No TMGB notes in data center per NEC and TIA-607-B)

Grounding (No ground grid noted bonding UPS. Batteries, Racks, Ladder Tray and Raised Floor Pedestals to TMGB)

UPS(s) Capacity, Multiple (34- Thirty Four) 2.2kW - 3kW units with a total maximum capacity of 99.9kVA. (Total IT load is 29kW,

remaining capacity stranded running at a low 88% efficiency)

UPS(s) Future Growth Potential (Based on existing UPS type and condition current units should be decommissioned)

Engine Generator Plant Monitoring & Alerts

UPS(s) Runtime (Unknown without adding and eneblining network management card. As stated above multiple units have battery

Engine Generator Plant Capacity to Support Current & Future Data Center Load Projection (Out of scope due to need to assess cooling

loads to confirm current load)

Electrical Distribution Documentation & Labeling (Some attention needed, Carousel will provide documents that should be in data

center and a centralized repository)

Lightning Protections (Multi story building, no noted antennas of lightning protection notes from ground level)

Computer Room Cooling AHU 1 & 2 Preventative Maintenance Contract & Records

Computer Room Cooling AHU 1 & 2 Total Capacity (AHU 2 off line only 15Tons / 52kW of capacity, significantly reduced due to air

mixing and ambient air temperature)

Computer Room Cooling AHU 1 & 2 Documentation, testing, training and service records.

Computer Room Cooling AHU 1 & 2 Remote monitoring and alerts

Computer Room Cooling AHU 1 & 2 Communication interface (Prevents demand fighting and facilitate round robin operation for even

wear)

Computer Room Cooling AHU 1 & 2 Humidification & Dehumidification Control




N/A

N/S

O/S

R&D

Ris

k Le

vel

Medium Risk

High Risk

Not Applicable

Not Surveyed



Meets or Exceeds

Low Risk

Co

mm

en

ts

Mechanical Infrastructure Systems

O/S

N/S O/S

N/S O/S

N/S O/S

N/S O/S

N/S O/S

N/S O/S

N/S O/S

N/S O/S

N/S O/S

Computer Room Air Flow Hot & Cold Aisles Orientation (Reduce air mixing)

Air Particulates & Contamination (No noted asbestos floor or ceiling tiles)

Fire & Smoke Detection, Protection & Annunciator Panel

Computer Room Environment Monitoring Temperature, Humidity & Leaks

Operations and Human Factors

Computer Room Rack Space Optimization. (Needs adjustmnets after raised floor replaced))

Computer Room Rack Side Panels (Increase airflow, reduce air mixing and security)

Computer Room Rack Blanking Panels (Increase airflow and Reduce air mixing)

Centralized Documentation (Blueprints, single line drawings, install & user guides, service records, maintenance contracts, SLA, etc.)

Centralized Change Control Procedures & Documentation

Wet or Precharge Fire Protection

Physical Security Site Access (Fencing, guards, security camera's, escorts, background checks, proof of insurance, bonding, etc.)

Inert Gas Fire Protection (Exiting Halon 1301 is EOL but funtioning as is)

Portable Inert Gas Fire Protection (One extinguisher at each egress)

Centralized Method of Procedures (MOP) Documentation

Centralized Emergency Operating Procedures (EOP) Documentation

Building Management and Control System (BMS)

Computer Room CRAC Cooling Capacity to Support All Data Center IT Loads, Lighting, Room Losses and Occupants

Physical Security Equipment Access (Rack level keys, keypads, biometric, card, etc.)

Environmental Monitoring & Control (Temperature, Humidity & Water Leaks)

Centralized Standard Operating and Procedure (SOP) Documentation

Physical Access & Service Clearances

Physical Security Building Access (Fencing, security camera's & video surveillance, escorts, key, keypads, biometric, card, etc.)

Physical Security Room Access (Keys, keypads, biometric, card, man cages, etc.) Project moved to Smith College facility due to

pending campus wide upgrade.




Facility Site Selection

Description

The ABC College data center is located in the ground level of the Presidents Hall Annex at 86 Turner Road, E. Falmouth,

MA 02539 (Latitude 00.318468 / Longitude -00.6356) and consists of al multistory brick faced commercial building. There’s

a nearby campus visitor parking garage and meter street parking in front of the building. There we no visible radio

antennas masts or lightning protection around the roof perimeter as viewed from the street.

Facility Site Selection Findings

The building is located approximately 5 miles from Massachusetts Bay, 5 miles from Long Island Sound and is 60’ above

sea level. Current FEMA Flood Maps indicate that the facility is not within the 100-year flood plain map. (See page 14

below).

Greenwood PCM was not asked to assess the site selection. However due to the criticality of this core data center

Greenwood PCM provided additional material and maps pertaining to the sites susceptibility to natural disasters.

The project sponsor was unable to locate the original blueprints, plot plans, documented lot line set back variance for

Mammoth Economizer, single line electrical drawings, existing generator load statistics, proximity and feasibility of

secondary utility source, existing natural gas service pathway & flow capacity, panel schedules or an asset list prior to the

onsite visit.

Google Map Town of E. Falmouth Plot Plan Map




Satellite Map

Note:

Approval to take onsite pictures of the interior and exterior data center was approved by David Drew during the first onsite

visit on 2/28/2015, satellite and flood maps are available online at multiple public sites.




Man Made Disasters

Potential risks that were not assessed include the proximity to prisons, nuclear power plants, freeways, stadiums, banks,

refineries, pipelines, tank farms, railroads, etc.

Natural Disasters

As with most areas in the United States there are known natural disaster areas that are more susceptible to outages

caused by tornados, hurricanes, flooding, wind, rain, snowfall, etc. that can disrupt data center services.

As indicated above the site selection is located approximately 5 miles from Massachusetts Bay, 15 miles from Long Island

Sound and is 69’ above sea level and has a low to moderate risk rating for natural disasters

FEMA Flood Map




Natural Disasters Maps

Please note that susceptibilities are not limited to forest fires, lightning storms, tornadoes, hurricanes, earthquakes, floods

(Ocean, rivers, brooks, stream & culverts), etc. and should include the risk to man-made disasters.




Facility Site Selection Recommendations

The site has known risk and susceptibility to snow and ice storms that caused recent power outage and services disruption.

The primary causes of the service disruption were due to the type and condition of the rack based UPS’s and the lack of a

dedicated data center generator. (These deficiencies are identified below along with recommendations to prevent similar

outages)

There are no out of scope site selection recommendations at this time other than being aware of the possible site access

concerns identified in FEMA Flood Map.




Facility and Operations

Description

The current data center consists of a 730 Sq./Ft area that includes a 136 Sq./Ft fire suppression room with sloped ceiling

that does not have sufficient height to support most IT of infrastructure equipment. There are two egresses with a key

pad access system and a 12” raised floor area that covers approximately 408 Sq./Ft with a perimeter area of poured

concrete. There is a partial drop ceiling with varying depths above the ceiling due to the sloped ceiling above and recessed

florescent lighting with limited coverage in several areas. The raised floor is used as a supply air plenum with a return air

supply to an outdoor air economizer behind rack row A via an exterior wall mounted grill.

Adjacent to the data center is an office space identifies as AG2 (John Smith) that contains 266 Sq./Ft of area that has been

identified as a potential location for additional infrastructure equipment.

There are no centralized documents to assist the onsite staff or 3rd party contractors to manage, test, change, maintain or

troubleshooting the current infrastructure.

Facility and Operations Findings

• Plot plan indicates the existing Mammoth Economizer does not have the proper 10’ setback from property line.

• No existing facility blue prints (Basic PDF insufficient to obtain construction permits, additional engineering services may be needed to alter data center or adjacent office space)

• No existing facility blue prints depicting utility feeds and pathways (Need to confirm natural gas feed pathways, pipe sizes and capacity to support new generator and existing 30kVA life & safety generator. Possibility of consolidating generators with proper sizing and input from facilities regarding existing load and any expansion plans).

• Floor loading concerns in office space should this space be used to move or expand data center.

• Existing raised floor material not rated for data center use.

• Existing raised floor is unstable in many areas no bolted stringer 4’ X 6’ section made with plywood and 2” x 4” bracing and non-standard raised floor pedestals.

• Raised floor combustible material in plenum (Plywood and 2x4 studs)

• Debris under the raised floor requires cleaning.

• Noncompliance to NEC, OSHA, NFPA, ADA or TIA standards for minimum service areas.

• Noncompliance to best practices for 4’ cold aisle and 3’ hot aisle clearances.

• Cable management and identification schemes need attention.

• Existing clean agent fire suppression system is a Halon 1301 system which is EOL (End of Life)

• Existing pre-charge wet sprinkler system has no wire guards and does not have the required 18” of vertical clearance.

• Fire stops around wall penetrations (Existing product appears to be non-compliant contractor grade foam fill)

• No portable inert gas fire extinguishers at either egress.

• No floor drains to quickly discharge water caused from sprinkler discharge of leaks from upper floors

• Insufficient light levels in several areas

• No Centralized Maintenance Records & Service Requests (UPS's, Batteries, CRAC's, Generator, etc.)

• No Centralized Change Control Procedures & Documentation

• No Centralized Standard Operating and Procedure (SOP) Documentation




• No Centralized Method of Procedures (MOP) Documentation

• No Centralized Emergency Operating Procedures (EOP) Documentation

Facilities & Operations Recommendations

• Existing Mammoth Economizer does not have the proper 10’ setback from property line. The project sponsor has indicated that St. Mary’s church approved this code violation which may now be deemed “non-compliant previously existing”. However, if the Mammoth Economizer is replaced with a generator ABC College needs to provide written approval from St. Mary’s and E. Falmouth Town Hall.

• Create a central repository of the newly created As Built drawings, single line electrical drawings and panel schedules for the data center. Reproduce multiple copies to have onsite and in a centralized document binder within the data center.

• Floor loading concerns in office space has been addressed by the project sponsor who has indicated that this area is considered ground level and has a poured concrete floor.

• Existing Tate raised floor has been confirmed to not rated for data center use at 800LBS per tile and EOL (End Of

Life) Phase one of the data center upgrade should include the full replacement of the 12” raised floor with the

correct tile type, include optional 24” bolted stringers, perforations, cut-outs sweep and under floor structure

rated for data centers. (This will also eliminate the combustible material (Plywood) used to patch floor)

• Clean under raised floor of all debris using a HEPA vacuum

• Add additional cable management including a color code scheme to assist with troubleshooting.

• Replace the Halon 1301 fire suppression system with FM-200 or Intergen and extend it into the adjacent office

space that has been allocated for two redundant 80kVA UPS’s.

• Replace fire suppression barriers with approved product.

• Add additional sprinkler heads and guards where required by LAHJ based on the 18” minimum clearances.

• Add clean agent portable fire extinguishers at each egress.

• Properly orientate all IT and electrical distribution equipment to meet all NEC, NFPA, OSHA, ADA and TIA standards

for minimum service areas requirements.

• Properly orientate IT racks based on 4’ cold aisles and 3’ hot aisle recommendations.

• Install under floor sump pump (Floor drains or pumps must have adequate capacity to drain water from the sprinkler system, air conditioner malfunctions, or spills within a reasonably short time)

• Windows in office area designated for new UPS’s, cover interior with ¾” plywood painted to match walls and insulate with rigid foam insulation. (Optional - Brick exterior similar to bricked data center windows to increase security, weather protection and aesthetics)

• Add additional lighting (Lighting in several areas does not appear to meet minimum 50 to 70 foot candles (538 to 753 lumens/m2) to meet the Occupational Safety and Health Act (OSHA) standards when measured 3 feet from the finished floor).

• Continue testing, updating and documenting maintenance records & schedules to ensure the availability levels are met, and that the design and operational goals are achieved. How well the center is documented and maintained is ultimately the deciding factor in actually achieving uptime goals and preventing human error.

• Documentation and training of qualified Invensys staff members or preferred electrical & mechanical contractors regarding the distribution, installation, specifications, operation, preventative maintenance, troubleshooting and repair of all electrical distribution & utilization equipment supplying power & cooling to the facility.




• Create, update and maintain Standard Operating Procedures (SOP) documentation that provides a how to guide to change out a CRAC (Computer Room Air Conditioner) filter or adjust the outlet air temperature on an AHU (Air Handler Units) typically task that don't require change control approval before starting.

• Create, update and maintain Method of Procedures (MOP) documentation that details maintenance procedures such as how to place a UPS in maintenance mode or load test a generator which will require change control approval before starting

• Create, update and maintain Emergency Operating Procedures (EOP) documentation detailing the step by step procedures to take during utility power loss, CRAC failures or a generator that won't start up. Not only is it key to have these documented procedures on how to deal with these problems, but employees should run through annual drills before the real-life situation arises.

Notes: Additional information regarding the Management Operations referenced above (SOP, MOP & EOP) can be found online at The Uptime Institute website which applies them to the following four categories.

1. Staffing and Organization- Referring to the proper number and qualifications of personnel comprising the operations team in the data center, as well as shift coverage and well defined roles and responsibilities which are taken in high regard by management.

2. Maintenance- Referring to the rigor of the preventative maintenance programs, housekeeping, maintenance management, service level agreements, and life-cycle planning.

3. Training- referring to personnel training programs for policies and procedures, incident response, etc., and the source of training including OJT (On Job Training), vendor delivered training, and external educational sources. The experience and competency of the staff is key to maintaining systems and components.

4. Planning, Coordination, and Management- Referring to the full scope of data center management factors including capacity planning, operational planning, policy creation and enforcement.




Existing Computer Room and Equipment Placement

Data Center & Adjacent Office Dimensions -Visio-




Data Center As Built Ladder Trays & Raised Tile -Visio-




Data Center As Built Raised Floor Perforations -Visio-




Data Center As Built Racks –Visio-




Data Center As Built All Assessed Data –Visio-




Data Center As Built Ceiling Elevations –Visio-

a




Optimized Computer Room and Equipment Placement Option 1 (Fixed 2 Post Rack) –Visio-

Optimized Computer Room and Equipment Placement Option 2 (Fixed 2 Post Rack) –Visio-




Optimized Computer Room and Equipment Placement Option 3 Perimeter Cooling –Visio-




Optimized Computer Room and Equipment Placement Option 4 Close Coupled Cooling –Visio-




Power Requirement Methodology & Caveats

Greenwood PCM method of power and cooling information collection is based on the enterprise demand load whenever

possible. Typically, this data is gathered from uninterruptable power supplies (UPS) and/or power distribution units

(Metered PDU’s) deployed in the facility. The power distribution system’s electrical load readings are recorded and used

as the baseline electrical load from which thermal cooling load calculations can also be made.

Maximum Wattage or Voltage/Amperage as represented by the manufacturer and listed on each device’s data plate is

accepted as the baseline for calculating theoretical potential cooling loads generated by a network device when trended

Demand Load data is not available which can lead to over sizing by 20% - 30%. The potential cooling load is expressed in

British Thermal Units produced per hour of operation (BTU/Hr.). This is calculated using the conversion factor of 3.414

BTU/Hr. per Watt of electrical energy consumed. For example, if a network device is specified by the manufacturer to

consume 100 watts at its maximum operational output, the corresponding cooling load of that device would be 341.4

BTU/Hrs.

Real time and demand load measurements represent only a snapshot in time. The instantaneous real time measurement

of electrical and cooling loads may be recorded during a down cycle or other anomaly causing an unrealistic view of power

consumption and cooling capacity requirements. Additionally, variations in real-time power usage preclude the

development of an accurate baseline unless usage data is recorded over an extended period of time and the raw data

subjected to rigorous statistical analysis.

For this assessment Greenwood PCM used point in time Amprobe reading on the MDP Panel (Basement) Panel K, (Data

Center) and Panel CR2 (Adjacent Office) as well as a full power quality assessment of the Panel K using a Fluke 435 Power

Quality Analyzer (See pages 50 - 59 of power quality findings)

Most importantly during the onsite visit to obtain these Amprobe reading and power quality statistics on 3/28/2015 the

elevators, building air conditioners and CRAC units were not active. Therefore, the peak load statistics and expansion

capacity for the 1,000A, 208V, MDP could not be accurately obtained to confirm sufficient capacity and breaker positions

to support the proposed solution at its maximum rating. (The day one IT load of 26kVA – 29kVA will not change for some

time which the existing utility feeds can support. However additional R&D and peak load statistic’s need to be obtained

from the ABC College facility team using historic billing information or metering during peak usage during summer months)




Fluke 435 Three-Phase Power Quality Analyzer Features

The project sponsor indicated several concerns with the existing power quality therefore Greenwood PCM included the

use of a Fluke 435 three-phase power quality analyzer to help identify, locate, predict, prevent and troubleshoot

problems in three phase power distribution systems.

• AutoTrend, which records everything on the screen automatically

• Supports GPS time synchronization

• System-Monitor, a quick diagnosis of system performance against user defined limits

• Logger Function with multi-parameter logging

• Measure all three phases and neutral: With included four current probes

• Mains Signaling, measures interference from ripple control signals at specific frequencies

• View graphs and generate reports: With included analysis software

• Logger function: Configure for any test condition with memory for over 400 parameters at user defined intervals.

• Mains signaling: Measure interference from ripple control signals at specific frequencies

• Inrush mode: For troubleshooting nuisance circuit breaker tripping.

• High safety rating: 600 V CAT IV/1000 V CAT III rated for use at the service entrance

Ideal 61-958 Inductive Circuit Tracer Features

The project sponsor was unable to locate the original blueprints, single line electrical drawings, panel schedules or an

asset list prior to the onsite visit. Greenwood PCM used and Ideal Inductive Circuit Tracer to identify the rack distribution

breakers and confirm the source paths from the main distribution panel.

• Numeric value and audible signal provide quick and easy-to-

understand tracing feedback

• Receiver display rotates automatically for easy viewing

• Identifies breakers and fuses

• Pinpoints opens and shorts

• Traces wires behind walls

• "0-99" display of signal strength

• Audible intensifies in tone/pitch

• Works on energized or de-energized circuits

• Four modes of sensitivity - search high, search low, trace, and

breaker

• Peak bar graph

• Can be used on circuits 0-600V AC/DC




Data Center Power

Description

The underground utility service lateral is located on the rear south side of the building entering into basement electrical

room at 2,400V. The power then enters a service disconnect and feed the primary side of 2,400V Delta transformer at

72A. The secondary side of the transformer is rated at 208Y/120V at 883A and feeds a 1,000A MDP (Main Distribution

Panel)

Within the MDP are the breakers that feed Panel K (150A) in the data center, Panel CR2 (200A) in the adjacent office

space, Mammoth Economizer (175A) as well at the elevators, facility A/C, 30kVA generator and other electrical distribution

panels throughout the facility. Review the existing single line drawing on page 34, MDP schedules & amperage reading

on pages 35 – 37 and the last two paragraphs in the Power Requirement Methodology & Caveats section above.

Notice:

Please note that the single line electrical drawings are specific to the data center and do not include the entire facility.

Data Center Power Finding

As stated above under the Facility Site Selection Findings. The project sponsor was unable to locate the original blueprints,

plot plans, documented lot line set back variance for Mammoth Economizer, single line electrical drawings, existing

generator load statistics, proximity and feasibility of secondary utility source, existing natural gas service pathway & flow

capacity, panel schedules or an asset list prior to the onsite visit. Fortunately, Greenwood PCM included circuit tracing

services in the original PSA (Professional Service Agreements) to provide this missing data. (See new single line drawing

created by Greenwood PCM on page 34 and the electrical panel schedules on pages 35 -37)

• No existing facility blue prints or single line drawings depicting electrical utility feeds or loads.

• No existing electrical panel scheduled identifying specific racks or outlets power from panel.

• Facility electricians are unable to confirm availability of separate electrical utility source needed for Tier III availability levers.

• No existing onsite generator dedicated to IT equipment & services (Runtime limited to UPS battery sizing)

• Existing power quality concerns

• No existing projections regarding future growth or consolidation over the next 3-5 years

• Existing UPS’s are line interactive instead of double conversion which should not be used for data center servers, disk arrays, SAN’s or network PoE equipment. (Line Interactive UPS’s are 84% efficient with a 10mS reaction time, Double Online Conversion UPS’s recommended for data centers that can be 99% efficient with no delay when transitioning to battery backup)

• No single line electrical drawing exists except out dated blue prints dated 1984.

• Incomplete / outdates electrical panel schedules with feeder and source information.

• No space in basement for electrical expansion… may already be non-compliant with new standards.

• No TVSS (Transient Voltage Surge Suppression)

• No TMGB (Telco main Ground Bus) for grounding/bonding of UPS, CRAC’s, Racks, Raised Floor, Ladder Tray

• UPS multiple battery alarms (Runtime unknown without network management card… likely 0 seconds -5 minutes

• UPS no network management cards, SMTP/SNMP alerts




• UPS no network management cards for graceful server shutdown

• No EPO Switch

• No maintenance bypass switch or external bypass device to bypass defective UPS

• No standardized rack based PDU’s for planned growth and onsite spares

• No branch circuit monitoring or rack based PDU monitoring of loads.

Data Center Power Recommendations

• Review the Gartner & Uptime Institute Data Center Tier Classifications documentation outlined on page 45 to

understand the basic technology needed for the four availability levels.

• Confirm the SLA requirements for all business/service units and assign the appropriate Gartner Tier level, revise

power and cooling designs to match.

• Confirm future growth plans for existing data center and any initiative for virtualization or cloud computing to

reduce the existing power & cooling requirements. (Project sponsor was unable to provide future growth plans for

the data center other than the need to add additional breaker positions for additional racks. Current IT load has

been measures at 26kVA – 29kVA which initially indicated the UPS sizing to be 40kVA. During the review of this

information ABC College and John Hastings reached a consensus to size a redundant solution at 80kVA)

• Create central repository of new as built drawings, single line electrical drawing.

• Review power quality report findings on pages 50 – 59 that indicate no power anomalies during the time the data

was collected.

• Replace all existing rack & tower based line interactive UPS with a centralized double online conversion UPS

• Consider a N+1 or 2N UPS solution to meet the availability levers required by ABC College

• Install two EPO switches (One at each egress as required by National Fire Protection Association - NFPA 70: This US

document, also called the National Electrical Code (NEC), specifies in clauses 645-10 and 645-11 that UPS’s above

750 VA installed within information technology rooms must be provided with “a means to disconnect power to all

electronic equipment in the information technology equipment room.”)

• Install natural gas generator and ATS dedicated to data center power, cooling, lighting and occupancy. (Diesel

generator option has addition site concerns with fuel storage, fuel delivery & SLA, exhaust fumes (All solutions will

require engineering and material to extent exhaust above building roof line), noise levels and site restrictions. ABC

College facilities were unable to provide load statistics for the existing 30kVA Onan generator which is designated

for building life safety only.)

• Install a centralized Telco Main Ground Bus (TMGB) and bond all UPS & Battery Cabinets, CRAC’s, Ladder Tray and

Raised Floor Pedestals using #6 AWG copper. One rack in each row should be bonded with a #6 AWG with #12 AWG

jumpers between racks. (Jumpers are normally supplied by rack manufacture)

• Install a Transient Voltage Surge Suppressor (TVSS) to protect the UPS distribution panel.

• Include a network management card and enhanced battery monitoring capabilities for alerts, trending data and

graceful server shutdown.

• Include an external maintenance by pass panel for future upgrades of replacement

• Include branch circuit monitoring and alerts in power distribution panels or rack based PDU’s.

• Update the existing electrical panel schedules using the Panel Schedules created by Greenwood PCM on pages 35 -

37 and update all new distribution panels based on these templates for the new solution.




• Label all utilization outlets and devices with the panel name and circuit breaker number. (Ex. An outlet labeled RPP1-

A1-18&20 would indicate the outlet or device is feed from a RPP (Remote Power Panel), Feed from UPS-A, Panel 1

and the double pole breaker at position 18 &20 controls/protects the outlet. This labeling scheme should be

extended to the individual PDU’s in each rack).

• Label UPS and CRAC units with electrical distribution source location and breaker.

• Include extended warranty to include preventative maintenance, onsite 4-hour response for no less than 3 years.

• Remove all unused power and communication wiring and raceways per NEC regulations. (Enhances airflow,

serviceability and troubleshooting)

• Consolidating stray servers into racks and remove workstation bench to facilitate pending implementation or

convert to small NOC (Service personnel need adequate service areas without obstructions to facilitate the service

need to update data center and minimize the risk of disturbing services).

• Standardize rack builds (Using common rack components allows for economies of scale, parts spares and

serviceability)

• Standardize on redundant metered rack PDU’s (Proposal includes (redundant) 30A, 3 Phase 208V units rated for

8.6kW (See Rack Load Summary on page 38 - 39 depicting the existing high, low and average loads per rack)

• Use short two-foot C-13/C-14 power cables to increase rear airflow and increase cable management.




Existing Data Center Single Line Drawings –Visio-




Existing Panel Schedules & Load Statistics

MDP Panel Schedule & Load Statistic




Panel K Schedule& Load Statistic




Panel CR2 Schedule & Load Statistic




Existing UPS by Rack Location Including Free Standing Tower Units




Existing Rack Load Summary

Fluke Power Quality Meter Graph Maximum kVA & kW Loads

Panel K, 4-hour monitoring duration, average26kVA with one peak anomaly to 30kVA that may be from a device feed from

Panel K not related to the data center IT loads. (See full power quality report on pages 50-59)




Data Center Cooling

Description

The existing facility CRAC units consist of 1- 10 Ton Trane down flow CRAC units and 1- 15 Ton Mammoth Economized

AHU’s for a total cooling capacity of 25 Tons as well as a wall mounted humidifier.

Data Center Cooling Findings

The combined cooling capacity of the two existing facility CRAC units is 25 Tons (88kW). Based on the following conversion

Tons to kW = (Tons X 3.516) the 1- 10 Ton Trane unit is rated for 35kW and the Mammoth Economizer is rated for 53kW

depending on the ambient air temperature.

The project sponsors have indicated that the Mammoth Economizer often has difficulty maintaining proper cooling during

summer months. And both the Trane CRAC unit and wall mounted humidifier circuit breaker were turned off during the

onsite assessment and their operation condition unknown.

Greenwood PCM believes that the original 10 Ton Trane CRAC unit did not have sufficient capacity or proper airflow to

cool the data center and the Mammoth Economizer was added to increase the overall cooling capacity without the use of

CFD analysis to confirm if original condition was caused by airflow pathways or capacity. In addition, the mix of CRAC

manufactures, lack of blanking panels, floor plenum supply air distribution, raised floor height and raised floor obstructions

are causing substantial losses in efficiency, air mixing and demand fighting between units if they were both activated to

increase capacity due to incompatible communication protocols.

• Existing Mammoth Economizer rated for 15 tons (Not functioning optimally possible due to ambient temperatures and airflow which was confirmed using CFD modeling))

• Existing Trane AHU rated for 10 tons not operational (Room AHU 20A breaker in Panel K OFF)

• Existing wall mounted humidifier not operational

• Insufficient CRAC capacity to support future maximum load potential of 80kVA UPS’s

• No rack blanking panels (Air mixing)

• No rack front or rear door security concerns

• No racks side panels missing racks A1 & A2 (Air mixing, cross wiring, security and human error concerns)

• No standardized rack based cable management of identification. (Air flow blockage and troubleshooting and human error concerns)

• Low humidity (Data logger results below indicate 17% relative humidity during data collection period, high risk of ESD (Electro Static Discharge))




Hobo Data Logger Temperature & Humidity Graphs

Note:

Please review CFD Modeling report on pages 56-69 for details regarding the existing as built cooling findings as well as the

results of the CFD modeling for the proposed close coupled cooling solution that confirms N+1 protection that allows for

concurrent maintenance.




Data Center Cooling Recommendations

• Review separate CFD Modeling Findings on pages 56 - 69 of as built findings, perimeter down flow finding and the

proposed close coupled solution results confirming N+1 redundancy.

• Review and apply findings of the first two bullet items under the power recommendations on page 32 above to

cooling requirements.

• After confirming the current and future cooling requirements from the bullet item above, work with the facilities

staff to confirm the appropriate preventative maintenance schedules and onsite SLA for this equipment meets the

designated availability levels.

• Consider standardizing on one CRAC manufacture and using the optional communication interfaces when upgrading

or replacing the existing CRAC units to avoid demand fighting. (Interface will allow for 1+N protection, round robin

exercise cycles to even wear and confirm run status on all units)

• Add humidification control within CRAC solution or separate dedicated wall mounted unit.

• Add additional environment probes and alerts to confirm rack level input temperatures and overall humidity levels.

• Add additional smoke & leak detection in CRAC area and airflow.

• Add front blanking plates to all racks to reduce air mixing and streamline front to rear air flow.

• Add rack side panels to all racks to reduce air mixing, streamline front to rear air flow and increase security.

• Add rack based cable management to increase air flow.

• Add raised floor tile cut out sweeps to reduce air leakage and mixing. (If perimeter CRAC units are installed)

• Move raised floor perforated tiles to cold aisles and adds solid tiles to hot aisles. (Current raised floor product is EOL

(End of Life) and not rated for data center use. Proposal includes costs to upgrade existing raised floor with proper

product and service to retrofit)




Budgetary Proposal

The budgetary cost estimate below should be used for budgetary purposes only and does not include taxes, fees or

freight unless otherwise noted. Firm, fixed pricing cannot be provided until the project’s scope and scale have been

clearly defined and detailed design, engineering, and construction documents have been created. This budget is not an

executable proposal.

The purpose of cost estimating, at this stage in the project, is to aid in determining the cost impact of decisions

regarding business objectives, systems availability, and power/cooling density. These factors are utilized in defining the

project’s scope and scale. Upon assembly of this information, design, engineering, and construction documents can be

created. Additionally, all resulting proposals should be leveled and compared for accuracy, completeness, and

qualification. Upon completion of these steps, a complete and accurate accounting of the costs associated with the

project can be established.

This budgetary estimate contains various cost data for the project as described herein. The costs were derived without

benefit of detailed engineering documents and/or contractor / sub‐contractor quotes and are based upon Greenwood

PCM & XYZ (XYZ is a wholly owned subsidiary of HP) experience with similar scope and scale projects.




UPS Potential ROI

The proposed UPS’s include an ESS (Energy Saver System) that increases the UPS efficiency from 94% to 99% efficiency

and would yield an annual energy cost savings of $13,385.03 based on a 32kW and 64kW load and .12 cents per kW/Hr.

cost. (Please note that when the redundant solution is selected only one UPS will be in ESS mode (99%)and the second

UPS will be running at 94% efficiency)

Link: http://powerquality.eaton.com/calculator/

http://powerquality.eaton.com/calculator/




Gartner & Uptime Institute Data Center Tier Classifications

Gartner and the Uptime Institute classify data centers into four tiers as shown in the table below. The

Telecommunications Industry Association (TIA) has further refined these recommendations into TIA-942 specifications

for structural, architectural, mechanical, and electrical requirements of a modern data center.

Assuming that all critical IT equipment is in optimal condition, serviced regularly and has an active onsite service

contract that meets the minimum onsite response times, the current tier level of ABC College data center would be

classified as a Tier I (Basic) that can expect to have up to 28.8 hours of services disruptions per year.

Tier Level Description

Estimated

Annual

Down Time

Site

Availability

Tier I:

Basic

Single path for power and cooling distribution

No redundant components 28.8 hours 99.671%

Tier II:

Some Redundant

Components

Single path for power and cooling distribution

Redundant components 22 hours 99.741%

Tier III:

Concurrently

Maintainable

Multiple power and cooling distribution paths, but only

one path active

Redundant components

Concurrently maintainable

1.6 hours 99.982%

Tier IV:

Fault Tolerant

Multiple active power and cooling distribution paths

Redundant components

Fault tolerant

0.4 hours 99.995%

Links:

http://searchdatacenter.techtarget.com/definition/Uptime-data-center-tier-standards

http://www.google.com/url?sa=t&source=web&cd=3&ved=0CDoQFjAC&url=http%3A%2F%2Fsearchdatacenter.techtarg

et.com%2Fdefinition%2FUptime-data-center-tier-

standards&ei=sEeTTa_pNYnC0QHm7uzMBw&usg=AFQjCNHW_auNIv_Hljv4YIGlmm43WxT0YA

http://searchdatacenter.techtarget.com/definition/Uptime-data-center-tier-standards

http://www.google.com/url?sa=t&source=web&cd=3&ved=0CDoQFjAC&url=http%3A%2F%2Fsearchdatacenter.techtarget.com%2Fdefinition%2FUptime-data-center-tier-standards&ei=sEeTTa_pNYnC0QHm7uzMBw&usg=AFQjCNHW_auNIv_Hljv4YIGlmm43WxT0YA






Fluke Power Quality Report Finding

As noted in the Power Requirement Methodology & Caveats section on page 29. The Amprobe reading and power quality

statistics collected on 2/29/2015 do not include the intermittent loads from the elevators, building air conditioners and

CRAC units which were not active during the 4-hour collection period.

As noted in the first page of the summary report pasted below, there were no noted power anomalies recorded from

2/29/2015 at 8:16 AM – 2/29/2015 at 12:16 PM




Computational Fluid Dynamics Findings

Through the use of Computational Fluid Dynamics (CFD) simulation software we have developed and tested an As-Built

model of the current ABC College Data Center as well as two potential solution configurations. The use of CFD on this

project allowed us to identify several issues with the current design of the facility which was reviewed via our web

conferences, documented in video/audio form, and reviewed below.

CFD Results As-Built Room Analysis

The facility’s IT infrastructure is split primarily between two rows of enclosures. Row A contains a total of six enclosures

whereas Row B contains a total of 10 enclosures. After gathering extensive detail on the IT hardware and location

throughout the facility a model of the facility was created in the CFD software suite. The resulting model can be seen

below:

Figure 1: Overhead 3-D View with Orientation – Blue Corresponds to Cold Aisle, Red Corresponds to Hot Aisle. With respect to the

tables in the room blue corresponds to the front and red the rear.




As can be seen in the below figures a great attention to detail was provided with respect to the U placement and

orientation of hardware. By maintaining identical placement of the hardware in our simulation we were able to

duplicate the expected airflow profile of the room.

Figure 2: Row A Hardware – Racks 1-6

Figure 3: Row B Hardware – Racks 1-10




There are several concerns that arouse from the visual inspection of the rows of racks. There is currently no blanking

being utilized and there are 4-Post shelves supporting Dell monitors throughout the rows. This poses an issue due to air

migration between the cold aisles and hot aisles which adversely affects the temperature of the cold aisle. With the current

configuration the cold aisle temperatures are as shown in the below figures.

Figure 4: Row A Racks 1-6 Server Inlet Temperatures




Figure 5: Row B Racks 1-10 Server Inlet Temperatures

After analyzing the airflow streamlines associated with the existing facility we have identified some issues directly

impacting the cold aisle. There is an extensive amount of exhaust air being drawn through, around the side, and over the

top of the racks. The following figures show the airflow passing into the cold aisle from the hot aisle. In these cases, the

opposite row is hidden and both the under-floor and above floor streamlines are visible. For reference the AC return of

the room is located on the wall directly below rack A6 which is the left most rack.




Figure 6: Row A Server Inlet Airflow Streamlines Colored by Temperature – Top Down

Figure 7: Row B Server Inlet Airflow Streamlines Colored by Temperature – Top Down




Figure 8: Rows A&B Server Inlet Airflow Streamlines Colored by Temperature – Top Down

The overall room temperatures are demonstrated by the below temperature result plane which is estimating temperature

at a height of 4ft above the floor. The room temperatures can be seen to be in the lower to mid 70F’s as a whole. There

are areas related to the hot aisles that show their associated heat plumes as expected however the perforated tile’s placed

behind row B are problematic. Since that is considered a hot aisle there should be minimum to no cold air in that space.

Due to the placement of the 2-Post rack, Wall Mount Rack, and 4-Post racks in that space there is some requirement for

cold air but nothing more than what a single well placed perforated floor tile can provide for once the hardware is adjusted

in terms of placement.




In addition to the figure of the temperature result plane you will also find information on the estimated airflow from the

perforated floor. Due to the number of tiles and placement of some there are areas with very minimum airflow passing

through. The underfloor has almost a double diamond shape due to the concrete pads which are present in parts of the

room. This leaves a relatively limited area where air can pass between the two spaces however this does not seem to be

a large issue in the down flow CRAC models I’ve run. The current Mammoth supply placement is approximately 2ft to the

right and down from rack A1 and is represented by the grey ducting.

Figure 9 – Temperature Result Plane @ 4ft




Figure 10 – Perforated Tile Airflow (CFM)

Figure 11 – Perforated Tile Airflow (CFM) Detailed View w/ Restricted Legend Boundaries




CFD Solution Design Options

In order to determine the most appropriate solution for the data center that will meet today’s level of equipment load as

well as ensures the ability to accommodate future growth and maintain a high level of availability two solutions were

reviewed. The first solution is a down flow CRAC solution and the second solution is a close coupled cooling solution. These

solutions will be explored further in the below figures and comments.

CFD Findings Down Flow CRAC Solution Analysis

With the down flow CRAC solution we were able to obtain a redundant cooling configuration with two down flow 25 Ton

CRAC units. With the placement of the perforated tiles only in the cold aisle we eliminate the issues found from excessive

floor tiles and we can maintain a very high average flow rate for the row of enclosures. At the current load the temperature

result plane appears very even although higher than the second proposed solution which utilizes close coupled cooling.

The results of the existing day one load and day X loading can be seen below along with the new room CRAC placement,

perforated tile placement, and rack placement. With respect to airflow management all open areas of the racks were

blanked off so that there is minimal risk of bypass air passing through the enclosures. Though the racks are blanked without

the use of a Cold Aisle Containment System (CACS) there will be air drawn from the hot aisles and into the cold aisle. If

this solution is preferred, then CACS is highly recommended as it will be required for the units to be able to provide

effective cooling as the equipment load in the room increases. The monitors in the racks were removed and 1U KMM

devices can be implemented in the enclosures to allow for console use.

Figure 12 – CRAC Solution Temperature Result Plane (Existing Load)




Figure 13 – CRAC Solution Perforated Tile Airflow (CFM)

Figure 14 – CRAC Solution Cold Aisle Temperature (Existing Load)




Figure 15 – CRAC Solution Temperature Result Plane at Simulated 80% load

Figure 16 – CRAC Solution Temperature Result Plane at Simulated 80% load




Figure 17 – CRAC Solution Cold Aisle at 80% loading w/ 1 Failed CRAC

Figure 18 – CRAC Solution Temperature Result Plane at 80% loading w/ 1 Failed CRAC & CACS




Figure 19 – CRAC Solution Cold Aisle at 80% loading w/ 1 Failed CRAC & CACS




CFD Results Close Coupled Cooling Analysis

Close coupled cooling units are known throughout the industry as strong players for non-raised floor environments and

high density deployments. In this simulation four close coupled cooling units were placed in the standard Hot / Cold aisle

orientation of the previous design however now the cooling units are placed within Row A and B to provide the close

coupled cooling. Below you can see the simulated results of the units operating at day one loads, at 80% loading, and in

failure scenarios.

Figure 20 – Close Coupled Temperature Result Plane

Figure 21 – Close Coupled Row A Inflow Temperature




Figure 22 – Close Coupled Temperature Result Plane @ 80% Load

Figure 23 – Close Coupled Temperature Result Plane @ 80% Load w/ One Unit Failed




Onsite Pictures & Reference Material

Raised Floor Debris Raised Floor Plywood Modifications

Raised Floor Debris Above Ceiling Obstructions & Abandon Cables

Fluke Power Quality Meter In Use Ideal Inductive Circuit Tracer




Existing Pre-Charge Dry Sprinkler Compressor Existing Non-Compliant Sprinkler Head (18”)

Exterior Property Line 10’ Setback Concerns Exterior Property Line 10’ Setback Concerns

Basement Main Distribution Panel (MDP) Basement Main Distribution Panel (MDP)




Existing 30kVA Life Safety Generator Existing 30kVA Life Safety Generator

Basement Electrical Room Transformer Basement 1000A Disconnect (Makeshift Lock Out)

Sample TMGB (Telco Main Ground Bus)




NEC EPO Requirement

National Fire Protection Association - NFPA 70: This US document, also called the National Electrical Code (NEC), specifies

in clauses 645-10 and 645-11 that UPS’s above 750 VA installed within information technology rooms must be provided

with “a means to disconnect power to all electronic equipment in the information technology equipment room.” This

includes all UPS ac and dc supply and output circuits. The disconnecting means shall also disconnect the battery from its

load, disconnect the power to all dedicated HVAC systems serving the room, and causes all required fire / smoke dampers

to close. Although the terms “emergency power off” or “EPO” are not used in the NEC, this code further requires that the

control for these disconnecting means shall be grouped, identified, and readily accessible at principal exit doors.

Raised Floor Weight Capacities

Three load types

1. Static Loads - Stationary loading of an object (lbs./sq. ft.)

2. Concentrated Load - The weight transferred directly to the floor from a wheel/caster or stabilizer foot (lbs./sq. in)

3. Rolling (Dynamic) Loads - Moving Loads the load placed on the floor as the object is pushed through the data center

into its final location (lbs./Tile)

Raised Floor Grounding Notes

The bolted-stringer raised floor is considered by the National Electrical Code (NEC) to be electrically bonded to itself. This

type of raised floor need only be interconnected to the building steel, electrical safety ground, and any other metallic

structure that penetrates the perimeter of the floor to meet the required specifications.

A raised-floor structure that does not have bolted stringers must be circled with a bonding conductor connected to the

top cap area of the pedestals, routed from pedestal to pedestal, in as short a route as possible. According to the NEC, the

conductor must be bare copper wire not smaller than No. 6 AWG.




Original Generator Sizing Worksheet

Item Data Required Calculations Value Value # Subtotals

Power Requirement - ELECTRICAL

Total Critical IT Load Total VA Rating All IT

Equipment (Total VA x 0.67)/1000 35000 1.00 23.45

Additional Load Name Plate

Rating

Total VA Rating (Life Safety,

Security, Monitoring) (Total VA x 0.67)/1000 6000 2.00 4.02

Future Load Total VA Rating All Future IT

Equipment (Total VA x 0.67)/1000 35000 2.00 23.45

Peak Power Draw Variations Total Steady State Critical

Load

(Value #1 + #2 + #3) x

1.05 4.00 53.47

UPS Inefficiency & Battery

Charging Actual Load Plus Future Load

(Value #1 + #2 + #3) x

0.32 5.00 16.29

Lighting Total Sq./Ft of Data Canter 0.002 x Sq./Ft 730 6.00 1.46

Total Power to Support

Electrical Demands

Total From Value #4, #5 and

#6 Value #4 + #5 + #6 7.00 71.22

Power Requirement - COOLING

Total Power to Support Cooling

Demand Total from Value #7

Chiller System (Y or

N), Value #7 x 0.7 Y 8.00 49.85

Total Power to Support Cooling

Demand Total from Value #7

DX System (Y or N),

Value #7 x 1.0 N 8.00 0

Total Power Requirement

Total Power to Support Electrical

& Cooling Demands Total from Value #7 & #8 Values # 7 + #8 9.00 121.07

Size of Electrical Service Estimate

Requirements to Meet NEC and

Other Regulations Total from Value # 9 Value #9 x 1.25 10.00 151.34

Three Phase A/C Voltage Provide

by Electrical Service A/C Voltage 208 11.00 208

Electrical Service Amperage

Required from Utility Company

Total from Value #10 and A/C

Voltage Value #11

(Value #10 x 1000)/

(Value #11 x 1.73) 420.59

Size of Standby Generator

Critical IT Load Requiring

Generator Backup Total from Value #7 Value #7 x 1.3 12.00 92.59

Cooling Load Requiring

Generator Backup Total from Value #8 Value #8 x 1.5 13.00 74.78

Approximate Size of Generator

Needed in kW Total from Value #12 and #13 Value #12 + #13 167.37

SAMPLE ABC College Data Center Assessment & Recommendations

Documents

Transcript of SAMPLE ABC College Data Center Assessment & Recommendations