Blue Waters and CloudsIntense Computing at the Petascale and Beyond

William Kramer, Thom Dunning, Marc Snir,

William Gropp, Wen-mei Hwu

Bernie Acs.Cristina Beldica, Brett Bode, Robert Fiedler,

Scott Lathrop, Mike ShowermanNational Center for Supercomputing Applications, Department of Chemistry, Department of Computer

Science, and Department of Electrical & Computer Engineering

Molecular Science Weather & Climate Forecasting

Earth ScienceAstronomy Health

Sustained Petascale computing will enable advances in abroad range of science and engineering disciplines:

Astrophysics

Life Science Materials

2

Blue Waters Project Components

Blue Waters Base System – Processors, Memory, Interconnect, On-line Storage, System Software, Programming Environment

Value added Software – Collaborations

Value added hardware and software

Petascale Application Collaboration Team Support

Petascale Applications (Computing Resource Allocations)

Outstanding User and Production SupportWAN connections, Consulting, System Management, Security,

Operations, …

Petascale Computing Facility

PetascaleEducation,

Industry and

Outreach

Great Lakes Consortium

for Petascale

Computing

3

NSF Petascale Computing Resource

Allocation (PRAC) AwardeesPIs Field Institutions

Schulten Bio-molecular Dynamics Illinois

Sugar Quantum

Chromodynamics

UC-Santa Barbara

O’Shea Early galaxy formation MSU

Nagamine Cosmology UNLV

Bartlett Parallel language,

Chemistry

U. FL

Bisset, Brown, Roberts Social networks,

Contagion

VA Tech, CMU, Research

Triangle Inst.

Yeung Turbulent flows GA Tech.

Zhang Materials science Wm. & Mary

Wilhelmson Tornadoes Illinois

NSF Petascale Computing Resource

Allocation (PRAC) Awardees(Cont’d)

PIs Field Institutions

Jordan Geophysics U. So. CA

Lamm Chemistry IA St. U.

Woodward Stellar hydrodynamics U. of MN

Campanelli General relativity,

compact binaries

Rochester Inst. Tech.

Stan, Kirtman, Large,

Randall

Climate COLA (MD), U. Miami,

UCAR, CO St. U.

Savrasov, Haule Materials science UC-Davis, Rutgers

Schnetter Gamma-ray bursts LSU

Tagkopoulos Evolution Princeton

Wang Geophysics U. of WY

Testing Hypotheses About Climate Prediction

• Hypotheses

• The large errors in current-generation climate models are associated with fundamentally flawed assumptions in the parameterizations of cloud processes and ocean eddy mixing processes

• Climate predictability, a synthetic quality entirely associated with a given model, increases with increasing model resolution by virtue of the changing representation of atmospheric and oceanic noise

• Target Problems

• Annual Cycle Experiment using the Co. St. U. Global Cloud-Resolving Atmospheric General Circulation Model

• Test if annual cycle of quantities such as the precipitation and surface temperatures are more accurately reproduced when both cloud processes and the ocean meso-scale are resolved and coupled

Stan, Kirtman, Large, Randall

Geographic Distribution of PRACs Leaders

From Chip to Entire Integrated System

8

Chip

Quad Chip MCM

Rack/Building Block

Blue Waters System

PCF

On-line Storage

Near-line Storage

Color indicates relative

amount of public information

multiple MCMs

Blue Waters Computing System

9

System Attribute Ranger Blue Waters

Vendor Sun IBMProcessor AMD Barcelona IBM Power7Peak Performance (PF) 0.579 >10

Sustained Performance (PF) <0.05 >1Number of Cores/Chip 4 8Number of Processor Cores 62,976 >300,000Amount of Memory (TB) 123 >1Interconnect Bisection BW (TB/s) ~4Amount of Disk Storage (PB) 1.73 18I/O Aggregate BW (TB/s) ? 1.5Amount of Archival Storage (PB) 2.5 (20) >500External Bandwidth (Gbps) 10 100-400

17>20

2~3.5

>8>>10

>10

>200>10

10

POWER7 Processor Chip

• 567mm2 Technology: 45nm lithography, Cu, SOI, eDRAM

• 1.2B transistors

• Equivalent function of 2.7B due to eDRAMefficiency

• Eight processor cores

• 12 execution units per core

• 4 Way SMT per core

• 32 Threads per chip

• 256KB L2 per core

• 32MB on chip eDRAM shared L3

• Dual DDR3 Memory Controllers

• 100GB/s Memory bandwidth per chip sustained

• Scalability up to 32 Sockets

• 360GB/s SMP bandwidth/chip

• 20,000 coherent operations in flight

• Advanced pre-fetching for Data and Instruction

• Binary Compatibility with POWER6 * Statements regarding SMP servers do not imply that IBM will introduce a system with this capability.

Feeds and Speeds per QCM

1 TF/s QCM

• 32 cores

• 8 Flop/cycle per core

• 4 threads per core max

• 3.5 – 4 GHz

• 32 MB L3

• 512 GB/s memory BW

(0.5 Byte/flop)

• 800 W (0.8 W/flop)

1.1 TB/s Hub Chip

• 192 GB/s Host

Connection

• 336 GB/s to 7 other local

nodes

• 240 GB/s to local-remote

nodes

• 320 GB/s to remote

nodes

• 40 GB/s to general

purpose I/O

11

Caches• Low latency L1 (32KB) and L2 (256KB) dedicated caches per core

• ~45x lower latency than memory

• 32MB shared L3 cache

• ~3x lower latency than memory

• Automatically migrates per-core private working set footprints (up to 4MB) to fast

local region per core at ~15x lower latency than memory

• Automatically clones shared data to multiple per core private regions

• Enables subset of cores to utilize entire L3 when remaining cores are not using it

12

Cache Level Capacity Type Policy Comment

L1 Data 32 KB Fast SRAM Store-thru Local thread storage update

Private L2 256KB Fast SRAM Store-In Coherency maintained throughout system

Fast L3 “Private”

Up to 4 MB eDRAM Partial Victim Reduced latency & power consumption

Shared L3 32MB eDRAM Adaptive Coherency maintained throughout system

13

Two Level Interconnect SuperNode 1

No

de

1

No

de

2

No

de

3

No

de

4

No

de

1

No

de

2

No

de

3

No

de

4

Node 1

Node 2

Node 3

Node 4

Node 1

Node 2

Node 3

Node 4

1st level: L-Links, 24 or 6 GB/s

Connect 4 drawers together to

form a SuperNode

Copper or Optical Cable

2nd level: D-Links, 10 GB/s

Optical Cable

Connects SuperNodes to

all other SuperNodes

Up to 512 SuperNodes

fully connected

SuperNode 2

SuperNode 5

SuperNode 7

Imaginations unbound

High-level Schedule 2006-2010• March 4, 2010 Substantial completion

• 88,000 GSF over two stories—45’ tall

• 30,000+ GSF of raised floor

• LEED Gold/Platinum + PUE ~1.02 to 1.20

projected

• Free cooling (On site cooling towers)

used 70% of the year

• Higher operating temperature in the

computer room

• Initially capable of 24 MW of power

• Substantial security: biometrics, cable beam

barricade

• 300 gigabit external connectivity

• Five acre site allows room for facility

expansion

What does this have to do with Clouds

• Many definitions of what ―cloud computing‖ means

• I like the explanations in Above the Clouds: A Berkeley View of

Cloud Computing for a start

• What a cloud means depends on the viewer’s role

• Cloud providers range from bare metal to very specific software

services

• Cloud uses range from ―turn key‖ applications to direct application

development

• Motivations for cloud computing

• Much of the ―buzz‖ is business related

• Many are related to resource concentration

• Around data, facility benefits, expertise, mission

• Optimize utilization and cost

• Reduce risk

• Low bar to access for many

15

HPC and Clouds

• HPC and ―distributed‖ computing have a very long and mutually beneficial relationship

• Remote use of large resources

• Cluster computing

• Parallel computing

• Distributed computing

• Internet

• Web

• Grid

• Clouds

• What is different today that could make clouds dominate

• The wiring of the world

• Software as a Service

• Standard interfaces

• Shrinking hardware options

• Consolidation

• Peoples laptops and desktops note enough to do their analysis

16

So are HPC systems cloud resources?

• Yes – of course HPC can be parts of clouds

Large concentration of modern physical resources co-located for efficiency

Consumers are geographically separated

Time and component multiplexing

Users expect immediate feedback

Mostly for independent, unsynchronized tasks

Utility computing provider want to capture and retain user base

? Users expect low bar to access/Little HW specifics and little optimization

? Relies on under utilization to succeed

• Yes - HPC systems and applications can and will be cloud services

• Portals and data sharing

• Expansion of computing and storage resources

• Ability to solve societal scale problems

• Needed to train next generation of computational and analytical experts

17

Cloud Challenges

• Parallelism

• Can clouds thrive when parallelism dominates?

• Memory wall implications

• For science and academia – is there enough bandwidth?

• Are simple tools sufficient for complex analysis?

• Data movement costs

• Persistent data storage costs

• Cyber security and privacy

• Simple rule of outsourcing – don’t outsource your core business advantages

• Does this still apply?

18

Blue Wave:

K-12 Education Next Generation ICLCS/WebMO

A Cloud to Grid Conduit model for Next Generation Science Gateway

NCSA Enterprise Cloud & Virtual Machine Services

ICLCS/WebMO Cloud Science Gateway

Passive LB Node

Active LB Node

CentralizeRelational Database

Shared Network File System

WorkerNode

WorkerNode Worker

NodeWorkerQueue

GRIDQueue Resource

Node

ResouceNodeResource

Node

Internet Users

Conduit

Dedicated

Dynamically

Scalable

Proxy

• Cloud to Grid Conduit • Blue Wave (HPC for K-12 Education)

• ICLCS WebMO as an example

• Cloud to Cloud Conduit• Model for Eucalyptus Cluster integration (UIUC CS)

• Model for Hadoop integration (HP/Intel/Yahoo UIUC/CS)

• Model for Site to Site integration (Intra-Institutional)

• Mixed Services Integrations• Persistent Virtual Instances and Dynamic Instances

• Enable Experimentation into using Amazon & Azure for instant access to large-scale limited-time usage.• Account for development and experimentation

• Investigate Virtual Network Interconnection

NCSA Enterprise Cloud Conduits

Summary

• Clouds and HPC are both needed

• HPC and clouds are mutually beneficial

• Both HPC and clouds have to deal with the largest scale

• Understanding complex, highly interconnected systems that evolve dyanmically

• A challenge is to have good understanding of what a cloud means and when

it is the best value solutions

• Performance, Effectiveness, Resiliency, Consistency, Usability

• A challenge is data

• Analytics is key to transformative science, engineering and business at any scale

• Why should the large scale community be pushing "Exaflops" rather than

"Yottabytes― in order to improve science productivity and quality?

22

Acknowledgements

This research is part of the Blue Waters sustained-petascale computing

project, which is supported by the National Science Foundation

(award number OCI 07-25070) and the state of Illinois. Blue Waters is

a joint effort of the University of Illinois at Urbana-Champaign, its

National Center for Supercomputing Applications, IBM, and the Great

Lakes Consortium for Petascale Computation.

The work described is only achievable through the efforts of the Blue

Waters Project.

23

Questions?

24

Dr. William Kramer

NCSA/University of Illinois

Blue Waters Deputy Director

wkramer@ncsa.uiuc.edu/ - http://www.ncsa.uiuc.edu/BlueWaters

(217) 333-6260