The OptIPuter Project—Eliminating Bandwidth as a Barrier

to Collaboration and Analysis

DARPA Microsystems Technology Office Arlington, VA

December 13, 2002

Dr. Larry Smarr

Director, California Institute for Telecommunications and Information Technologies

Harry E. Gruber Professor,

Dept. of Computer Science and Engineering

Jacobs School of Engineering, UCSD

AbstractThe OptIPuter is a radical distributed visualization, teleimmersion, data mining, and computing architecture. The National Science Foundation recently awarded a six-campus research consortium a five-year large Information Technology Research grant to construct working prototypes of the OptIPuter on campus, regional, national, and international scales. The OptIPuter project is driven by applications leadership from two scientific communities, the US National NSF's EarthScope and the National Institutes of Health's Biomedical Imaging Research Network (BIRN), both of which are beginning to produce a flood of large 3D data objects (e.g., 3D brain images or a SAR terrain datasets) which are stored in distributed federated data repositories.

Essentially, the OptIPuter is a "virtual metacomputer" in which the individual "processors" are widely distributed Linux PC clusters; the "backplane" is provided by Internet Protocol (IP) delivered over multiple dedicated 1-10 Gbps optical wavelengths; and, the "mass storage systems" are large distributed scientific data repositories, fed by scientific instruments as OptIPuter peripheral devices, operated in near real-time. Collaboration, visualization, and teleimmersion tools are provided on tiled mono or stereo super-high definition screens directly connected to the OptIPuter to enable distributed analysis and decision making. The OptIPuter project aims at the re-optimization of the entire Grid stack of software abstractions, learning how, as George Gilder suggests, to "waste" bandwidth and storage in order to conserve increasingly "scarce" high-end computing and people time in this new world of inverted values.

The Move to Data-Intensive Science & Engineering-e-Science Community Resources

ATLAS

Sloan Digital Sky Survey

LHC

ALMA

CONTROL

PLANE

Clusters

DynamicallyAllocatedLightpaths

Switch Fabrics

PhysicalMonitoring

Apps Middleware

A LambdaGrid Will Be the Backbone for an e-Science Network

Source: Joe Mambretti, NU

Just Like in Computing --Different FLOPS for Different Folks

DSL GigE LAN

C

A

B

A -> Need Full Internet Routing

B -> Need VPN Services On/And Full Internet Routing

C -> Need Very Fat Pipes, Limited Multiple Virtual Organizations

Source: Cees Delaat

Number of users

Bandwidth consumed

OptIPuter NSF Proposal Partnered with National Experts and Infrastructure

Vancouver

Seattle

Portland

San Francisco

Los Angeles

San Diego(SDSC)

NCSA

SURFnet CERNCA*net4

AsiaPacific

AsiaPacific

AMPATH

PSC

Atlanta

CA*net4

Source: Tom DeFanti and Maxine Brown, UIC

NYC

TeraGrid DTFnet

CENIC

Pacific LightRail

Chicago

UICNU

USC

UCSD, SDSUUCI

The OptIPuter is an Experimental Network Research Project

• Driven by Large Neuroscience and Earth Science Data• Multiple Lambdas Linking Clusters and Storage

– LambdaGrid Software Stack– Integration with PC Clusters– Interactive Collaborative Volume Visualization– Lambda Peer to Peer Storage With Optimized Storewidth– Enhance Security Mechanisms– Rethink TCP/IP Protocols

• NSF Large Information Technology Research Proposal– UCSD and UIC Lead Campuses—Larry Smarr PI– USC, UCI, SDSU, NW Partnering Campuses– Industrial Partners: IBM, Telcordia/SAIC, Chiaro Networks– $13.5 Million Over Five Years

The OptIPuter Frontier Advisory Board

• Optical Component Research– Shaya Fainman, UCSD– Sadik Esener, UCSD– Alan Willner, USC– Frank Shi, UCI– Joe Ford, UCSD

• Optical Networking Systems– Dan Blumenthal, UCSB– George Papen, UCSD– Joe Mambretti, Northwestern University– Steve Wallach, Chiaro Networks, Ltd.– George Clapp, Telcordia/SAIC– Tom West, CENIC

• Data and Storage– Yannis Papakonstantinou, UCSD– Paul Siegel, UCSD

• Clusters, Grid, and Computing– Alan Benner, IBM eServer Group, Systems Architecture and Performance department – Fran Berman, SDSC director– Ian Foster, Argonne National Laboratory

• Generalists– Franz Birkner, FXB Ventures and San Diego Telecom Council– Forest Baskett, Venture Partner with New Enterprise Associates– Mohan Trivedi, UCSD

First Meeting February 6-7, 2003

The First OptIPuter Workshopon Optical Switch Products

• Hosted by Calit2 @ UCSD– October 25, 2002– Organized by Maxine Brown (UIC) and Greg Hidley (UCSD)– Full Day Open Presentations by Vendors and OptIPuter Team

• Examined Variety of Technology Offerings:– OEOEO

– TeraBurst Networks

– OEO– Chiaro Networks

– OOO– Glimmerglass– Calient– IMMI

Coherence

DRAM - 4 GB - HIGHLY INTERLEAVEDMULTI-LAMBDAOptical Network

VLIW/RISC CORE40 GFLOPS

10 GHz

240 GB/s24 Bytes wide

240 GB/s24 Bytes wide

VLIW/RISC CORE 40 GFLOPS 10 GHz

...

2nd LEVEL CACHE8 MB

2nd LEVEL CACHE 8 MB

CROSS BAR

DRAM – 16 GB64/256 MB - HIGHLY INTERLEAVED

640GB/s

OptIPuter Inspiration--Node of a 2009 PetaFLOPS Supercomputer

Updated From Steve Wallach, Supercomputing 2000 Keynote

5 Terabits/s

Global Architecture of a 2009 COTS PetaFLOPS System

I/O

ALL-OPTICAL SWITCH

Multi-DieMulti-Processor

1

23

64

63

49

48

4 516

17

18

32

3347 46

128 Die/Box4 CPU/Die

10 meters= 50 nanosec Delay

...

...

...

...

LAN/WAN

Source: Steve Wallach, Supercomputing 2000 Keynote

Systems Become GRID Enabled

Convergence of Networking Fabrics

• Today's Computer Room– Router For External Communications (WAN)– Ethernet Switch For Internal Networking (LAN)– Fibre Channel For Internal Networked Storage (SAN)

• Tomorrow's Grid Room– A Unified Architecture Of LAN/WAN/SAN Switching– More Cost Effective

– One Network Element vs. Many

– One Sphere of Scalability– ALL Resources are GRID Enabled

– Layer 3 Switching and Addressing Throughout

Source: Steve Wallach, Chiaro Networks

The OptIPuter Philosophy

“A global economy designed to waste transistors, power, and silicon area

-and conserve bandwidth above all- is breaking apart and reorganizing itself

to waste bandwidth and conserve power, silicon area, and transistors."

George Gilder Telecosm (2000)

Bandwidth is getting cheaper faster than storage.Storage is getting cheaper faster than computing.

Exponentials are crossing.

From SuperComputers to SuperNetworks--Changing the Grid Design Point

• The TeraGrid is Optimized for Computing– 1024 IA-64 Nodes Linux Cluster– Assume 1 GigE per Node = 1 Terabit/s I/O– Grid Optical Connection 4x10Gig Lambdas = 40 Gigabit/s– Optical Connections are Only 4% Bisection Bandwidth

• The OptIPuter is Optimized for Bandwidth– 32 IA-64 Node Linux Cluster– Assume 1 GigE per Processor = 32 gigabit/s I/O– Grid Optical Connection 4x10GigE = 40 Gigabit/s– Optical Connections are Over 100% Bisection Bandwidth

Data Intensive Scientific Applications Require Experimental Optical Networks

• Large Data Challenges in Neuro and Earth Sciences– Each Data Object is 3D and Gigabytes– Data are Generated and Stored in Distributed Archives– Research is Carried Out on Federated Repository

• Requirements– Computing Requirements PC Clusters– Communications Dedicated Lambdas Over Fiber– Data Large Peer-to-Peer Lambda Attached Storage – Visualization Collaborative Volume Algorithms

• Response– OptIPuter Research Project

The Biomedical Informatics Research Network a Multi-Scale Brain Imaging Federated Repository

BIRN Test-bedsBIRN Test-beds::Multiscale Mouse Models of Disease, Human Brain Morphometrics, and Multiscale Mouse Models of Disease, Human Brain Morphometrics, and

FIRST BIRN (FIRST BIRN (10 site project for fMRI’s of Schizophrenics)10 site project for fMRI’s of Schizophrenics)

NIH Plans to Expand to Other Organs

and Many Laboratories

Microscopy Imaging of Neural TissueMarketta Bobik Francisco Capani & Eric Bushong

Confocal image of a sagittal section through rat cortex triple labeled for

glial fibrillary acidic protein (blue), neurofilaments (green) and actin (red)

Projection of a series of optical sections through a Purkinje neuron

revealing both the overall morphology (red) and the dendritic spines (green)

http://ncmir.ucsd.edu/gallery.html

Interactive Visual Analysis of Large Datasets --East Pacific Rise Seafloor Topography

http://siovizcenter.ucsd.edu/library/gallery/shoot1/index.shtml

Scripps Institution of Oceanography Visualization Center

Tidal Wave Threat AnalysisUsing Lake Tahoe Bathymetry

http://siovizcenter.ucsd.edu/library/gallery/shoot1/index.shtml

Scripps Institution of Oceanography Visualization Center

Graham Kent, SIO

SIO Uses the Visualization Center to Teach a Wide Variety of Graduate Classes

• Geodesy • Gravity and Geomagnetism • Planetary Physics • Radar and Sonar Interferometry

• Seismology • Tectonics • Time Series Analysis

Multiple Interactive Views of Seismic Epicenter and Topography Databases

http://siovizcenter.ucsd.edu/library/gallery/shoot2/index.shtml

Deborah Kilb & Frank Vernon, SIO

NSF’s EarthScopeRollout Over 14 Years Starting

With Existing Broadband Stations

Metro Optically Linked Visualization Wallswith Industrial Partners Set Stage for Federal Grant

• Driven by SensorNets Data– Real Time Seismic– Environmental Monitoring – Distributed Collaboration– Emergency Response

• Linked UCSD and SDSU– Dedication March 4, 2002

Linking Control Rooms

Cox, Panoram,SAIC, SGI, IBM,

TeraBurst NetworksSD Telecom Council

UCSD SDSU44 Miles of Cox Fiber

Extending the Optical Grid to Oil and Gas Research

• Society for Exploration Geophysicists in Salt Lake City Oct. 6-11, 2002• Optically Linked Visualization Walls

– 80 Miles of Fiber from BP Visualization Lab from Univ. of Colorado – OC-48 Both Ways– Interactive Collaborative Visualization of Seismic Cubes & Reservoir Models– SGI, TeraBurst Industrial Partners

• Organized by SDSU and Cal-(IT)2

Source: Eric Frost, SDSU

½ Mile

The UCSD OptIPuter Deployment

SIO

SDSC

CRCA

Phys. Sci -Keck

SOM

JSOE Preuss

6th College

Phase I, Fall 02

Phase II, 2003

SDSCAnnex

Collocation point

Node M

The OptIPuter Experimental UCSD Campus Optical Network

Earth Sciences

SDSC

Arts

Chemistry

Medicine

Engineering

High School

UndergradCollege

Phase I, Fall 02

Phase II, 2003

SDSCAnnex

To CENIC

Collocation point

Collocation

Chiaro Router (Installed Nov 18, 2002)

Production Router (Planned)

Source: Phil Papadopoulos, SDSC; Greg Hidley, Cal-(IT)2

Roughly, $0.20 / Strand-Foot

UCSD New Cost Sharing Roughly $250k of Dedicated Fiber

• Cluster – Disk

• Disk – Disk

• Viz – Disk

• DB – Cluster

• Cluster – Cluster

OptIPuter LambdaGridEnabled by Chiaro Networking Router

www.calit2.net/news/2002/11-18-chiaro.html

switch switch

switchswitch

Medical Imaging and Microscopy

Chemistry, Engineering, Arts

San Diego Supercomputer Center

Scripps Institution of Oceanography

ChiaroEnstara

Image Source: Phil Papadopoulos, SDSC

We Chose OptIPuter for Fast Switching and Scalability

Lithium Lithium

NiobateNiobate

ElectricalElectrical

FabricsFabricsBubbleBubble

Packet

Switching

Speeds

(ns)

Switching

Speeds

(ms)

Small

Port

Count

Large

Port

Count

ChiaroChiaro

OpticalOptical

PhasedPhased

ArrayArray

MEMSMEMS

ElectricalElectrical

FabricsFabrics

InputOptical Fiber

WG #1 WG #128

• • • • • •

Optical Phased Array –Multiple Parallel Optical Waveguides

OutputOutputFibersFibers

Air G

ap

Air G

ap

GaAs Waveguides

Chiaro Has a Scalable, Fully Fault Tolerant Architecture

• Significant Technical Innovation– OPA Fabric Enables

Large Port Count– Global Arbitration

Provides Guaranteed Performance

– Fault-Tolerant Control System Provides Non-stop Performance

• Smart Line Cards – ASICs With

Programmable Network Processors

– Software Downloads For Features And Standards Evolution

NetworkNetworkProc.Proc.LineLineCardCard

NetworkNetworkProc.Proc.LineLineCardCard

GlobalGlobalArbitrationArbitration

Optical Electrical

ChiaroOPA

Fabric

NetworkNetworkProc.Proc.LineLineCardCard

NetworkNetworkProc.Proc.Line Line CardCard

Planned Chicago Metro Lambda Switching OptIPuter Laboratory

Int’l GE, 10GE

Nat’l GE, 10GE

Metro GE, 10GE

16x1 GE 16x10 GE

16-Processor McKinley at University of Illinois at Chicago

16-Processor Montecito/Chivano

at Northwestern StarLight

10x1 GE

+

1x10GE

Nationals: Illinois, California, Wisconsin, Indiana, Abilene, FedNets. Washington, Pennsylvania…

Internationals: Canada, Holland, CERN, GTRN, AmPATH, Asia…

Source: Tom DeFanti, UIC

OptIPuter Software Research

• Near-term: Build Software To Support Advancement Of Applications With Traditional Models– High Speed IP Protocol Variations (RBUDP, SABUL, …)– Switch Control Software For DWDM Management And Dynamic Setup– Distributed Configuration Management For OptIPuter Systems

• Long-Term Goals To Develop: – System Model Which Supports Grid, Single System, And Multi-System Views– Architectures Which Can:

– Harness High Speed DWDM– Present To The Applications And Protocols

– New Communication Abstractions Which Make Lambda-Based Communication Easily Usable

– New Communication & Data Services Which Exploit The Underlying Communication Abstractions

– Underlying Data Movement & Management Protocols Supporting These Services

– “Killer App” Drivers And Demonstrations Which Leverage This Capability Into The Wireless Internet

Source: Andrew Chien, UCSD

OptIPuter System Opportunities

• What’s The Right View Of The System?• Grid View

– Federation Of Systems – Autonomously Managed, Separate Security, No Implied Trust Relationships, No Transitive Trust

– High Overhead – Administrative And Performance– Web Services And Grid Services View

• Single System View– More Static Federation Of Systems– A Single Trusted Administrative Control, Implied Trust Relationships,

Transitive Trust Relationships– But This Is Not Quite A Closed System Box

– High Performance– Securing A Basic System And Its Capabilities

– Communication, Data, Operating System Coordination Issues

• Multi-System View– Can We Create Single System Views Out Of Grid System Views?– Delivering The Performance; Boundaries On Trust

Source: Andrew Chien, UCSD

OptIPuter Communication Challenges

• Terminating A Terabit Link In An Application System– --> Not A Router

• Parallel Termination With Commodity Components– N 10GigE Links -> N Clustered Machines (Low Cost)– Community-Based Communication

• What Are:– Efficient Protocols to Move Data in Local, Metropolitan, Wide Area?

– High Bandwidth, Low Startup– Dedicated Channels, Shared Endpoints

– Good Parallel Abstractions For Communication?– Coordinate Management And Use Of Endpoints And Channels– Convenient For Application, Storage System

– Secure Models For “Single System View”– Enabled By “Lambda” Private Channels– Exploit Flexible Dispersion Of Data And Computation

Source: Andrew Chien, UCSD

OptIPuter Storage Challenges

• DWDM Enables Uniform Performance View Of Storage– How To Exploit Capability? – Other Challenges Remain: Security, Coherence, Parallelism– “Storage Is a Network Device”

• Grid View: High-Level Storage Federation– GridFTP (Distributed File Sharing)– NAS – File System Protocols– Access-control and Security in Protocol– Performance?

• Single-System View: Low-Level Storage Federation– Secure Single System View– SAN – Block Level Disk and Controller Protocols– High Performance– Security? Access Control?

• Secure Distributed Storage: Threshold Cryptography Based Distribution– PASIS Style – Distributed Shared Secrets– Lambda’s Minimize Performance Penalty

Source: Andrew Chien, UCSD

OptIPuter is Exploring Quanta as a High Performance Middleware

• Quanta is a high performance networking toolkit / API.• Reliable Blast UDP:

– Assumes you are running over an over-provisioned or dedicated network.

– Excellent for photonic networks, don’t try this on commodity Internet.– It is FAST!– It is very predictable.– We give you a prediction equation to predict performance. This is

useful for the application.– It is most suited for transfering very large payloads.– At higher data rates processor is 100% loaded so dual processors

are needed for your application to move data and do useful work at the same time.

Source: Jason Leigh, UIC

TeraVision Over WAN : Greece to Chicago Throughput

TCP Performance Over WAN Is Poor; Windows Performance Is Lower Than Linux; Synchronization Reduces Frame Rate.

Reliable Blast UDP (RBUDP)

• At IGrid 2002 all applications which were able to make the most effective use of the 10G link from Chicago to Amsterdam used UDP

• RBUDP[1], SABUL[2] and Tsunami[3] are all similar protocols that use UDP for bulk data transfer- all of which are based on NETBLT- RFC969

• RBUDP has fewer memory copies & a prediction function to let applications know what kind of performance to expect.– [1] J. Leigh, O. Yu, D. Schonfeld, R. Ansari, et al., Adaptive

Networking for Tele-Immersion, Proc. Immersive Projection Technology/Eurographics Virtual Environments Workshop (IPT/EGVE), May 16-18, Stuttgart, Germany, 2001.

– [2] Sivakumar Harinath, Data Management Support for Distributed Data Mining of Large Datasets over High Speed Wide Area Networks, PhD thesis, University of Illinois at Chicago, 2002.

– [3] http://www.indiana.edu/~anml/anmlresearch.html

Source: Jason Leigh, UIC

5x3 Grid of 1280x1024 Pixel LCD Panels Driven by 16-PC Cluster

Resolution=6400x3072 Pixels,

or ~3000x1500 pixels in Autostereo

Visualization at Near Photographic ResolutionThe OptIPanel Version I

Source: Tom DeFanti, EVL--UIC

NTT Super High Definition Video (NTT 4Kx2K=8 Megapixels) Over Internet2

Starlight in Chicago

USC In Los Angeles

SHD= 4xHDTV = 16xDVD

www.ntt.co.jp/news/news02e/0211/021113.html

Applications:Astronomy

MathematicsEntertainment

The Continuum at EVL and TRECCOptIPuter Amplified Work Environment

Passive stereo display AccessGrid Digital white board

Tiled display

Source: Tom DeFanti, Electronic Visualization Lab, UIC

Fast polygon and volume rendering with stereographics

GeoWall

Earth Science

GeoFusion GeoMatrix Toolkit

Underground Earth Science

Rob Mellors and Eric Frost, SDSUSDSC Volume Explorer

Dave Nadeau, SDSC, BIRNSDSC Volume Explorer

NeuroscienceAnatomy

Visible Human ProjectNLM, Brooks AFB,

SDSC Volume Explorer

3D APPLICATIONS:

+

=

OptIPuter Transforms Individual Laboratory Visualization, Computation, & Analysis Facilities

The Preuss School UCSD OptIPuter Facility

Providing a 21st Century Internet Grid Infrastructure

Tightly Coupled Optically-Connected OptIPuter Core

Wireless Sensor Nets, Personal Communicators

Loosely Coupled Peer-to-Peer Computing & Storage

Routers

Routers