Roland Dreier Technical Lead – Cisco Systems, Inc. OpenIB Maintainer Sean Hefty Software Engineer...

Roland DreierRoland DreierTechnical Lead – Cisco Systems, Inc.Technical Lead – Cisco Systems, Inc.

OpenIB MaintainerOpenIB Maintainer

Sean HeftySean HeftySoftware Engineer – Intel CorporationSoftware Engineer – Intel Corporation

OpenIB MaintainerOpenIB Maintainer

Yaron HavivYaron HavivCTO – VoltaireCTO – Voltaire

Stack Development Stack Development Update, Testing Update, Testing Status and Target Status and Target Release GoalsRelease Goals

2*Third party marks and brands are the property of their respective owners

AgendaAgenda

www.openib.orgwww.openib.org Linux Kernel InfrastructureLinux Kernel Infrastructure Linux Userspace InfrastructureLinux Userspace Infrastructure RDMA Enabled ApplicationsRDMA Enabled Applications InfiniBand and BeyondInfiniBand and Beyond

The InfiniBand Software SolutionThe InfiniBand Software Solution


OpenIB AllianceOpenIB Alliancewww.openib.org

•Improve interoperability

•Reduce the time and cost for deployment

•Improve interoperability

•Reduce the time and cost for deployment

Open-Source InfiniBand RDMA InfrastructureOpen-Source InfiniBand RDMA Infrastructure


OpenIB AllianceOpenIB Alliancewww.openib.org

…and the open source community!…and the open source community!


PhilosophyPhilosophy

Open source developmentOpen source development–Dual BSD / GPL LicenseDual BSD / GPL License

Integrated with OS infrastructureIntegrated with OS infrastructure Working with companion RDMA standardsWorking with companion RDMA standards AnyoneAnyone can contribute can contribute

www.openib.org

Code Quality,Code Quality, Stability,Stability, PerformancePerformance


AchievementsAchievementswww.openib.org

JunJun NovNov MarMar MayMay JulJul‘05‘04

Will support Windows & RDMA enabled Ethernet

Will support Windows & RDMA enabled Ethernet

Infiniband Leaders Form OpenIB Alliance

Infiniband Leaders Form OpenIB Alliance

Deliver a high-performance, scalable, and interoperable Infiniband software stack

Deliver a high-performance, scalable, and interoperable Infiniband software stack

“This will enable … a robust, resilient, and manageable infrastructure for data center, Grid, and HPC.”

“This will enable … a robust, resilient, and manageable infrastructure for data center, Grid, and HPC.”

“… accelerate the creation of a single Infiniband software stack”

“… accelerate the creation of a single Infiniband software stack” OpenIB Linux

release

OpenIB Linux release


Software ArchitectureSoftware Architecture

HCA

Core InfiniBand Modules

Verbs

MAD Services

SMI GSI RMPP

PMASMA SA ClientCM

HCA Driver

User-level HCA Driver

User-level InfiniBand Services

VerbsMAD Services SA ClientCM

User-level Access Modules

CM SA ClientMAD Services

user-space

kernel

User Verbs

kDAPLIPoIB SRPSDP

iSER NFS-R

Portals

Lustre

OpenSM OpenMPI uDAPL RDMA App.

www.openib.org

HCA


Verbs

MAD Services

SMI GSI RMPP

PMASMA SA ClientCM

HCA DriverHCA


Verbs

MAD Services

SMI GSI RMPP

PMASMA SA ClientCM

HCA Driver

kDAPLIPoIB SRPSDP

iSER NFS-R

Portals

Lustre


kDAPLIPoIB SRPSDP

iSER NFS-R

Portals

Lustre







user-space

kernel

User Verbs






user-space

kernel

User Verbs


Linux Kernel InfrastructureLinux Kernel Infrastructure

HCA


Verbs

MAD Services

SMI GSI RMPP

kDAPL

PMASMA SA ClientCM

HCA Driver

IPoIB SRPSDPiSER NFS-R

Portals

Lustre

OpenSM OpenMPI uDAPL






user-space

kernel

User Verbs

RDMA App.


OverviewOverview

Verbs API derived from VAPIVerbs API derived from VAPI Best of breed software solutionBest of breed software solution Support multiple HCASupport multiple HCA Optimized for Linux kernelOptimized for Linux kernel Support latest stable kernel releaseSupport latest stable kernel release

Linux Kernel Infrastructure

Complete InfiniBand infrastructureComplete InfiniBand infrastructure


ArchitectureArchitecture

HCA

HCA Driver

Linux Kernel Infrastructure

ib_mthca


ib_core Verbs

ib_mad

ib_sa

SA Client

ib_ipoibIPoIB

Linux 2.6.11

Linux 2.6.13CM

kDAPLSRPSDP

iSER NFS-R

Portals

Lustre

OpenIB Release

Development

External Project

MAD Services SMI GSI

PMASMAib_cm

RMPP


Linux Userspace InfrastructureLinux Userspace Infrastructure

HCA


Verbs

MAD Services

SMI GSI RMPP

kDAPL

PMASMA SA ClientCM

HCA Driver


Portals

Lustre







user-space

kernel

User Verbs

RDMA App.


OverviewOverview

Library exposes verbs to usermode clientsLibrary exposes verbs to usermode clients Secure userspace accessSecure userspace access Kernel bypass for speed path operationsKernel bypass for speed path operations RDMA to / from application memoryRDMA to / from application memory

Linux Userspace Infrastructure

Direct hardware access for applicationsDirect hardware access for applications




user-space

kernel

ArchitectureArchitectureLinux Userspace Infrastructure

HCA


HCA Driver


Verbs

User Verbs

MAD Services

MAD Services

CM

CM

SA Client

SA Client


openib.orgopenib.org

2.6.132.6.13

14*Third party marks and brands are the property of their respective ownersHCA


Verbs

MAD Services

SMI GSI RMPP

kDAPL

PMASMA SA ClientCM

HCA Driver


Portals

Lustre







user-space

kernel

User Verbs

RDMA App.

RDMA Enabled ApplicationsRDMA Enabled Applications


One Fabric, Many UsesOne Fabric, Many Uses

Low latency IPC for clusteringLow latency IPC for clustering Variety of high speed file and block Variety of high speed file and block

protocols, remote boot capable protocols, remote boot capable Bridging to external GbE and FCBridging to external GbE and FC

Multi-Service Switches

Management

Servers

ParallelFC / IB / IP

Storage

InfiniBand

IB HCA

IPC (MPI & DAPL)

Applications

Network (SDP & IPoIB)

Storage (File & Block)

InfiniBand Host Software Stack

RDMA Enabled Applications

One fabric has it allOne fabric has it all


Networking / IPC APIsNetworking / IPC APIsRDMA Enabled Applications

Tunnels IP packets over IB

Tunnels IP packets over IB

Application APIs unchanged

Application APIs unchanged

HCA


HCA Driver

Userspace HCA Driver

Userspace InfiniBand Services

user-space

kernel

IPC Application

NetPipe

Intel MPI

DAPL MPI

Sockets

IPoIB

IP

TCP

SDP

Perftest

OpenIB software enables applications

OpenIB software enables applications

Fabric agnostic RDMA API

Fabric agnostic RDMA API

Standard API for scientific clustering

Standard API for scientific clustering

Bypasses TCP/IP stack for direct

HW access

Bypasses TCP/IP stack for direct

HW access


Networking / IPC PerformanceNetworking / IPC PerformanceRDMA Enabled Applications

How well can apps perform?How well can apps perform?

Ciara VXB-7520J

3.4 GHz Dual EM64T Xeon3.4 GHz Dual EM64T Xeon 4 GB DDR4 GB DDR RedHat EL4.0RedHat EL4.0 OpenIB SVN 2833 backportedOpenIB SVN 2833 backported 10 Gb/sec (4X) - 8X PCI Express HCA10 Gb/sec (4X) - 8X PCI Express HCA

MTS2400

MTLP25208


Networking / IPC PerformanceNetworking / IPC Performance

Performance tests and ratings are measured using Performance tests and ratings are measured using specific computer systems and/or components and specific computer systems and/or components and reflect the approximate performance of Intel products reflect the approximate performance of Intel products as measured by those tests. Any difference in system as measured by those tests. Any difference in system hardware or software design or configuration may hardware or software design or configuration may affect actual performance. Buyers should consult affect actual performance. Buyers should consult other sources of information to evaluate the other sources of information to evaluate the performance of systems or components they are performance of systems or components they are considering purchasing.considering purchasing.


I.e. your results may varyI.e. your results may vary


Networking / IPC PerformanceNetworking / IPC Performance

0

5

10

15

20

25

30

usecs0

200

400

600

800

1000

MBps


Small message latencySmall message latency

IPoIBIPoIB SDPSDP MPIMPI

3.46 us3.46 us

924 MBps924 MBps

VerbsVerbs

Large message bandwidthLarge message bandwidth


InfiniBand Revolutionize InfiniBand Revolutionize StorageStorage

High-speed NAS and SAN on the same fabricHigh-speed NAS and SAN on the same fabric Scale out storage through clustering and Scale out storage through clustering and

virtualizationvirtualization

Server Racks A Bunch of low cost Storage Arrays

A lot of bandwidth

(InfiniBand)

Commodity Servers running NAS Gateways or Cluster File System

Switch



Storage in OpenIBStorage in OpenIBRDMA Enabled Applications

Storage

NICR-NICIB HCA

Switch

SCSI Midlayer

SRP

Core InfiniBand Services

HCA Driver NIC Driver

TCP/IP

iSCSI

RNIC Driver

kDAPL

iWarp Services

iSER


NFS/RDMANFS/RDMA

High-speed RDMA High-speed RDMA RPC transportRPC transport

Maps transparently to Maps transparently to NFS and applicationsNFS and applications

Reside over kDAPL Reside over kDAPL for fabric for fabric independenceindependence

NFS

RPC/XDR Switch

UDPTCPRDMA

RDMA Standard NIC

Direct Access

kDAPL



Xen Virtualization over IBXen Virtualization over IB

VM1

OS

VM2

OS

VM3

OS

GbE NIC FC HCA

Software based Network and Storage switching

XEN

StorageNetwork

Slower I/OSlower I/O No isolation No isolation Multiple cards and fabrics Multiple cards and fabrics

Direct HW access for I/ODirect HW access for I/O One 10Gb/s card for Network, One 10Gb/s card for Network,

Storage, and IPCStorage, and IPC Scale out using clustering (IPC)Scale out using clustering (IPC)

VM1

OS

VM2

OS

VM3

OS

IB HCA

XEN

IPC

Direct HW access

Xen with IBXen, the traditional way



Infiniband and BeyondInfiniband and Beyond

HCA


Verbs

MAD Services

SMI GSI RMPP

kDAPL

PMASMA SA ClientCM

HCA Driver


Portals

Lustre







user-space

kernel

User Verbs

RDMA App.

What’s left?What’s left?


Future EffortsFuture Efforts

Scale out testingScale out testing Scalability enhancementsScalability enhancements Integrate upper-layer protocol softwareIntegrate upper-layer protocol software Windows software releaseWindows software release Xen supportXen support

InfiniBand and Beyond


Companion TechnologiesCompanion Technologies

Working with OpenRDMA GroupWorking with OpenRDMA Group Common RDMA interface for LinuxCommon RDMA interface for Linux Support InfiniBand and iWarpSupport InfiniBand and iWarp


HCA

HCA Driver

RDMA Application

RNIC

InfiniBand Services

RNIC Driver

iWarp Services

RDMA Verbs

RDMA Application Library

Connection Mgmt

InfiniBand Verbs

HCA

HCA Driver

InfiniBand Services

SA ClientCM

RDMA Application

kDAPL

HCA

HCA Driver

RDMA Application

InfiniBand Services

RDMA Verbs


HCA

HCA Driver

RDMA Application

InfiniBand Services

RDMA Verbs


Connection Mgmt


SummarySummary

Solid InfiniBand infrastructureSolid InfiniBand infrastructure Available in Linux today, Windows soonAvailable in Linux today, Windows soon Low latency, high bandwidth applicationsLow latency, high bandwidth applications

OpenIB – Open-Source Infiniband RDMA Infrastructure

OpenIB = InfiniBand softwareOpenIB = InfiniBand software


Call to ActionCall to Action

Join the open source effortJoin the open source effort Request OpenIB software from your Linux Request OpenIB software from your Linux

distributordistributor Ask your IB vendor about OpenIB supportAsk your IB vendor about OpenIB support


Join OpenIBJoin OpenIB


Please fill out the Please fill out the Session Evaluation Session Evaluation Form.Form.

Thank You!Thank You!


GlossaryGlossary CM – communication managerCM – communication manager DAPL – direct access provider libraryDAPL – direct access provider library GSI – general services agentGSI – general services agent HCA – host channel adapterHCA – host channel adapter IPC – interprocess communicationIPC – interprocess communication IPoIB – IP over InfinibandIPoIB – IP over Infiniband iSER – iSCSI extensions for RDMAiSER – iSCSI extensions for RDMA PMA – performance management agentPMA – performance management agent SDP – sockets direct protocolSDP – sockets direct protocol SM – subnet managerSM – subnet manager SMI – subnet management interfaceSMI – subnet management interface SRP – SCSI RDMA procotolSRP – SCSI RDMA procotol

OpenIB – Open-Source Infiniband RDMA Infrastructure


Additional Resources for this Additional Resources for this SessionSession Session presentation can be downloaded Session presentation can be downloaded

from the IDF web site – when prompted from the IDF web site – when prompted enter:enter:

–Username: idfUsername: idf

–Password: fall2005Password: fall2005

More web based info: www.openib.orgMore web based info: www.openib.org

Roland Dreier Technical Lead – Cisco Systems, Inc. OpenIB Maintainer Sean Hefty Software Engineer...

Documents

Transcript of Roland Dreier Technical Lead – Cisco Systems, Inc. OpenIB Maintainer Sean Hefty Software Engineer...