OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI...

27
www.openfabrics.org OFED Storage Protocols R. Pearson System Fabric Works, Inc.

Transcript of OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI...

Page 1: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

www.openfabrics.org

OFED Storage Protocols

R. PearsonSystem Fabric Works, Inc.

Page 2: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

2www.openfabrics.org

Agenda

Why OFED StorageIntroduction to OFED Storage ProtocolsOFED Storage Protocol Update

Page 3: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

3www.openfabrics.org

Why OFED Storage

Page 4: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

4www.openfabrics.org

Goals of I/O Consolidation

Cluster of ServersMultiple Fabrics

High CapEx and High TCOSingle InfiniBand Fabric

Low CapEx and Optimal TCO

“One Wire”

Page 5: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

5www.openfabrics.org

Unified InfiniBand Fabric

PerformanceUltimate scalabilityCost of ownershipDedicated I/O channels enable consolidation

Page 6: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

6www.openfabrics.org

Industry Support for Native InfiniBand Storage

Page 7: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

7www.openfabrics.org

Storage Interconnects

Type Max length Peak data rate Max devices

PATA (IDE) 46cm 133MB/s 2

SCSI 12m 320MB/s 16

iSCSI (Ethernet) ? ? ?

IB Cu (DDR) 15m/5m 1gbps/2gbps 16M(48K?)

IB optical (mm/WAN) ~300m/? 1GB/s 16M

SATA/eSATA 1m/2m 300MB/s 1

SAS 6m 300MB/s 16K

FC 50km 400MB/s 16M

Page 8: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

8www.openfabrics.org

Introduction to OFED Storage Protocols

Page 9: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

9www.openfabrics.org

Linux Storage Stack (das/san)

ApplicationUser

KernelVFS Layer

File Systems

Block Drivers

SCSIMid Layer

HBA Drivers

HBA

Storage

(ext2, ext3, reiser, xfs, …)

(sd, md, lvm, ..)

(scsi, fc, sata, ide, …)

(read, write, open, close, …)

Page 10: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

10www.openfabrics.org

Linux Storage Stack (nas)

ApplicationUser

KernelVFS Layer

NFSD(cache)

RPC

TCP/IP

NIC Driver

NIC

Filer

(read, write, open, close, …)

Page 11: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

11www.openfabrics.org

Linux Storage Stack (iSCSI)

ApplicationUser

KernelVFS Layer

File Systems

Block Drivers

SCSIMid Layer

iSCSI

NIC

Storage

TCP/IP

NIC Driver

Page 12: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

12www.openfabrics.org

OpenFabrics Software Stack

RDMA NICR-NIC

Host Channel Adapter

HCA

User Direct Access Programming Lib

UDAPL

Reliable Datagram Service

RDS

iSCSI RDMA Protocol (Initiator)

iSER

SCSI RDMA Protocol (Initiator)

SRP

Sockets Direct Protocol

SDP

IP over InfiniBandIPoIB

Performance Manager Agent

PMA

Subnet Manager Agent

SMA

Management Datagram

MAD

Subnet Administrator

SA

Common

InfiniBand

iWARP

Key

InfiniBand HCAInfiniBand HCA iWARP RiWARP R--NICNIC

HardwareSpecific Driver

Hardware SpecificDriver

ConnectionManagerMAD

InfiniBand Verbs / API

SA Client

ConnectionManager

Connection ManagerAbstraction (CMA)

User Level Verbs / API

SDPIPoIB SRP iSER RDS

UDAPL

SDP Library

User Level MAD API

Open SM

DiagTools

Hardware

Provider

Mid-Layer

Upper Layer Protocol

User APIs

Kernel SpaceUser Space

NFS-RDMARPC

ClusterFile Sys

Application Level

SMA

R-NIC Driver API

ClusteredDB Access

(Oracle10g RAC)

SocketsBasedAccess

(IBM DB2)

VariousMPIs

Access toFile

Systems

BlockStorageAccess

IP BasedApp

Access

Apps & Access

Methodsfor usingOF Stack

Page 13: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

13www.openfabrics.org

Linux Storage Stack (das/san)

ApplicationUser

KernelVFS Layer

File Systems

Block Drivers

SCSIMid Layer

SRP/iSER

OFED

Storage

(ext2, ext3, reiser, xfs, …)

(sd, md, lvm, ..)

(read, write, open, close, …)

HCA

Page 14: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

14www.openfabrics.org

Linux Storage Stack (nas)

ApplicationUser

KernelVFS Layer

NFSD(cache)

RPC

TCP/IP

IPoIB

OFED

Filer

(read, write, open, close, …)

HCA

Page 15: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

15www.openfabrics.org

Linux Storage Stack (iSCSI)

ApplicationUser

KernelVFS Layer

File Systems

Block Drivers

SCSIMid Layer

iSCSI

NIC

StorageTCP/IP

IPoIB

OFED

Page 16: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

16www.openfabrics.org

Common I/O Scheme

Initiator Fabric Target

LogicalView

PhysicalView

RDMA read/write

System Memory CPU CPUcommand

response

Buffers

Storage hardware

Page 17: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

17www.openfabrics.org

Common I/O Scheme

Initiator Fabric Target

LogicalView

PhysicalView

RDMA read/write

System Memory CPU CPUcommand

response

Buffers

Storage hardware

rkey

lkey

MR

MR

Page 18: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

18www.openfabrics.org

Common I/O Scheme

Initiator Fabric Target

LogicalView

PhysicalView

RDMA read/write

System Memory CPU CPUcommand

response

Buffers

Storage hardware

rkey

lkey

Page 19: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

19www.openfabrics.org

Common I/O Scheme

Initiator Fabric Target

LogicalView

PhysicalView

RDMA read/write

System Memory CPU CPUcommand

response

Buffers

Storage hardware

rkey

lkey

FMR

MR

Page 20: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

20www.openfabrics.org

OFED Storage Protocol Update

Page 21: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

21www.openfabrics.org

SRP Update

SCSI RDMA Protocol (SCSI over IB)T10.org specification, similar to FCP (SCSI over FC)All data transfers use InifiniBand RDMA

InitiatorLinux proprietary and open source “Gen1”-based in productionOFED 1.0 SRP Initiator in Beta (still in development)Windows Beta release in July

TargetLinux open source “Gen1”-based target on OFAVendors in production or in final qualification

Linux and other RTOSs

Page 22: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

22www.openfabrics.org

SRP Next Steps

InitiatorHost failover/multipathingBoot support for Linux and Windows

TargetCall for interest to port target to OFED

Mixed file system / block storage Linux targets require this capability

Page 23: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

23www.openfabrics.org

iSER Update

iSCSI Extensions for RDMAIETF IP storage specification, Leverages iSCSI management and discovery protocolsiSER eliminates iSCSI/TCP bottlenecks

RDMA, hardware transportInitiator

Production supported in OFED 1.0Tested against Voltaire iSER target running in IB/FC router

TargetLinux open source “Gen1”-based target on OFAVendors porting to OFED

Page 24: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

24www.openfabrics.org

NFS RDMA Status

Release 5 NFS/RDMA client and server for Linux 2.6.16.16 on SourceForge

Supports OFA CM and RDMA verbs APIClient maintainer is Tom Talpey, NetappServer maintainer is Tom Tucker, Open Grid Computing

Linux NFS RDMA client and server testingPasses the Connectathon test suite, IOzone and network stress testsTested with several Mellanox-based IB HCAs and standard Linux x86 serversInitial test data shows 800MB/s+ read to cache

Release 5 ready for evaluation by users, system integrators and OEMs

Page 25: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

25www.openfabrics.org

OFED Storage Summary

Benefits of consolidated InfiniBand I/O are clear and being realized by real-world deployment today

Vendors in production or in final qualification stages of block storage solutions

SRP and iSER will both continue development in OF

NFSoRDMA ready for community trialsStay in lock step with OFED

Vendors porting clustered file systems to OFED

Page 26: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

26www.openfabrics.org

System Fabric Works

Software/architecture/consulting servicesVendor neutralExperienced teamCustomers include:

DDN/LSI/Chelsio/QLogic/Cisco/Silver Storm/Trilabs/Agilent/USDoD/GE/Panta/Themis…

Page 27: OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

27www.openfabrics.org

Embedded Products

Vendor Part Number DescriptionDiversified Technology

SBS/GE Fanuc IB4X-CPCI-2A 2 port HCA cPCI 3U/6U

TDB VITA41 IB SBC

VxWorks 5.5 IB stack

IB4X-PMC-2A 2 port HCA PMC moduleTDB VITA41 IB switch

Sky Computers

ATC5232 IB ATCA SBC

ATC2148 ATCA IB Switch

AMS0508 5 slot ATCA chassis IB compatible

AMS1412 14 slot ATCA chassis IB compatible

Various IB based systems