Slide 1

DESIGN, IMPLEMENTATION, AND PERFORMANCE ANALYSIS OF THE ISCSI PROTOCOL FOR SCSI OVER

TCP/IP

ByAnshul Chadda (Trebia Networks)-Speaker

Ashish Palekar (Trebia Networks)Narendran Ganapathy (Sun Microsystems)R.D.Russell (University of New Hampshire)

Slide 2

Overview

• Background – SCSI Concepts– SAN– iSCSI– SCSI Subsystem in Linux

• iSCSI Initiator Implementation• Resources Used• Basic Performance Analysis• Conclusions and Future Work• Questions

Slide 3

What is SCSI?

• Acronym for Small Computer System Interface

• SCSI device identified by ID• Logical Units present within the SCSI device• Logical Unit Numbers: for logical addressing

TARGET Hard disk

INITIATOR(Host)

LUN 0 LUN 1 LUN 2 LUN 3

ID 7 ID 0

SCSI bus

Slide 4

SCSI Terms

Limitations• Length of the SCSI bus (25 m) • Number of SCSI devices supported• Single host point-to-point

INITIATOR TARGET

COMMAND

STATUS

MESSAGE IN

MESSAGE OUT

DATA IN

DATA OUT

Slide 5

Storage Area Network (SAN)

• Host accesses the remote storage as a device• Unit of Access: BLOCK• Security, Consistency, Integrity Issues at

BLOCK level

SAN

ApplicationClient 1

Block Access Protocol

ApplicationClient 2

Disk

Slide 6

New SAN Approaches

TCP/IP/Ethernet Network

ApplicationClient 1

Block Access Protocol

ApplicationClient 2

Disk

Ideal Solution

Our Approach

ApplicationClient 1

Block Access Protocol

ApplicationClient 2 Disk

TCP/IP/Ethernet Network

Target Emulator

Slide 7

iSCSI Protocol (an IETF standard)

• Requires Reliable Transport Layer (TCP)• Session established between Initiator and Target• Multiple connections within a session • Header size

– Large (48-byte) Basic header segment– Additional Header Segment can be present

• Sequence Numbers • Header and Data Digests optional• Security and Authentication optional• Parameter Negotiation• Target Discovery Mechanism

Slide 8

User space

Kernel space

Target EmulatorInitiator

User Application

File System

SCSI

SCSI HBA Driver

Disk

User Application

File System

SCSI

iSCSI Subsystem

Disk

Transport Network

Network Stack

Network Stack

SCSI

iSCSI

Subsystem

SCSI HBA Driver

Standard SCSI SCSI Over Network

Slide 9

SCSI Subsystem in Linux SCSI Upper level

SCSI Mid level

SCSI Host Bus Adapter

(Low level driver)

Target (Disk)

Target (Disk)

Target (Disk)

TransportNetwork

SCSI Host Bus Adapter

(Low level driver)

Target (Disk)

Target (Disk)

Target (Disk)

TransportNetwork

SCSI Command SCSI Status,Response

SCSI Command SCSI Status,Response

Slide 10

iSCSI Initiator Implementation

Slide 11

Resources Used For Testing

• Initiator– 2.4 GHz CPU, 1GB Memory– Fast/Gigabit Ethernet Cards– Linux Kernel (2.4.19)– UNH iSCSI Initiator kernel modules

• Target– 2.4 GHz CPU, 1GB Memory – Fast/Gigabit Ethernet Cards– Linux Kernel (2.4.19)– UNH iSCSI Target kernel modules

Slide 12

Performance Metrics & Variables

• Metrics: Bandwidth, Latency and % CPU utilization

• Variables – Initiator Scatter-Gather List Size – Ethernet Link Speed– Target Sector Size– Header & Data Digests– TCP Nagle Algorithm

Slide 13

Baseline Performance Values

• For the following Performance Variable values:– Initiator Scatter-Gather List Size=64 – Ethernet Link Speed=1000 Mbps– Target Sector Size=512 B– Header & Data Digests=OFF– TCP Nagle Algorithm=OFF

• READ:– Bandwidth=50.6761 MB/s, – %CPU util. On target=18.65%– %CPU util. On initiator=27%

• WRITE:– Bandwidth=53.3937 MB/s – %CPU util. On target=14.3%– %CPU util. On initiator=26.9%

Slide 14

Effect of Initiator Scatter-Gather list size on Bandwidth

50.6761

46.0205

53.3937

49.841

0

10

20

30

40

50

60

Ban

dwid

th (

MB

/s)

1 2

READ WRITE(9.18% BW decrease) (6.65% BW decrease)

(~10%CPU util. Increase) (~10%CPU util. increase)

scatter-gather list size=64

scatter-gather list size=8

Slide 15

Effect of Ethernet Link Speed on Bandwidth

50.6761

11.0265

53.3937

11.0297

0

10

20

30

40

50

60

Ban

dwid

th (

MB

/s)

1 2

READ WRITE(78.24% BW decrease) (79.4% BW decrease)

(~50 %CPU util. decrease ) (~50%CPU util. decrease)

1000 MB Ethernet Link: % Bandwidth util.=42%

100 MB Ethernet Link: %Bandwidth util.=88%

Slide 16

Effect of Target Sector Size on Bandwidth

50.676152.5764 53.3937

60.2563

0

10

20

30

40

50

60

70

Ban

dwid

th (

MB

/s)

1 2

READ WRITE(3.8% BW increase) (12.9% BW increase)

(~10%CPU util. decrease) (~10%CPU util. decrease)

Target Sector Size=512B

Target Sector Size=4096B

Slide 17

Effect of Header & Data Digests on Bandwidth

50.6761

42.1175

53.3937

39.8592

0

10

20

30

40

50

60

Ban

dwid

th (

MB

/s)

1 2

READ WRITE(16.8% BW decrease) (25.3% BW decrease)(~8% CPU util. increase) (~8% CPU util. increase)

Header, Data Digests OFF

Header, Data Digests ON

Slide 18

Effect of TCP Nagle Algorithm on Bandwidth

50.676152.1742 53.3937 53.3481

0

10

20

30

40

50

60

Bandw

idth

(M

B/s

)

1 2

READ WRITE(2.96% BW increase) (0.085% BW decrease)(<5% CPU util. Change) (<5% CPU util. Change)

Nagle Algorithm=OFF

Nagle Algorithm=ON

Slide 19

Conclusions

• To Increase Bandwidth– Increase Initiator Scatter-Gather List Size – Increase Ethernet Link Speed– Increase Target Sector Size – Turn OFF Header & Data Digests– TCP Nagle Algorithm has negligible effect

Slide 20

Future Work

• Run multiple simultaneous applications on the initiator system to utilize the CPU and bandwidth

• Do Latency measurements too• Detailed Performance Analysis for each network

stack involved• Analyze the effect of iSCSI protocol parameters

on performance. • Test hardware implementation for TCP/IP and

iSCSI• Compare with Fiber Channel technology for

bandwidth, latency and CPU utilization

Slide 21

Acknowledgements

• SUN/EMC for funding the project• Inter-Operability Lab. And Trebia

Networks for providing system resources

Slide 22

Questions???