Ethernet Over PCI Express Presented by Kallol Biswas NucleoDyne Systems, Inc. 19925 Stevens Creek...
-
Upload
lillian-candler -
Category
Documents
-
view
214 -
download
0
Transcript of Ethernet Over PCI Express Presented by Kallol Biswas NucleoDyne Systems, Inc. 19925 Stevens Creek...
![Page 1: Ethernet Over PCI Express Presented by Kallol Biswas NucleoDyne Systems, Inc. 19925 Stevens Creek Blvd Cupertino, CA.](https://reader036.fdocuments.us/reader036/viewer/2022062619/551637de550346a2308b62de/html5/thumbnails/1.jpg)
Ethernet Over PCI Express
Presented by
Kallol Biswas
NucleoDyne Systems, Inc.19925 Stevens Creek Blvd
Cupertino, CA
![Page 2: Ethernet Over PCI Express Presented by Kallol Biswas NucleoDyne Systems, Inc. 19925 Stevens Creek Blvd Cupertino, CA.](https://reader036.fdocuments.us/reader036/viewer/2022062619/551637de550346a2308b62de/html5/thumbnails/2.jpg)
Outline of Today’s Presentation
• Overview
• Highlights of PCI Express Protocol
• System to System communication using PCIe ( PCI Express protocol )
• Advantages and Issues
• Application Areas
![Page 3: Ethernet Over PCI Express Presented by Kallol Biswas NucleoDyne Systems, Inc. 19925 Stevens Creek Blvd Cupertino, CA.](https://reader036.fdocuments.us/reader036/viewer/2022062619/551637de550346a2308b62de/html5/thumbnails/3.jpg)
Overview
Traditional Use model
• Ethernet devices connect two or more computer systems.
• PCI Express bus links motherboard mounted peripherals or add-in devices.
New Use Model
• PCI Express switch connects two or more computer systems at system bus level.
![Page 4: Ethernet Over PCI Express Presented by Kallol Biswas NucleoDyne Systems, Inc. 19925 Stevens Creek Blvd Cupertino, CA.](https://reader036.fdocuments.us/reader036/viewer/2022062619/551637de550346a2308b62de/html5/thumbnails/4.jpg)
Multiple Computer Systems Connected through PCIe switch
Remote system’s memory is available to local system for load/store CPU instructions and DMA operations.
![Page 5: Ethernet Over PCI Express Presented by Kallol Biswas NucleoDyne Systems, Inc. 19925 Stevens Creek Blvd Cupertino, CA.](https://reader036.fdocuments.us/reader036/viewer/2022062619/551637de550346a2308b62de/html5/thumbnails/5.jpg)
New Use ModelA process writes to a remote process’ address space
![Page 6: Ethernet Over PCI Express Presented by Kallol Biswas NucleoDyne Systems, Inc. 19925 Stevens Creek Blvd Cupertino, CA.](https://reader036.fdocuments.us/reader036/viewer/2022062619/551637de550346a2308b62de/html5/thumbnails/6.jpg)
Inter process communication schemes
Direct Communication• One process read/writes information to remote process on a different system
No socket API is necessary Low latency data access
TCP/IP based Communication Ethernet frames are sent over PCIe Bus
Observation Low Latency and High Throughput Lower Power consumption and Low cost
![Page 7: Ethernet Over PCI Express Presented by Kallol Biswas NucleoDyne Systems, Inc. 19925 Stevens Creek Blvd Cupertino, CA.](https://reader036.fdocuments.us/reader036/viewer/2022062619/551637de550346a2308b62de/html5/thumbnails/7.jpg)
Highlights of PCI Express Protocol
Packet based Serial bus protocol Point to Point, bidirectional Effective data rate on each lane 2, 4 or 8 gbps Layer architecture
• Software, Transaction, Data Link, • Physical
Path based routing, globally addressable fabric QoS support Universal acceptance
![Page 8: Ethernet Over PCI Express Presented by Kallol Biswas NucleoDyne Systems, Inc. 19925 Stevens Creek Blvd Cupertino, CA.](https://reader036.fdocuments.us/reader036/viewer/2022062619/551637de550346a2308b62de/html5/thumbnails/8.jpg)
Two PCs are connected with an external PCIe cable
System to System Communication
Reference board with PCIe switch
Reference board with PCIe switchPCIe Cable
![Page 9: Ethernet Over PCI Express Presented by Kallol Biswas NucleoDyne Systems, Inc. 19925 Stevens Creek Blvd Cupertino, CA.](https://reader036.fdocuments.us/reader036/viewer/2022062619/551637de550346a2308b62de/html5/thumbnails/9.jpg)
Two Node Communication in PIO Mode
Processor
Root ComplexDDR3
Switch
FSB
System A
Processor
Root ComplexDDR3
Switch
FSB
System B
Store r3, <address in System B’s DDR>
TLP with modified header
ACK/NACK DLLP
Header Content of r3 ECRC LCRCSeq# EndStart
![Page 10: Ethernet Over PCI Express Presented by Kallol Biswas NucleoDyne Systems, Inc. 19925 Stevens Creek Blvd Cupertino, CA.](https://reader036.fdocuments.us/reader036/viewer/2022062619/551637de550346a2308b62de/html5/thumbnails/10.jpg)
TCP/IP Stack over PCIe Bus
System to System Communication Ethernet Over PCI Express
![Page 11: Ethernet Over PCI Express Presented by Kallol Biswas NucleoDyne Systems, Inc. 19925 Stevens Creek Blvd Cupertino, CA.](https://reader036.fdocuments.us/reader036/viewer/2022062619/551637de550346a2308b62de/html5/thumbnails/11.jpg)
Throughput in gbps
Test Results Throughput & CPU Utilization
![Page 12: Ethernet Over PCI Express Presented by Kallol Biswas NucleoDyne Systems, Inc. 19925 Stevens Creek Blvd Cupertino, CA.](https://reader036.fdocuments.us/reader036/viewer/2022062619/551637de550346a2308b62de/html5/thumbnails/12.jpg)
Netperf TCP_RR numbers RR/sec vs Packet size in bytes
Test Result - Latency
![Page 13: Ethernet Over PCI Express Presented by Kallol Biswas NucleoDyne Systems, Inc. 19925 Stevens Creek Blvd Cupertino, CA.](https://reader036.fdocuments.us/reader036/viewer/2022062619/551637de550346a2308b62de/html5/thumbnails/13.jpg)
Advantages
• Lower Power ConsumptionA 16 lane Gen2 switch consumes ~2.5W
• Lower Cost Cost is around $1 per len, x8 lane device costs ~$8• Savings in ecosystemLower cost for each components, clocks, connectors, cables and test equipments
• Lower latency and higher throughput
![Page 14: Ethernet Over PCI Express Presented by Kallol Biswas NucleoDyne Systems, Inc. 19925 Stevens Creek Blvd Cupertino, CA.](https://reader036.fdocuments.us/reader036/viewer/2022062619/551637de550346a2308b62de/html5/thumbnails/14.jpg)
Issues
• Length Limitation Max External cable length is approx 5m, with optical connector max length upto 100m
• Maximum number of nodes supported in a fabricMax number of Bus x device x func 256 x 32 x 8 = 65536 nodes, Gen3 supports much larger number
![Page 15: Ethernet Over PCI Express Presented by Kallol Biswas NucleoDyne Systems, Inc. 19925 Stevens Creek Blvd Cupertino, CA.](https://reader036.fdocuments.us/reader036/viewer/2022062619/551637de550346a2308b62de/html5/thumbnails/15.jpg)
Application Areas
• Replacement of parallel buses like VME (Versa Modular Eurocard bus) 9.97 Gbps over x4 PCIe switch vs 2.56 Gbps • Multi-node cluster development • NVRAM mirroring
• In trading systems or banking industry
![Page 16: Ethernet Over PCI Express Presented by Kallol Biswas NucleoDyne Systems, Inc. 19925 Stevens Creek Blvd Cupertino, CA.](https://reader036.fdocuments.us/reader036/viewer/2022062619/551637de550346a2308b62de/html5/thumbnails/16.jpg)
www.nucleodyne.com
• US System software services company
• Low Level Kernel & system software • Low level device drivers for storage and communication protocols • End to end system development
• processor customization• system board development• OS port or write new OS• custom application development
![Page 17: Ethernet Over PCI Express Presented by Kallol Biswas NucleoDyne Systems, Inc. 19925 Stevens Creek Blvd Cupertino, CA.](https://reader036.fdocuments.us/reader036/viewer/2022062619/551637de550346a2308b62de/html5/thumbnails/17.jpg)
NucleoDyne Systems, Inc.19925 Stevens Creek Blvd, Cupertino, CA
Thank You