PCI Express (PCIe) Overview - Flash Memory Summit
Transcript of PCI Express (PCIe) Overview - Flash Memory Summit
![Page 1: PCI Express (PCIe) Overview - Flash Memory Summit](https://reader031.fdocuments.us/reader031/viewer/2022021006/620388eada24ad121e4a8a37/html5/thumbnails/1.jpg)
PCI Express (PCIe)
Overview
Flash Memory Summit 2013
Santa Clara, CA
1
Peter Onufryk Sr. Director, Product Development
PMC-Sierra
![Page 2: PCI Express (PCIe) Overview - Flash Memory Summit](https://reader031.fdocuments.us/reader031/viewer/2022021006/620388eada24ad121e4a8a37/html5/thumbnails/2.jpg)
NVMe and PCIe Layering
NVMe
• Driver interface (registers)
• Queuing interface
• Command set
• Command processing model
PCIe
• Reliable memory read/write transactions
• Discovery and configuration
• Switching & routing
• Physical layer
Flash Memory Summit 2013
Santa Clara, CA
2
![Page 3: PCI Express (PCIe) Overview - Flash Memory Summit](https://reader031.fdocuments.us/reader031/viewer/2022021006/620388eada24ad121e4a8a37/html5/thumbnails/3.jpg)
PCIe Characteristics
Specification defined by PCI-SIG
• www.pcisig.com
Packet based protocol over serial links
• Software compatible with PCI and PCI-X
• Reliable in-order packet transfer
High performance and scalable from consumer to enterprise
• Scalable link speed (2.5 GT/s, 5.0 GT/s, 8.0 GT/s)
• Scalable link width (x1, x2, x4, …. x32)
Primary application is as an I/O interconnect
• Not a CPU interconnect
• Some multi-host applications (NTB, MR-IOV)
• Some outside the box applications (PCIe cable)
Flash Memory Summit 2013
Santa Clara, CA
3
![Page 4: PCI Express (PCIe) Overview - Flash Memory Summit](https://reader031.fdocuments.us/reader031/viewer/2022021006/620388eada24ad121e4a8a37/html5/thumbnails/4.jpg)
PCIe and Server Architecture
Flash Memory Summit 2013
Santa Clara, CA
4
*Source - Intel
![Page 5: PCI Express (PCIe) Overview - Flash Memory Summit](https://reader031.fdocuments.us/reader031/viewer/2022021006/620388eada24ad121e4a8a37/html5/thumbnails/5.jpg)
PCIe is Everywhere
Flash Memory Summit 2013
Santa Clara, CA
5
Blade Server
SERVERS
Mid-range / High End
COMMUNICATIONS
Metro Switch / Router Enterprise Switch / Router Wireless
STORAGE
RAID System SAN Switch NAS
CONSUMER
Small Business
Entertainment Imaging Mobile
![Page 6: PCI Express (PCIe) Overview - Flash Memory Summit](https://reader031.fdocuments.us/reader031/viewer/2022021006/620388eada24ad121e4a8a37/html5/thumbnails/6.jpg)
PCIe Fabric Topology
Flash Memory Summit 2013
Santa Clara, CA
6
Root Complex
DP DPDP
PCIe Switch
En
dp
oin
t
UP UP
DP DPDP
PCIe Switch
UP
DP DPDP
PCIe Switch
UP
DP DPDP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
![Page 7: PCI Express (PCIe) Overview - Flash Memory Summit](https://reader031.fdocuments.us/reader031/viewer/2022021006/620388eada24ad121e4a8a37/html5/thumbnails/7.jpg)
Data Transfers
Flash Memory Summit 2013
Santa Clara, CA
7
Root Complex
DP DPDP
PCIe Switch
En
dp
oin
t
UP UP
DP DPDP
PCIe Switch
UP
DP DPDP
PCIe Switch
UP
DP DPDP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UPE
nd
po
int
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
![Page 8: PCI Express (PCIe) Overview - Flash Memory Summit](https://reader031.fdocuments.us/reader031/viewer/2022021006/620388eada24ad121e4a8a37/html5/thumbnails/8.jpg)
PCIe and NVMe
Flash Memory Summit 2013
Santa Clara, CA
8
NVMe Host
NVMe
SSD
NVMe
SSD
NVMe
SSD
NVMe
SSD
NVMe
SSD
NVMe
SSD
NVMe
SSD
NVMe
SSD
Root Complex
DP DPDP
PCIe Switch
NVMe
SSD
En
dp
oin
t
UP UP
DP DPDP
PCIe Switch
UP
DP DPDP
PCIe Switch
UP
DP DPDP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
tUP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
![Page 9: PCI Express (PCIe) Overview - Flash Memory Summit](https://reader031.fdocuments.us/reader031/viewer/2022021006/620388eada24ad121e4a8a37/html5/thumbnails/9.jpg)
PCIe Layers
Flash Memory Summit 2013
Santa Clara, CA
9
TransactionLayer
Data LinkLayer
PhysicalLayer
TransactionLayer
Data LinkLayer
PhysicalLayer
PCI
Architecture
PCI
Architecture
PCIe Link
![Page 10: PCI Express (PCIe) Overview - Flash Memory Summit](https://reader031.fdocuments.us/reader031/viewer/2022021006/620388eada24ad121e4a8a37/html5/thumbnails/10.jpg)
Upstream Port
PC
Ie Lane
Downstream Port
PC
Ie Lane
PC
Ie Lane
PC
Ie Lane
PCIe Link
Physical Layer
Scalable Speed
• Gen1 – 2.5 GT/s
• Gen2 – 5.0 GT/s
• Gen3 – 8.0 GT/s
Scalable Width
• x1, x2, x4, x8, x12, x16, x32
Encoding
• 8b10b: 2.5 GT/s & 5.0 GT/s
• 128b/130b: 8 GT/s
Flash Memory Summit 2013
Santa Clara, CA
10
![Page 11: PCI Express (PCIe) Overview - Flash Memory Summit](https://reader031.fdocuments.us/reader031/viewer/2022021006/620388eada24ad121e4a8a37/html5/thumbnails/11.jpg)
PCIe Performance
Flash Memory Summit 2013
Santa Clara, CA
11
Generation Raw
Bit Rate
Bandwidth
Per Lane
Each Direction
Total x16
Link Bandwidth#
Gen 1 2.5 GT/s ~ 250 MB/s ~ 8 GB/s
Gen 2 5.0 GT/s ~500 MB/s ~16 GB/s
Gen 3 8 GT/s ~ 1 GB/s ~ 32 GB/s
Source – PCI-SIG PCI Express 3.0 FAQ
# Link bandwidth in each direction x 2 (full duplex)
![Page 12: PCI Express (PCIe) Overview - Flash Memory Summit](https://reader031.fdocuments.us/reader031/viewer/2022021006/620388eada24ad121e4a8a37/html5/thumbnails/12.jpg)
Data Link Layer
Primary Function
• Reliable exchange of Transaction Layer Packets (TLPs)
between the two components of a Link
Other Functions
• Initialization (flow control credits)
• Power management
• Track and report link state to transaction layer (i.e., DL_up &
DL_Down)
Flash Memory Summit 2013
Santa Clara, CA
12
![Page 13: PCI Express (PCIe) Overview - Flash Memory Summit](https://reader031.fdocuments.us/reader031/viewer/2022021006/620388eada24ad121e4a8a37/html5/thumbnails/13.jpg)
Transaction Layer
Primary Function
• Assembly and disassembly of
Transaction Layer Packets
(TLPs) for read and write
transactions
Other Functions
• Event signaling (e.g., interrupts,
power management, errors)
• Management of TLP credit
based flow control
Flash Memory Summit 2013
Santa Clara, CA
13
MR
d
Cpl
D
Cpl
D ... Cpl
D
Completer
Requester
![Page 14: PCI Express (PCIe) Overview - Flash Memory Summit](https://reader031.fdocuments.us/reader031/viewer/2022021006/620388eada24ad121e4a8a37/html5/thumbnails/14.jpg)
Address Spaces
Flash Memory Summit 2013
Santa Clara, CA
14
Address Space Function
Memory Data transfer to/from memory mapped locations
- 64-bit memory address space
I/O Data transfer to/from IO-mapped locations
- 32-bit I/O space
Configuration Device Function configuration & setup
- 16-bit configuration space
Message Event signaling & general purpose messaging
![Page 15: PCI Express (PCIe) Overview - Flash Memory Summit](https://reader031.fdocuments.us/reader031/viewer/2022021006/620388eada24ad121e4a8a37/html5/thumbnails/15.jpg)
Functions and Configuration Space
Function – an addressable entity in configuration
space
• Architecturally visible (i.e., can be discovered and configured)
• Capable of issuing requests and generating completions
Flash Memory Summit 2013
Santa Clara, CA
15
Bus FuncDev
8 5 3
Bus Func
8 8
Alternative Routing-ID Interpretation (ARI) IDTraditional ID
![Page 16: PCI Express (PCIe) Overview - Flash Memory Summit](https://reader031.fdocuments.us/reader031/viewer/2022021006/620388eada24ad121e4a8a37/html5/thumbnails/16.jpg)
Type 0 and Type 1 Functions
Flash Memory Summit 2013
Santa Clara, CA
16
Type 0
Function
Function 0
Type 0
Function
Function 0
Type 0
Function
Function 1
Type 0
Function
Function 2
Bus Bus
Single Function
DeviceMulti-Function Device
Type 1
FunctionPCI-to-PCI
Bridge
Bus
Bus
![Page 17: PCI Express (PCIe) Overview - Flash Memory Summit](https://reader031.fdocuments.us/reader031/viewer/2022021006/620388eada24ad121e4a8a37/html5/thumbnails/17.jpg)
Function Config. Space Registers
Flash Memory Summit 2013
Santa Clara, CA
17
PCI Compatible
Configuration Space
Header000h
0FFh
FFFh
PCI
Configuration
Space
PCI Express Extended
Configuration
Space
PCI Express
Extended Capability
MSI
Capability Structure
PCI Express
Capability Structure
PCI Express
Extended Capability
Advanced Error
Reporting
![Page 18: PCI Express (PCIe) Overview - Flash Memory Summit](https://reader031.fdocuments.us/reader031/viewer/2022021006/620388eada24ad121e4a8a37/html5/thumbnails/18.jpg)
PCIe Switches and I/O Fan-out
Flash Memory Summit 2013
Santa Clara, CA
18
Root Complex
DP DPDP
PCIe Switch
En
dp
oin
t
UP UP
DP DPDP
PCIe Switch
UP
DP DPDP
PCIe Switch
UP
DP DPDP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
![Page 19: PCI Express (PCIe) Overview - Flash Memory Summit](https://reader031.fdocuments.us/reader031/viewer/2022021006/620388eada24ad121e4a8a37/html5/thumbnails/19.jpg)
PCIe Switch
Flash Memory Summit 2013
Santa Clara, CA
19
PCIe Switch
UP
DP DPDP Type 1
FunctionPCI-to-PCI
Bridge
Type 1
FunctionPCI-to-PCI
Bridge
Type 1
FunctionPCI-to-PCI
Bridge
Type 1
FunctionPCI-to-PCI
Bridge
PCIe Switch
Physical View
PCIe Switch Logical View
UP
DP DP DP
Virtual PCI Bus
![Page 20: PCI Express (PCIe) Overview - Flash Memory Summit](https://reader031.fdocuments.us/reader031/viewer/2022021006/620388eada24ad121e4a8a37/html5/thumbnails/20.jpg)
Non-Transparent Bridge (NTB)
Flash Memory Summit 2013
Santa Clara, CA
20
Root
Complex
DP
PCIe Switch
UP
DP DPDP
En
dp
oin
t
UP
En
dp
oin
t
UP
Root
Complex
DP
PCIe Switch
UP
DPDP DP
En
dp
oin
t
UP
En
dp
oin
t
UPType 0
Function
Non-Transparent Bridge
Proprietary
Logic
Type 0
FunctionUPUP
![Page 21: PCI Express (PCIe) Overview - Flash Memory Summit](https://reader031.fdocuments.us/reader031/viewer/2022021006/620388eada24ad121e4a8a37/html5/thumbnails/21.jpg)
PCIe Multi-Path Usage Model
Flash Memory Summit 2013
Santa Clara, CA
21
PCIe
SSD
PCIe
SSD
PCIe
SSD
PCIe
SSDPCIe
SSD
PCIe
SSD
PCIe Switch
PCIe
SSD
PCIe
SSD
PCIe Switch
Host Host
Inerconnect
PCIe PCIe
![Page 22: PCI Express (PCIe) Overview - Flash Memory Summit](https://reader031.fdocuments.us/reader031/viewer/2022021006/620388eada24ad121e4a8a37/html5/thumbnails/22.jpg)
PCIe Overview Summary
PCIe is ubiquitous
PCIe provides a scalable interface for SSDs
• Scalable link width and speed
PCIe is not a bottleneck
• Highest performance standard I/O attach point
PCIe switches provide I/O fan-out
• Allows multiple SSDs to be connected to a Root Port
NVMe works within the standard PCIe framework
• Allows use of off-the-shelf Root Complexes and Switches
PCIe may be used to connect multiple hosts to an SSD
Flash Memory Summit 2013
Santa Clara, CA
22