Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media...

22
genzconsortium.org Gen-Z Technology: Enabling Memory Centric Architecture

Transcript of Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media...

Page 1: Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media Controller Gen-Z Logic Processor Gen-Z Logic Media Module Gen-Z Fabric SCM SCM SCM SCM Storage

genzconsortium.org

Gen-Z Technology: Enabling Memory Centric

Architecture

Page 2: Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media Controller Gen-Z Logic Processor Gen-Z Logic Media Module Gen-Z Fabric SCM SCM SCM SCM Storage

genzconsortium.orggenzconsortium.org © Gen-Z Consortium 2017 2

Why Gen-Z?

Page 3: Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media Controller Gen-Z Logic Processor Gen-Z Logic Media Module Gen-Z Fabric SCM SCM SCM SCM Storage

genzconsortium.orggenzconsortium.org © Gen-Z Consortium 2017 3

Why Gen-Z?

Page 4: Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media Controller Gen-Z Logic Processor Gen-Z Logic Media Module Gen-Z Fabric SCM SCM SCM SCM Storage

genzconsortium.org© Gen-Z Consortium 2017 4

Why Gen-Z?

genzconsortium.org

Big Data

AI

Machine Learning

Deep Learning

BI

Businesses’ Need to ‘Monetize’ Data

Page 5: Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media Controller Gen-Z Logic Processor Gen-Z Logic Media Module Gen-Z Fabric SCM SCM SCM SCM Storage

genzconsortium.orggenzconsortium.org © Gen-Z Consortium 2017 5

Why Gen-Z?

Page 6: Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media Controller Gen-Z Logic Processor Gen-Z Logic Media Module Gen-Z Fabric SCM SCM SCM SCM Storage

genzconsortium.org6© Gen-Z Consortium 2017genzconsortium.org

Challenge: Multiple Memory Types Need Support

Processor1 ns

SRAM5 ns

DRAM100 ns

Flash200,000 ns

Hard Drive10,000,000 ns Tape

40,000,000,000 ns

Block OperationsMemory Operations

• Byte Addressability

• High Bandwidth/Low Latency

• Simple & Efficient Load/Store operations

• Data Protection methods (RAID, EC)

• Device management: Hotplug-ability

• Rich Data Services (ie deduplication, replication, thin provisioning, etc)

Storage Class Memory (SCM)

Page 7: Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media Controller Gen-Z Logic Processor Gen-Z Logic Media Module Gen-Z Fabric SCM SCM SCM SCM Storage

genzconsortium.org7© Gen-Z Consortium 2017genzconsortium.org

0

500

1000

1500

2000

2500

3000

3500

4000

4500

2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020

# Processor Pins & DDR Channels

3CH

4CH

6CH

8CH

I1366 pins

I2011 pins

Courtesy: Ideas International

I3647 pins

I~ 4200 pins

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

2012(8)

2013(12)

2014(18)

2016(22)

2017(28)

2019(64)

GB/

s

Memory Bandwidth per Core

(Projected)

(~60)

Pin limitations lead to memory channel limitations that leads to memory bandwidth limitations.

Challenge: Memory Bandwidth Per Core Is Shrinking

Page 8: Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media Controller Gen-Z Logic Processor Gen-Z Logic Media Module Gen-Z Fabric SCM SCM SCM SCM Storage

genzconsortium.org

Power, Cooling, and Physical Limitations• CPU power consumption is increasing• DIMM power consumption is increasing• Number of memory channels is slowly increasing• Physical size of CPU is increasing• Physical size of DIMM is increasing• Speed of Memory Channel is increasing• Number of Cores in each Socket is increasing• Dataset Memory Requirements are increasing

CPU1 CPU2

Challenge: We Are Out Of Real Estate & Resources

genzconsortium.org © Gen-Z Consortium 2017 8

Page 9: Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media Controller Gen-Z Logic Processor Gen-Z Logic Media Module Gen-Z Fabric SCM SCM SCM SCM Storage

genzconsortium.org9

Storage NIC

x16

GPUCPU

DDR4

x16

HBM64 GB/s

Memory Bandwidth: 100 GB/s

HBM Memory Bandwidth:732+ GB/s

SAS Ethernet

0.0

1.0

2.0

3.0

4.0

5.0

6.0

2012(8)

2013(12)

2014(18)

2016(22)

2017(28)

2019(64)

GB/

s

PCIe Bandwidth per core

(Projected)

x16

• Cores have limited bandwidth as the number of cores increases

• Bottleneck for heterogeneous compute environment

PCIe

Challenge: IO Bandwidth Per Core is Shrinking

(~60)(~60)

genzconsortium.org © Gen-Z Consortium 2017

Page 10: Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media Controller Gen-Z Logic Processor Gen-Z Logic Media Module Gen-Z Fabric SCM SCM SCM SCM Storage

genzconsortium.org10© Gen-Z Consortium 2017genzconsortium.org

I/O

Communication at the speed of memory

What is a Memory Semantic Fabric?• A communication protocol that speaks the

same language the CPU speaks: load/store, put/get, and read/write

• Connectivity that extends beyond the server to the rack

Accelerators

FPGA GPU

CPUs

SoC SoC GPUFPGA

MemoryMemory

Memory Semantic Fabric

Memory Memory

Pooled MemoryNetwork Storage

What Can Address These Challenges?

A Memory Semantic Fabric!

Page 11: Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media Controller Gen-Z Logic Processor Gen-Z Logic Media Module Gen-Z Fabric SCM SCM SCM SCM Storage

genzconsortium.org11© Gen-Z Consortium 2017genzconsortium.org

Gen-Z: A New Data Access Technology

Accelerators

FPGA GPU

CPUs

SoC SoC GPUFPGA

Memory MemoryMemory

I/O

Pooled Memory Network Storage

Direct Attach, Switched, or Fabric TopologyGen-Z

Packetized Memory Semantic Protocol

High Bandwidth, Low Latency

Simplify HW/SW Boundary

Compatible & Economical

Universal democratized communication

Memory

Abstracted Hardware Interface

Page 12: Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media Controller Gen-Z Logic Processor Gen-Z Logic Media Module Gen-Z Fabric SCM SCM SCM SCM Storage

genzconsortium.org

Gen-Z Allows Memory Innovation

Processor

DRAM

DRAM

DRAM

DRAM

Memory Bus

Media Module

4-8 Memory Channels17-25 GB/s / Channel

288pins / DIMMSynchronous Interface

Med

iaCo

ntro

ller

12genzconsortium.org © Gen-Z Consortium 2017

Page 13: Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media Controller Gen-Z Logic Processor Gen-Z Logic Media Module Gen-Z Fabric SCM SCM SCM SCM Storage

genzconsortium.org

Gen-Z Allows Memory Innovation

Processor

DRAM

DRAM

DRAM

DRAM

Media Module

Gen

-Z L

ogic

Gen-Z MemorySemantic Fabric

Med

iaCo

ntro

ller

Gen

-Z L

ogic

2-8 High-speed Serial LinksLow Latency, High performance

Split Memory ControllerAsynchronous Interface

Processor is media agnostic13genzconsortium.org © Gen-Z Consortium 2017

Page 14: Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media Controller Gen-Z Logic Processor Gen-Z Logic Media Module Gen-Z Fabric SCM SCM SCM SCM Storage

genzconsortium.org

Gen-Z is Media Agnostic & Composable

14

Med

iaCo

ntro

ller

Gen

-Z L

ogic

Processor

Gen

-Z L

ogic

Media Module

Gen-Z Fabric

SCM

SCM

SCM

SCM

Storage Class Memory

Processor

Gen

-Z L

ogic

Media Module

Gen-Z Fabric GPU or

FPGA

Accelerator

Med

iaCo

ntro

ller

ProcessorMedia Module

SCM

SCM

SCM

SCM

Media Module

GPU or

FPGA

Gen

-Z S

wit

ch

Med

iaCo

ntro

ller

DRAM

DRAM

DRAM

DRAM

Media Module

Processor

CPU

CPU

Med

iaCo

ntro

ller

Media

Media

Media

Media

Media Module

Multiple resources enabled by Universal Interconnect

genzconsortium.org © Gen-Z Consortium 2017

Page 15: Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media Controller Gen-Z Logic Processor Gen-Z Logic Media Module Gen-Z Fabric SCM SCM SCM SCM Storage

genzconsortium.org

ServerServerServer

Gen-Z Creates More Real Estate• Gen-Z can be used to connect to devices inside and outside the server enabling a composable system

• Gen-Z’s pin efficient interface and ability to multiplex with PCIe pins allows for an overall increase in memory bandwidth

Processor

On Package Memory

DRAM, NVM, SCM

Memory Channels

Local Memory

Accelerators Gen-Z Fabric Switch

Rack-Level Pooled Memory

Pooled Accelerators

genzconsortium.org © Gen-Z Consortium 2017 15

Page 16: Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media Controller Gen-Z Logic Processor Gen-Z Logic Media Module Gen-Z Fabric SCM SCM SCM SCM Storage

genzconsortium.org

Fully Composable Through Disaggregation Using Gen-Z

genzconsortium.org © Gen-Z Consortium 2017 16

Page 17: Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media Controller Gen-Z Logic Processor Gen-Z Logic Media Module Gen-Z Fabric SCM SCM SCM SCM Storage

genzconsortium.orggenzconsortium.org © Gen-Z Consortium 2017 17

Example Gen-Z System with Gen-Z & PCIe® I/O

• Gen-Z systems may contain a mix of Gen-Z I/O components and PCI Express® I/O devices• PCIe worked immediately with unmodified OSs by appearing to SW as conventional PCI devices & bridges• Gen-Z is using a similar strategy via the concept of logical PCI devices (LPDs)

Host System

PCIe Root ComplexGen-Z Interface Logic

PCIe Switch

PCIe I/O Device

PCIe I/O Device

...

PCIe I/O Device

Gen-Z LPD Component

Gen-Z Switch

Gen-Z LPD Component

Gen-Z LPD Component

...

Gen-Z Switch

Gen-Z LPD Component

Gen-Z LPD Component

...

PhysicalConfiguration

Page 18: Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media Controller Gen-Z Logic Processor Gen-Z Logic Media Module Gen-Z Fabric SCM SCM SCM SCM Storage

genzconsortium.orggenzconsortium.org © Gen-Z Consortium 2017 18

I/O Component Sharing• With suitable Gen-Z fabric management SW,

a single Gen-Z I/O component can be shared among multiple Gen-Z systems

• Component sharing offers significant availability benefits, e.g., system failover for continued operation and avoidance of stranded resources

• Each Gen-Z system running an unmodified OS only “sees” LPDs assigned to it by Gen-Z fabric management SW

• Each Gen-Z system’s BDF space for LPDs is orthogonal to the BDF space in other systems

• A Gen-Z I/O component designed to be shared by multiple systems must ensure that the virtual device instances do not interfere with each other

• Gen-Z can deliver the major benefits of PCIe MR-IOV* with less cost & complexity

*MR-IOV: Multi-Root I/O Virtualization and Sharing

Page 19: Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media Controller Gen-Z Logic Processor Gen-Z Logic Media Module Gen-Z Fabric SCM SCM SCM SCM Storage

genzconsortium.org

More Gen-Z Benefits

genzconsortium.org © Gen-Z Consortium 2017 20

• Gen-Z’s 112GT/s Connector The connector supports vertical, horizontal, right angle and cabled

solutions Same connectors for memory, accelerators, I/O, etc. Gen-Z defined the connector and is donating it to SFF

• Leverage High Volume Industry PHYs Derivative of the IEEE 802.3 PHY PCIe PHY Lower development and product costs

• Unmodified OS Support Minimize cost & effort to adopt Gen-Z solutions

• Enables Right Sized Solutions Enables the right amount of compute, memory, storage,

acceleration, and networking to be applied to each workload

DDR25-50GB/s

Gen464GB/s

Gen-Z 4C100-448GB/s

Gen-Z 2C50-224GB/s

Gen-Z 1C25-112GB/s

Page 20: Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media Controller Gen-Z Logic Processor Gen-Z Logic Media Module Gen-Z Fabric SCM SCM SCM SCM Storage

genzconsortium.org

Open With Broad Industry Support

genzconsortium.org © Gen-Z Consortium 2017 21

Consortium Members

• Alpha Data• AMD• Amphenol• ARM• Avery Design

Systems• Broadcom• Cadence• Cavium• Cray• Dell EMC• Everspin• FIT• HP Enterprise• Huawei

• IBM• IDT• IntelliProp• Jabil• Lenovo• Lotes• Luxshare-ICT• Mellanox• Mentor Graphics• Micron• Microsemi• Molex• NetApp• Nokia• Numascale

• PLDA Group• Red Hat• Samsung• Seagate• SK hynix• Smart Modular• Spin Transfer

Technologies• TE• Tyco Electronics• VMware• Western Digital• Xilinx• Yadro

Page 21: Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media Controller Gen-Z Logic Processor Gen-Z Logic Media Module Gen-Z Fabric SCM SCM SCM SCM Storage

genzconsortium.org

Gen-Z Summary

• Scalable system interconnect and protocol

• Optimized for memory-semantic communications

• Breaks processor-memory innovation dependency

• Opportunity to simplify software overhead and complexity

• Unmodified OS support

• Common modular connector and mechanical form factors

• Visit the Gen-Z booth, #739, for more information and see the demonstrations

• Attend “Chat with the Experts” tonight to speak with Mike Krause about Gen-Z at the Fabrics Table

• Visit us at genzconsortium.org

genzconsortium.org © Gen-Z Consortium 2017 22

Page 22: Gen-Z Technology: Enabling Memory Centric … Gen-Z is Media Agnostic & Composable 14 Media Controller Gen-Z Logic Processor Gen-Z Logic Media Module Gen-Z Fabric SCM SCM SCM SCM Storage

genzconsortium.orggenzconsortium.org © Gen-Z Consortium 2017 23