Heterogeneous Integration for HPC and Data Centers

TWG Chair: Kanad Ghose, Ph.D.Professor of Computer Science, SUNY-BinghamtonPhD (CS), M.Tech (EE)and Site Director of Center for Energy-Smart Electronic Systems,a NSF Industry/University Collaborative Research Center

TWG Co-Chair: Dale Becker, Ph.D.Chief Engineer of Electronic Packaging Integration, IBM SystemsPhD (EE), MS (EE)Fellow IEEEChair, IEEE EPS TC-EDMS

6/5/2019

Scaling Trends

• Everything is getting intrinsically power and memory constrained; new applications are exacerbating the dependence on memory

• Accelerators are more energy-efficient than general-purpose designs

This TWG focuses on the need, requirements and solutions for realizing SYSTEM-IN-PACKAGE that integrate processing elements, accelerators, storage, IO etc. for the following market segments:• HPC Systems• Scale-out Systems• Data Centers• High-end Networking

Specifically, this TWG focuses on the system-level implications related to performance, power management, security, power distribution issues and others

Drivers for Heterogeneous Integration in the HPC/DC Segment

• Die cost per unit area increasing with nodeshrinks and refinements (hypernodes)

• Package IO, latency/BW and powerconstraining single-die substrates

• It’s all about moving data!• Emerging applications demand

domain-specific accelerators• Analytics/Intelligence on demand• Big data processing• IoTs• Blockchain processing

• Emerging processing paradigms, devices

Heterogeneous Integration as the Panacea• Overcomes the need to design large, low-yield, high cost dies (aka

“chiplets”)• Overcomes IO limitations of realistic size-limited single die/chiplet• Enables tight coupling of critical components such as

processors/accelerators and memory with shorter interconnections• Accommodates independently designed components to be integrated• Accommodates dies from diverse processes to be integrated – faster time-to-

market• Limits frequency and volume of off-package traffic, saving power

• Frequent communications localized to package - reduces power dissipation dramatically

Heterogeneous Integration Challenge Areas• On-package interconnections• Off-package interconnections• Signal integrity and distribution needs• Power distribution and regulation• SiP-level global power management• Security and reliability issues• Design tools• Supply chain

Interconnections within the Package: 1• Driven primarily by the need to tightly couple processing

logic and memory• Examples: GPU+HBM, Multicore CPU with vector units + HBM/HMC,

FPGA+HBM etc.• Now and near-term:

• Embedded interconnection bridges between dies nearadjacent edges of dies, such as Intel’s EMIB will play adominant role

• Ultra large interposers for 2.5D integration with single-axisreticle stitching will also be used

• Gradual transition to low-cost organic interposers• Limited use of 3D integration: limited logic layer at bottom of HBM,

vertically-stacked low-power SiPs with face-to-face integration or vias etc. Significant advantages against bridges:

• Reduces interconnection length from ~2 mm (PHY on logic to IO in HBM) to 50 um (die thickness), resulting in up to 90% power reduction on the links, cost benefits (lower area, bridge areas not needed) etc.

Interconnections within the Package - 2• 5 Years:

• Enhanced bridges – more lanes, finer pitch, smaller pillars, closer die-placements, aggressive ECC, improved reliability/manufacturability

• Additional and shorter lanes needed to accommodate higher bandwidth of HBM3 die (512 GB/sec. to 1TB/sec.) and beyond, to support fast connection to logic layer at bottom of HBM stack while reducing power/lane and permitting faster link rates

• Enhanced logic layer under HBM layers can implement local processing on wide words, serve as stream buffers, serve asswitch/router to other dies etc.

• Stacked SRAMs seeing increasing use.• SRAM layer below CPU die serves as L4 cache backed

up by HBM dies, SRAM die(s) on top of accelerator(including FPGA)

• Production units will see increasing use of non-volatilememory (such as MRAM, Optane) as one or more layersat the bottom of the HBM/HMC stack or stand-alone

• Higher lane count continues to be a solution for dealingwith SERDES limits of logic circuitry, dynamic link powermanagement used (lane count and link rate adaptation)

Interconnections within the Package - 3• 5 Years (continued):

• 3D integration becomes mainstream + a necessity for scaling up• Requires TSV diameter reduction while maintaining diameter-to-length aspect ratio at ~ 1:10

(from 5 um X 50 um at presentto ~ 2 um X 20 um in production)

• Addressing thermal challenges in 3D dies within SiPs:• Microarchitectural/Physical placement co-design to move away critical areas away from hot

spots in adjacent layers• Conformal lid design use becomes widespread:

cavities in spreader makes spreader contact hotareas directly

• Generous use of thermal vias/heatpipes• Coldplates used for high-power SiPs

• Increasing use of tessellated architectures for scaling up (e.g., tiling of 512-core processor into 8 dies of 64-cores connected using fast fabric in 2D, 3D or hybrid 2D/3D configurations, increased use of vertical partitioning

• Photonics interconnections begin to appear, albeit somewhat limitedly, with integrated photonics transceivers and polymer waveguides in the interposer

Interconnections within the Package - 4• Longer-term: (WORK IN PROGRESS)

• Limits of copper based connections reached• Complex symbol encoding, signal recovery and error correction all running out of steam• Interconnection power dominates and inhibits scaling

• Photonics interconnections becomes a major player for express links and possibly other types of links that span longer lengths

• Plasmonic devices are promising• Optical vias in 3D• AttoJoule photonics devices

• Practical microchannel-based cooling to support 3D deserves consideration

Package IO Interconnections• With SiPs and at high link rates distinction between intra and inter-package are

blurred• Now and near-term:

• Standards like PCIe, GenZ, Omni-Path and NVLink will continue to prevail• CXL (based on PCIe) offering memory coherence• Optical IO links may be used in some SiPs, mostly making use of separate optical TX/RX dies

in package, with or without use of WDM

• 5 Years and beyond:• PAM 4, PAM 8 symbol encoding becomes widespread as link rates exceed 50 Gbps• Crosstalk, losses need to be addressed aggressively – new materials, packaging advances

will be needed• Optical WDM IO links become more common to support increasing package IO rates• Traditional electrical links will also remain in use

• Added challenge beyond 5 years: As Vdd reduces to save energy at higher link rates: signal integrity/noise issues with lower voltage swings on signal lines

SiP Power Distribution• Scaling trends and device refinements indicate gradual and steady transition to lower

threshold devices - this drives up package level current to several 100s of amps• Ohmic losses on power delivery connections tend to dominate, local decoupling needs exacerbated

• Solution: higher voltage DC distribution, downconversion to required DC levels with regulatoron/near die (and on-die) within package

• 48V DC may be a good choice here as 48V DC poweredequipment is used widely in telco racks and an ecosystemof power supplies vendors exist

• GaN power devices used when higher voltage DC to chip-levelvoltage conversions at high-efficiency are needed

• GaN ecosystem seems to bedeveloping, GaN technology scaling well to meet needs

• Global power management scheme for package can control in-package regulators for DVFS• Embedded ultracapacitors on interposers for decoupling (ALD-deposited dielectric layers),

embedded inductors – innovations needed in both areas for widespread, reliable applications

SiP Power Distribution - 2• Concerns:

• Power supply noise close to chips/packages can cause malfunctions• Power devices inside the package are potential hot spots – adequate cooling

imperative• Switched-capacitor converters need efficiency improvement

• Potential solutions:• Noise reduction techniques:

• Use pre-regulation and lower switching frequencies (1.5 MHz or lower) – GaN devices support use of lower switching frequencies

• Use noise cancelation circuitry or EMI filters or advanced techniques

• Improving conversion efficiency: non-linear control, fewer stages, charge recovery

• Hot spots: adequate packaging/cooling solutions for thermal isolation AND heat removal – single and multi-phase, others including new materials

SiP Power Management• Smart global power management essential - open area for research and

development• Possible approaches: • Short-term:

• Reactive solutions based on sensing of power and/or temperature• Software solutions: algorithmic, programmer-driven• Traditional DVFS and extensions to as many dies as possible

• Advanced, 5 Years and beyond: • Proactive power budget allocation to subsystems within package and dynamic

budget allocation based on demand/needed response• Machine learning based adaptation

• Must be tied to the power delivery/regulation network• How much of the power management hooks will be exposed to software?

Potential security risks here for abuse (DoS, power viruses)

Enabling SiP Integration: Standards• Standards for integrating various chiplets, possibly from different

vendors/technologies emerging. Two notable efforts are:• CXL (Coherent eXpress Link)• ODSA (Open Domain-Specific Architecture) standard

• Active interposers – longer term approach• Similar standards for testing needed• Other standards possible in terms of power distribution, power

management, security enforcement etc.

Thermal Challenges• Need for spot cooling, dealing with dynamic/mobile hot spots

• Some hot spots may be relatively sizeable (such as GPU areas)• Package-internal DC-to-DC converter dies (GaN-based)• DRAM-stack neighborhood cooling critical

• 3D cooling is a challenge area – especially in DRAM/logic stacks, DRAM/SRAM stacks (heat pipes/thermal vias, intervening cooling layers, others)

• Air-cooled/Warm-water colplate cooled packages: to ~250+ W TDP• Liquid cooled (single and two-phase) packages: ~250 to 800 W+ TDP• New types of TIMs needed (ballistic TIMs, for example)• Lid/heat spreader solutions – passive (e.g., conformal lids), active spreaders• Potential for using intelligent/active package-level cooling• Potential for integration with SiP-level power management schemes

SiP Security Issues• Opening package exposes sensitive information and proprietary IP:

• Sensitive information decrypted for use from memory dies can be read out - again tamper-proof packaging helps (encrypting all internal links will be prohibitively expensive)

• Reverse engineering is easier as interconnections are exposed on opening package• Tamper-proof packaging and other destructive tamper-proofing techniques are useful

• One compromised die can bring entire SiP down:• Requires capability to isolate compromised device to permit graceful degradation• Same isolation/reconfiguration mechanism can be used to deal with faulty dies and extend the

useful lifetime of an expensive SiP• Test techniques can detect some of these in advance

• Need to detect compromises and deal with them dynamically - security monitor for interconnections, dynamic reconfiguration around isolated dies, test/validation algorithms

• TPM (Trusted Platform Module) like attestation can be done with separate logic/NVM inside the package – this is good!

• System power management hooks exposed to software can be abused – hardware-implemented overrides needed against these exploits (DoS, power viruses)

Some Applicable Metrics to Track Advances• Some metrics to track the overall objectives – all are quite well-known

and obvious but difficult to apply (next slide)• Server SiPs: performance scaling: Throughput doubles every 2 years, based

on baseline performance (Note: throughput, not latency)• HPC SiPs: FLOPs scaling following current trend line• Memory capacity per processing element doubles every 3 to 5 years• Link data rate scales commensurately with others to maintain overall

balance, especially to provide enough bandwidth and concurrency to couple processing elements with future generations of HBM

• Interconnection power budget scaling: few pJ/bit to AttoJ per bit in 10 years• Power density: commensurate power/mm2 to stay within package TDP

Problems with Formulating Metrics• The issues: it all depends on what’s inside the package and how big

the interposer is• Applicable metrics vary depending on contents (general-purpose HPC

multicore/HBM/HMC/GPUs/High-speed IO to ML accelerators combing HBM/HMC/Stacked-SRAM and accelerators/FPGAs to Graph processing with distributed SRAM/HBM to high-end network router switch, SiPs with analog neuromorphic accelerators etc. – too many choices/too many applicable metrics

• Interposer size limits are not clear – seems to be closely guarded• Possible way out: focus on micro-level metrics?

• Core-HBM, GPU-HBM, Cores-Accelerators, Cores-to-IO interface interconnection metrics

• Power density?• End-to-end energy per bit transported?

Needs and Metrics: 1Broader Issue Specific Needs, Potential Solutions:

5-Year HorizonSpecific Needs, Potential Solutions:

10-Year Horizon

Logic Integration: Processor/Logic Subsystem/AcceleratorIntegration

Tightly-coupled 2D tiled configuration; Wide-lane connection among adjacent dies• Silicon/EMIB bridges to support inter-tile

connections up to 4000 lanes at low latency• ECC + Symbol encoding on links• Large interposers realized with reticle stitching• Locally-synchronous, globally-asynchronous clock

Tightly-coupled 2D tiled and 3D configuration (stacking)• Dense vias implementing multiple 1000+ lane links• Up to 8000 low-latency lanes for 2D tiling• Plasmonic interconnections

Logic/DRAM Integration Interconnections to support up to 4000 lanes/256GBper sec. per HBM to accommodate HBM2 thru HBM 3• Silicon/EMIB bridge• Alternative imposers• DRAM stack on logic/SRAM layer implementing

memory acceleration artifacts and L4-Cache• Advanced symbol encoding on links + ECC

Interconnections to support HBM 3 and beyond with 4000-8000 lanes/ >512 GB/sec. per HBM; DRAM stacked over processing elements; SRAM stack on top of processing die;SRAM, DRAM stacks with processing element die(s)• Dense vias implementing multiple 1000 lanes• Combination of 2.5D and 3D subsystems• Photonic/Plasmonic interconnections

Logic/SRAM Integration SRAM stacks at edge with lane counts similar to DRAM;SRAM stack serving as cache for DRAM stack, external memory; Stacked SRAM for use by FPGA engines etc.• Silicon/EMIB bridge• HMC – future generations• Alternative interposers

SRAM stack on top of processing die; Distributed SRAM for supporting big data/ML/AI applications; SRAM, DRAM stacks with processing element die(s)• Dense vias implementing 4000+ lanes with low latency• Combination of 2.5D and 3D subsystems• Limited photonic connections/optical vias to SRAM stack

Needs and Metrics: 2Broader Issue Specific Needs, Potential Solutions:

5-Year HorizonSpecific Needs, Potential Solutions:

10-Year Horizon

Logic/NVRAM Integration

Needs and solutions parallel those for SRAM/DRAM depending on type of NVM

Needs and solutions parallel those for SRAM/DRAM depending on type of NVM

Package IO High bandwidth wide-lane IO channels;• Limited use of optical transceivers on high BW IO

links using integrated photonics TXRX die(s)• Aggressive signal equalization• Advanced symbol encoding• Integrated photonics component with high thermal

immunity• Limited number (2 to 16) of wavelengths on WDM

Multiple high-BW IO channels• AJ/bit photonic links• Advanced symbol encoding• Dense WDM• Advanced copper IO

SiP-level Power Management

Fine-grained DVFS control of processing, memory elements, package IO interface• Rich sensing• Distributed power management in package at

common voltage islands and within die• Application-specified power budgets for subsystems

within SiP

Advanced power-management of SiP• Power-budget assigned dynamically with ML controller with

fully-decentralized power controllers• Tight integration with in-package converters• Potential coordination with active package cooling system• Thermally-aware load shifting/distribution inside package

Needs and Metrics: 3Broader Issue Specific Needs, Potential Solutions:

5-Year HorizonSpecific Needs, Potential Solutions:

10-Year Horizon

Power Delivery to Package and Distribution Inside Package

Support for multiple voltage islands;Reduction of Ohmic losses in power delivery network –power limited to 200 to 250 W per package;Noise reduction in power delivery system;• Use of high-voltage (48 VDC) to package with few

DC-to-DC converters inside package• Use of noise reduction techniques based mainly on

passive components• Advanced inductors and capacitors• Advanced DC-to-DC converter designs: switched

capacitor converter with non-linear control and GaN power device(s), 2-stage conversion

Reduction of Ohmic losses in power delivery network – up to 250 to 800 W per package;Noise reduction in power delivery system;• Active coordination with power management system• Use of high-voltage (48 VDC) to package with more

distributed DC-to-DC converters inside package• Use of active noise reduction techniques• Advanced switched capacitor DC-to-DC converter design

using GaN power devices; 2-stage conversion; lower-frequency switching for noise reduction

• Advanced solutions for mitigating side channels based on power line noise and other EMF emissions

Security Needs IP Protection against reverse engineering/tampering;Potential information leakage via interconnection probing in opened package;• Tamper-proof package; self-destruction fuses;• Certified supply chain• Limited forms of link-encoding• EM shielding of radiating components inside

package to mitigate side channels• Static and run-time testing

Protection against compromised dies;• Active side channel mitigation techniques• Full-blown security management co-processor monitoring SiP• Isolation of compromised dies (also used for isolating faulty

dies)• Active protection against power viruses• Aggressive run-time test/diagnosis/repair

Needs and Metrics: 4Broader Issue Specific Needs, Potential Solutions:

5-Year HorizonSpecific Needs, Potential Solutions:

10-Year HorizonPackage Cooling Up to 250 W heat removal with air/warm water

coldplate cooling;Ability to deal with hot spots near power-conversion devices/specific logic/optical TXRX etc.• Potential need for thermal shielding• Conformal lids• Thermal vias• Coldplates• Other Potential Solutions: TBD

Up to 800 W heat removal with advanced cooling solutions;Potential need for significant thermal shielding;• Coordination with SiP power management system;• Heatpipes/dense thermal vias for stacked SiPs• Inter-layer cooling• Widespread use of liquid/2-phase cooling• Other Potential Solutions: TBD

Others: Alternative processing paradigms, analog component integration,

WORK IN PROGRESS WORK IN PROGRESS

Topics to be Addressed in the Next Version• Quantum computing and communications• Accelerators for big data and graph processing• Accelerators beyond MAC arrays for neural networks• Neuromorphic computing

Thanks to all of the members of this TWG:

Ivor Barber (AMD), Bill Bottoms (3MTS), Tahir Cader (HP), William Chen (ASEUS), Luke England (Global Foundries), Eric Eisenbraun (SUNYPoly), Ali Heydari (Rigetti), Rockwell Hsu (Cisco), Madhu Iyengar (Google), Michael Liehr (SUNYPoly), Ravi Mahajan (Intel), Michael Patterson (Intel), Gamal Refai-Ahmed (Xilinx), Tom Salmon (Semi), Lei Shan (IBM), Bahgat Sammakia(SUNY), Siamak Tavallaei (Microsoft)

* Lead contributors listed in boldface