The Smarter Car for Autonomous DrivingHeiko Joerg Schick

Chief Architect

Huawei Technologies

Requirements of automotive SoCs• High-performance application processor with 64-bit which execute multiple vision based applications.• Acceleration for algorithms such as Convolutional Neural Networks (CNN) are a must.• Support for multimedia interfaces (e.g. HDMI and MIPI D-PHY).

• Support for multiple cameras and radar sensors (e.g. 77 GHz long range radar, infrared, video and ultrasonic).• Up to 16 GB automotive-grade LPDDR4 memory.

• Extended system connectivity with Automotive Ethernet (including Audio Video Bridging, Time Sensitive Networking for multimedia traffic).• Additional interfaces (e.g. PCI Express, SATA, UARTs, SPI, QSPI, CAN or FlexRay).

• Support for cloud connectivity (Bluetooth Smart, WiFi, and 5G radio IC.• Support for secure boot secure identification and authentication.

• Support for encryption and decryption.• ISO26262 and AEC Q100 reliability qualification, and TS16949 quality management standard:

- Temperature range: -40 °C - 85/155 °C- Operation time: up to 15 years- Humidity 0% up to 100%- Tolerated filed failure rates: zero failure- Documentation “True” and in English / German- Supply: up to 30 years

L0 L1 L2 L3 L4 L5No Automation Driver Assistance

SOP 2013-2015PartialAutomation SOP 2016-2017

ConditionalAutomation. SOP 2018-2019

HighAutomation. SOP 2020-2023

Full Automation SOP >2025

Scenario • Autonomous Emergency Braking• Lane Keep Assist• Auto High Beam• Adaptive Cruise Control• Cross Traffic Alert• Surround View

• Lane Change Assist• Lane Centering• Advanced Parking Assist• Traffic Jam / Highway Assist• Advanced Emergency Braking• Adaptive Cruise Control with LKA• Rush Hour Pilot

• Automated Lane Change• Traffic Jam Pilot• Automated Parking• Highway Pilot

• Piloted Highway Driving• Geo-fenced City Pilot• Unattended Valet Parking• Mobility on Demand

Auto Pilot / Driverless Driving

Hardware 1-6 Sensors+ Optional Control Unit• Mono-Vison• Mid Range Rader• 2 Front Corner Radars• 2 Rear Corner Radars• ADAS ECU1,500 DMIPS6 MB RAM

2-10 Sensors + Control Unit• Mono-Vision• 2 Front Corner Radars• Mid Range Radar• 2 Rear corner Radars• Far Infrared Camera• ADAS ECU• E-Horizon• Rear/Surround View• Driver Monitoring• HD Map3,000 DMIPS 16 MB RAM 25W

>15 Sensors + Control Unit (incl. AI) and Driver Monitoring• 4 Corner Radars• Stereo Vision• Mid Range Radar• Far Infrared Camera• ADAS ECU• Mono-Vision Rear• V2X• LiDAR• HD Map• E-Horizon• Driver Monitoring System• Rear/Surround View>40,000 DMIPS >25 TOPS512 MB – 3 GB RAM 200W

25 Sensors + Control Unit (incl. AI) and Driver Monitoring• Stereo-Vison, Long Range Radar• 4 Corner Radars, Satellites• LiDAR Front, Mono-Vision Rear• Driver and / or Passenger Monitoring System• HD Map• V2X• Surround View• Far Infrared Camera• Mid-Range Side Sensors• AD ECU260,000 – 845,000 DMIPS>300 TOPS 32 GB RAM 600W

Software 40+ Features• Classic AUTOSAR

50+ Features• Classic + Adaptive AUTOSAR• POSIX Operating System

55+ Features• Classic + Adaptive AUTOSAR• POSIX Operating System

60+ Features• Adaptive AUTOSAR• POSIX Operating System

Safety concept Fail-safe Fail-operationalNo intervening vehicle system active.

Vehicle assisted longitudinal and lateral control.

Vehicle assisted longitudinal and lateral control (for a period of time and/or in specific use cases).

System as longitudinal and lateral control in specific use cases (for a period of time).

System has longitudinal and lateral control in a specific use case. Recognize its performance limits and requests driver to resume control with sufficient time margin

System can cope with all situations automatically during the entire journey. Driver does not monitor the system.

E/E architecture • Each function has his ECU with functional integration

• Central Domain ECUs• Central cross domain

ECUs

• Central Domain ECUs• Central cross domain ECUs

• Central Domain ECUs• Central cross domain ECUs

• Central cross domain ECUs• Zone oriented architecture and

vehicle control computer

• Zone oriented architecture and vehicle control computer

• Zone oriented architecture and vehicle control computer

Global adoption rate

2017 :2020 :2023 :

1%8%

19%2020 :2023 :

2%6% 2023 : 3%

Different sensor locationsHigh-level sensor fusion Low-level sensor fusion

• High-level fusion systems draw man decisions at early stages on an incomplete knowledge basis

• Some of the incorrect decisions cannot be correct at alter steps

• Decisions are drawn at the very last processing step; no feedback loops

• Need for central high performance unit

Source: Autonomous Driving – A mobility revolution, Helge Neuner, Head of Automated Driving for Group Research, Volkswagen AG

Radar

Camera

Lidar

Radar

Camera

Lidar

Seg

Seg

Seg

Fusion Fusion Seg

Strongest specialized cores, hundreds of cores, stacked memory with highest bandwidth, strongest IO interface à 16 PCIe 3.0 lanes, special interconnect for accelerator to accelerator communication, acceleration for artificial intelligence à training and inference, acceleration for linear algebra, strongest vector units, highest energy consumption à 300W.

Stronger general cores, tens of cores à 12 – 24 cores, standard memory interfaces with high memory bandwidth, strongest IO interfaces à 48-128 PCIe 3.0 lanes, strong vector units, high energy consumption à 120-170W.

Medium general cores à 8 – 16 cores, standard memory interface with medium bandwidth, strong IO interface à 16 PCIe 3.0 lanes, low energy consumption à10 – 30W.

Strong general cores, tens of cores à 48 - 64, standard memory interfaces with high memory bandwidth, strongest IO interfaces à 48-128 PCIe 3.0 lanes, strong vector units in high-end version, integrated controllers à USB, SATA, cryptography, integrated network àStandard Ethernet and RoCEE, medium energy consumption à 50-60W.

Medium general cores à 12 – 24, specialized engines à [GPU, image, vision, signal processing, artificial intelligence, security and compression], integrated network à CAN/CAN-FD and Automotive Ethernet, integrated camera interfaces à 8-12, integrated display interface à 3 – 6, automotive functional safety à [dual execution including lockstep operation, functional diversity, build-in self test] àASIL B – ASIL D, lower power mode à [adaptive voltage scaling, dynamic voltage and frequency scaling, power and power gating], high energy efficiency à 15W – 60W.

Weak to medium general cores à 8, very specialized engines à[GPU, image, vision, signal processing and artificial intelligence], high memory bandwidth, strongest IO interface à 16 PCIe 3.0 lanes, integrated network à CAN/CAN-FD and Automotive Ethernet, integrated camera interfaces à 12-18, automotive functional safety à [dual execution including lockstep operation, functional diversity, build-in self test] à ASIL B – ASIL D, highest energy efficiency à 5W – 30W.

IT & Cloud Embedded & Edge Automotive

Accelerators

CPUs

IntegratedSystems

Smart Devices

Cloud Embedded

Specialization in vertical industry HighLow

Scalable across devicesDevice Edge Cloud

Earphone Always-on Smartphone Laptop IPC Edge Server Data Centre

Compute 20 MOPS 100 GOPS 1-10 TOPS 10-20 TOPS 10-20 TOPS 10-100 TOPS 200+ TOPS

Power budget 1 mW 10 mW 1-2 W 3-10 W 3-10 W 10-100 W 200+ W

Model size 10 KB 100 KB 10 MB 10-100 MB 10-100 MB 100+ MB 300+ MB

Latency < 10 ms ~10 ms 10-100 ms 10-500 ms 10-500 ms ms ~ s Ms ~ s

Inference? Y Y Y Y Y Y Y

Training? N N Y Y Y Y Y

Ascend-SKU Nano Tiny >Lite Mini Mini Multi-Mini Max

Unified and scalable HW architectureScalable compute:• Scalable cube: 16x16xN， N=16/8/4/2/1

• Multiple precision: int8/int32/FP16/FP32

• Multiple Compute units: Tensor/Vector/Scalar

• Current control in picoseconds

• Hardware-assisted task scheduler

Scalable architecture:

• Dedicated & distributed, tiling-friendly, explicit memory design

Scalable on-chip interconnection• Ultra-high bandwidth mesh network

FHD Video ImageCodec Peripherals/IO DDR/HBM

SystemCache/Buffer

ARM

CubeLSU

Cache/Buffer

Vector

Scalar

N

Unified and versatile SW architecture

CANN(Compute Architecture for Neural Networks)

Ascend

All Scenarios

Full Stack

AI Applications

Application Enablement

Framework

Chip Enablement

IP & Chip

Application enablement:• Full-pipeline services (ModelArts), hierarchical APIs,

and pre-integrated solutions

MindSpore:• Unified training and inference framework for device,

edge, and cloud (both standalone and cooperative)

CANN:• Chip operators library and highly automated

operators development toolkit

Ascend:• AI IP and chip series based on unified scalable

architecture

Industrial IoT DeviceEdge ComputingEdge ComputingPrivate CloudPublic CloudConsumer Device

Ascend-MaxAscend-MiniAscend-LiteAscend-TinyAscend-Nano

PaddlePaddlePyTorchTensorFlowMindSpore …

ModelArts

General APIs Advanced APIs Pre-integrated SolutionsHiAI Service

HiAI Engine

Next steps: Deep sematic scene understanding

Challenges: Neural hacking

Call to action

Algorithmic pattern view

Scenarios Key technologiesHardware | Software | Connectivity Safety concept E/E architecture Business

Runtime / OS / middleware view

Performance Cores Safety Cores

Safety OSPOSIX OS

Adaptive AUTOSAR Classic AUTOSAR

Source: Self-Driving Vehicles That (Fore) See, Dariu M. Gavrila, Intelligent Vehicle, TU Delft

Functional view

Sense

Think

Act

Safety viewISO 26262 ASIL level A – D

FusionASIL B

LocalizationASIL B

CheckASIL D

Trajectory planningASIL B

ActuationASIL D

1

HUAWEI | GERMAN RESEARCH CENTER

Source: Bosch, BMW, Frost & Sullivan, Euro NCAP, Gartner, IHS, NXP, Strategy Analytics, Visteon, Veoneer

L0 L1 L2 L3 L4 L5No Automation Driver Assistance

SOP 2013-2015PartialAutomation SOP 2016-2017

ConditionalAutomation. SOP 2018-2019

HighAutomation. SOP 2020-2023

Full Automation SOP >2025

Scenario • Autonomous Emergency Braking• Lane Keep Assist• Auto High Beam• Adaptive Cruise Control• Cross Traffic Alert• Surround View

• Lane Change Assist• Lane Centering• Advanced Parking Assist• Traffic Jam / Highway Assist• Advanced Emergency Braking• Adaptive Cruise Control with LKA• Rush Hour Pilot

• Automated Lane Change• Traffic Jam Pilot• Automated Parking• Highway Pilot

• Piloted Highway Driving• Geo-fenced City Pilot• Unattended Valet Parking• Mobility on Demand

Auto Pilot / Driverless Driving

Hardware 1-6 Sensors+ Optional Control Unit• Mono-Vison• Mid Range Rader• 2 Front Corner Radars• 2 Rear Corner Radars• ADAS ECU1,500 DMIPS6 MB RAM

2-10 Sensors + Control Unit• Mono-Vision• 2 Front Corner Radars• Mid Range Radar• 2 Rear corner Radars• Far Infrared Camera• ADAS ECU• E-Horizon• Rear/Surround View• Driver Monitoring• HD Map3,000 DMIPS 16 MB RAM 25W

>15 Sensors + Control Unit (incl. AI) and Driver Monitoring• 4 Corner Radars• Stereo Vision• Mid Range Radar• Far Infrared Camera• ADAS ECU• Mono-Vision Rear• V2X• LiDAR• HD Map• E-Horizon• Driver Monitoring System• Rear/Surround View>40,000 DMIPS >25 TOPS512 MB – 3 GB RAM 200W

25 Sensors + Control Unit (incl. AI) and Driver Monitoring• Stereo-Vison, Long Range Radar• 4 Corner Radars, Satellites• LiDAR Front, Mono-Vision Rear• Driver and / or Passenger Monitoring System• HD Map• V2X• Surround View• Far Infrared Camera• Mid-Range Side Sensors• AD ECU260,000 – 845,000 DMIPS>300 TOPS 32 GB RAM 600W

Software 40+ Features• Classic AUTOSAR

50+ Features• Classic + Adaptive AUTOSAR• POSIX Operating System

55+ Features• Classic + Adaptive AUTOSAR• POSIX Operating System

60+ Features• Adaptive AUTOSAR• POSIX Operating System

Safety concept Fail-safe Fail-operationalNo intervening vehicle system active.

Vehicle assisted longitudinal and lateral control.

Vehicle assisted longitudinal and lateral control (for a period of time and/or in specific use cases).

System as longitudinal and lateral control in specific use cases (for a period of time).

System has longitudinal and lateral control in a specific use case. Recognize its performance limits and requests driver to resume control with sufficient time margin

System can cope with all situations automatically during the entire journey. Driver does not monitor the system.

E/E architecture • Each function has his ECU with functional integration

• Central Domain ECUs• Central cross domain

ECUs

• Central Domain ECUs• Central cross domain ECUs

• Central Domain ECUs• Central cross domain ECUs

• Central cross domain ECUs• Zone oriented architecture and

vehicle control computer

• Zone oriented architecture and vehicle control computer

• Zone oriented architecture and vehicle control computer

Global adoption rate

2017 :2020 :2023 :2025 :

1%8%

19%28%

2020 :2023 :2025 :

2%6%

12%2023 :2025 :

3%4%

Semiconductor value per car

$50 $150 $550 $1,150 $1,800

Total system value per car

$100-500 $500-800 $1,500-2,000 $4,000-10,000

ConnectivityC4 C5+ 4G-LTE Advanced, V2X Security

Fully Connected Car

4G-LTEAM/FM Digital | WiFi, BT

5GAM/FM Digital | WiFi, BT, NFC

Improved passenger car safety (Vision Zero / Euro NCAP)Primary safety scenario

• Autonomous Emergency Braking • Autonomous Emergency Braking• AEB – Junction and Cross

Traffic Assist• AEB – Head-on• Driver Monitoring• Pre Sense Side

• Autonomous Emergency Braking• AEB – Junction and Cross Traffic

Assist• AEB – Head-on• Driver Monitoring• Pre Sense Side• Automatic Emergency Steering• V2X

• Autonomous Emergency Braking• AEB – Junction and Cross Traffic

Assist• AEB – Head-on• Driver and Occupants Monitoring• Pre Sense Side• Automatic Emergency Steering• V2X

• Autonomous Emergency Braking• AEB – Junction and Cross Traffic

Assist• AEB – Head-on• Occupants Monitoring• Pre Sense Side• Automatic Emergency Steering• V2X

Secondary safety scenario

• Child Presence Detection • Child Presence Detection• Pedestrian and Cyclist Safety

• Child Presence Detection• Pedestrian and Cyclist Safety

• Child Presence Detection• Pedestrian and Cyclist Safety

Detailed Description

Target Critical corner cases Proposed action Timeframe (for legislation) Sensors Sensor fusion

Driver Monitoring Mitigate the very significant problems of driver distraction and impairment through inexperience, alcohol, drugs, etc.

• Give appropriate warning• Speed limitation• Initiate safe evasive manoeuvre• Limp home mode• Increased sensitivity of electronic

stability control, lane support, speed, etc.

2020

Autonomous Emergency Steering

2020, 2022

Autonomous Emergency Braking

Address cross-junction, head-on and reversing accidents, etc.

• Detect the presence of persons behind the car

• Detect crossing accidents of running a red light, lack of visibility, driver inattentiveness or speeding

• Warning• Initiate braking• Preventing acceleration

2020, 2022

Pedestrian and Cyclist Safety

• Occluded traffic participants• Running pedestrian parked cars• Fall over pedestrian towards road• Pedestrians at night time• …

2022

Child Presence Detection

Monitor a child’s presence in the vehicle to alert if the situation becomes dangerous.

• Alert car owner or emergency services

2022

V2X Vehicles exchanging data with each o the and the infrastructure.

• Transmit and receive messages like “emergency brake light”, “motorcycle is approaching” or “road work ahead”

2024

Automated driving assessment based on functionalities

Speed range [km/h] Type of road Level of automation Examples of required emergency functions

Parking 0 – 10 • Parallel parking• Perpendicular parking

Assisted Automated • AEB City• AEB Pedestrian

City Driving 5 – 255 – 50

• City roads (crossings, traffic lights, etc.)

• Any type of lane marking

Assisted Automated • AEB City• AEB Pedestrian• AEB Cyclist• Speed Assist System• Lane Support System

Inter-urban driving 0 – 80 • Fully marked lane• Single lane marking• No lane marking

Assisted Automated • AEB City• AEB Pedestrian• AEB Cyclist• Speed Assist System

Traffic jam (inter-urban & highway)

0 – 60 • Fully marked lane• (Single lane marking)• (No lane marking)

Assisted Automated • AEB City• (AEB Pedestrian)• (AEB Cyclist)• Lane Support System

Highway driving 50 – 130 • Fully marked lane Assisted Automated • AEB Inter-urban• Speed Assist system• Lane Support System

Need and value for various levels of autonomous driving

Version 3