Data Plane Acceleration Open Source Margaret T. Chiosi
AT&T Labs Distinguished Network Architect
Open Platform for NFV – OPNFV President (Linux Foundation )
Bob Monkman, ARM
Keith Wiles, Intel
Uri Elzur, Intel
© 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change
2007 2008 2009 2010 2011 2012 2013 2014
2
CUSTOMER DEMAND
© 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change
Faster
Highly Secure
More Flexible
More Reliable
Easier to Scale
Cheaper Agile
RETHINKING THE NETWORK
© 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change
3
4 © 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change
VIRTUALIZE AND CONTROL
Business Internal & External
Voice
Strategic Data
Mobility Video
Hardware Platform
AT&T Integrated Cloud Platform
Software Platform
All Legacy non-D2.0 WAN, Access, Cloud, Sites, etc.
5 5 5
Infrastructure • Infrastructure Controllers • IaaS • Compute, Storage,
Network • Hypervisor (KVM, ESXi)
Network • Virtual Network Functions
(e.g. vEdge, vCE) • SDN/Network Controllers
(Packet, Optical, Overlay)
Application • Reusable platform capabilities • Modular virtual functions &
apps, XaaS • Application Controllers • Common Data Repositories • Cloud Software Elasticity
Legacy Base Infrastructure
AT&T Finished Products
Scope of Domain 2.0 N
ot
D2
.0
ECOMP • Overall AIC control,
orchestration, mgt, policy • SD&C, MSO, A&AI, DCAE • Control loop pattern • Overall platform & app
support components
Do
mai
n 2
.0
Next Gen WAN • Packet Optical White Boxes
Physical Infrastructure Assets • Compute & Storage • Site commons (power, cooling,
racks, etc.)
Next Gen Access • Wireline Broadband White Boxes
(GPON, G.Fast) • Wireless Broadband White Boxes
(xRan, etc.)
Next Gen LAN/Underlay • Local Network and Fabric • Topology scaling
Programmable platform with API based automation, analytics & security throughout
Common / Reusable in Support of AT&T Finished Products & All Other Workloads
------------------------------------------------------------------------- Loose Coupling ---------------------------------------------------------------------------
Includes All Workloads
© 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change
VPN Connectivity for Virtual Private Clouds – Target
6
© 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change
Virtual Private Data Centers
VPN-1
VPN-2
CBB
RR
Customer-1 Sites
Customer-2 Sites
VRF VRF
IRB I-PE/ I-CE
Service Orchestrator
IRSCP
Subnets Subnets
Subnets Subnets
Mo
bile
an
d B
road
ban
d
Inte
rnet
Acc
ess
VM
vSwitch/Rtr
Network Orchestrator
VM
VM VM
Domain 2.0 – 3 Pillars, 1 platform, 1 vision
© 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change
7
New Paradigms for AT&T and Telecommunications
NFV
SD
N
ECO
MP
AT&T integrated Cloud
Customers
Cycle Time
Composition
Efficiency
SDN NFV
eCOMP
People Process
Culture
SDN NFV
eCOMP
Business Programs Voice, Strategic Data, Mobility, Video
AIC Request Interface
AT&T Integrated Cloud High-level Architecture
AT&T Technical Centers (IT & Mobility Centers, U-verse, Central Offices)
Shared Server, Storage and Network
Virtual Machines
Virtualization Abstraction (Openstack)
Network and Application Orchestration
AIC
AP
I In
terf
ace
Laye
r
Platform Business and Network Applications, Services
or Functions. Examples: • OS Image • DB • Web Services • vRouter • SDN • DNS • FW • Network on Demand / NoD
Policies and analytics defined for each of the Applications/Virtual Network Functions run within a common control loop for policy, orchestration and management.
Commodity, white box vendor agnostic equipment and software.
Handles the request(s). This includes workflow assemble of AIC services and resources, configuration of end services unique to the request as needed and sends completion confirmation back.
Handles the provisioning and assembly of cloud services and lower level resources.
Notes: 1. APIs are used between layers. 2. APIs are used between services and components within a layers 3. This architecture includes Opensource, vendor and AT&T code.
Controls and manages the AIC infrastructure resources.
Containers Bare metal servers
© 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change
8
AT&T Integrated Cloud Distribution Choices
© 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change
Virtualization of Home and Enterprise Networks
Legend VNF Hardware resources
Hardware resource pool
BS BS
CO
Virtualization of CDNs
Virtualization of Base Stations (vBS)
Virtualization of Fixed Access
Virtualization of Mobile Core/ IMS
HW HW HW HW
CSCF SGW CSCF MME
DNS DHCP SGW SGW Firewall
HW HW HW HW
MME CSCF LB DHCP
HW HW HW HW
CSCF PGQ App
Server App
Server App
Server
HW HW
RGW NAT STB
HW HW
CDN 1 CDN 2
HW HW
vBS LTE
vBS 3G
vBS 2G
vBS WiMax
HW HW
C-plane ADSL
C-Plane VDSL
C-Plane ITU-T/G
HW
Big Deployments / Low Distribution (<50 sites)
• Centralized Functions
• Large scale facilities, e.g., compute centric
National
Medium Deployments / Distribution (300-600 sites)
• Regional/metro functions
• Hub locations, proximity centric
Regional/Metro
Small/Medium Deployments / Large Distribution (5000+ Sites)
• End Office, LATA, and MOW functions
• Access hand-off points, highly proximity centric
End Office
Small / Very Small Deployments, Extreme Distribution (100K to 100M)
• Remote terminal, campus/venue
• Customer premises CPE, Customer devices
Remote
Trend to distribute functions towards the edge to reduce cost of transport, latency, concentration/disaster risk
Trend to centralize functions into the network to reduce Access & CPE TCO
© 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change 9
10 © 2014 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. AT&T Proprietary (Internal Use Only) Not for use or disclosure outside the AT&T companies except under written agreement.
10
Transformation: Converging to Single, Common Infrastructure
H
H
100’s+ to 1000’s
© 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change
Broad range of locations:
• Data Center
• Central Offices
• Huts
• Customer Premises
Therefore broad range of servers based on different chips – Intel, ARM, AMD, Power….
Broad range of applications – Networking, Web, IT, Video, Voice, …
Networking requires high throughput techniques
Expediency in solving the throughput challenges
Platform Requirement
© 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change
11
Application Requirement in Virtualized World
12 © 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change
Intel, ARM, Power, MIPS & Other Platforms
Linux
Hypervisor
V N F
V N F
V N F
Need one framework to support VNFs running on different platforms • Today there are two leading proposals
with similarities, but also fundamental differences
• Requires collaboration to converge and support multiple use cases
OVS/R
V N F
ODP ODP
NIC SoC
DPDK DPDK
Bring Data Plane to NFV Applications via DPDK
13
• Open-source (BSD license) community project 5+ years, current version is 2.1 – http://dpdk.org All code is completely Open Source from 1.2 release (Sept 2012 01.org) Multi-architecture: x86, IBM, Freescale, EZChip(Tilera) support Supports KVM, VMWare and HyperV plus Linux Containers
• DPDK has a large application install base and included in Linux Distro’s CentOS, Ubuntu, Red Hat, …(Fedora)
Included in frameworks like VirtIO/Vhost and OpenvSwitch Integrated into many projects: OVS, mTCP, Pktgen, xdpd, NetVM, MICA, ….
• Encompasses legacy platforms and newer acceleration platforms Common API to shield the VNF developer from the hardware differences No compromise in performance for DPDK or ODP/SoC devices Supports a common binary for NFV platforms
• Scalable solution to meet different NFV use cases VM to VM traffic as an example via OVS/DPDK
• Hardware acceleration complementing software for consistent VNF services
© 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change
13
SUPPORTING ODP IN THE DPDK FRAMEWORK
14 © 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change
DPDK – Architecture
DPDK-AE (Acceleration Enhancements)
DPDK – API
Software
Hardware
SoC SDK
ODP PMD
external memory manager 3rd Party
VNF Application
Crypto Device + others
Simple model for SOC integration
Ethernet Device
ODP PMD: Poll Mode driver model for ODP SoC devices Provides a clean integration of SoC/ODP via a PMD in DPDK
• Hardware abstraction in DPDK is at the PMD layer
• DPDK-API: A generic API extended to support SoCs
– DPDK provides a two layer device model to support many devices at the same time/binary
– Need to enhance DPDK with some SoC specific needs
– DPDK runs on all platforms Intel, ARM, MIPS, Power, …
– Large application supported features Hash, Rings, lpm, …
• ODP-PMD: Poll Mode Driver model for ODP
– Retains ODP’s current code base
• Unifies DPDK and ODP to create one solution for NFV
FreeScale, EZchip,
Broadcom, AMD
Open Data Plane API • General data plane API to enable networking applications to
accelerate across most any silicon on any architecture
• Open support for ARM, x86, MIPS, Power 7, Other Architectures
• Structured to enable future innovation
• Lightweight abstraction preserves performance without prescribing lower –level processing structure
• Unique, Direct Access and Management of HW accelerators
• Supports optional schedulers in an event driven model to provision easy management and dynamic traffic load balancing
• Vendor run-time environments sit underneath to preserve vendor optimizations
ODP embraces and extends existing, vendor-optimized RTEs, enabling an efficient, truly cross-platform standardized data plane programming interface
ODP is more than a packet
processing framework- It is a
Data Plane Application-level
Programming Interface
© 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change
15
Community of ODP Applications
ODP Application
ARM MIPS or
ARM
ODP API Layer
QorIQ – DPAA (Power or ARM)
Application can run in simple Polled Mode (DPDK Mapping) or Advanced Event Driven Model for maximum Multicore Scalability
SDK N Cavium
SDK Freescale
SDK DPDK
X86
ODP - Application View ODP demonstrated to run at negligible overhead on several Platform SDKs Intel x86/DPDK a work in progress. Currently achieve <2% overhead Developers can always to choose to use underlying interfaces- at expense of portability
© 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change
16
Applications in a Container World
17 © 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change
Intel, ARM, Power, MIPS & Other Platforms
Linux
V N F
V N F
V N F
Need one framework to support applications running on different container systems. As both DPDK and ODP run on the same platforms which interface is to be used?
NIC
V N F
ODP ODP
SoC
DPDK DPDK
OVS/R
DPDK ODP
Different vendor applications running on same version of Linux or even same Linux vendor – a lot of tweaking based on a vendor implementation
Security Concerns due to sharing same host OS
Running in Root – own name space solves this?
Container Challenges in the Networking World
18 © 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change
vSwitch Challenges… and Opportunities
Many demands as adoption
picks up, need to support …
Increased E W traffic
Network Appliances place new demands
Higher core density =>
higher VM VM traffic
L3 based multi-tenant segmentation
Security, Isolation, SLA
Monitoring, visibility
Many implementations with varying
features, performance and API
Some requirements for High performance
VMs and Network Appliances
High performance VM (40Gb / 100G NICs)
Adequate throughput, low CPU utilization
Low latency, Controlled latency (jitter)
As-is Open vSwitch small packet performance may fall behind NFV applications needs!
19 © 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change
NIC-OVS-NIC Performance and Features • Performance has many aspects. Clearly NFV/SDN requires top performance to meet workload
requirements and meet economical feasibility and infrastructure efficiency goals
• Performance numbers and gaps are presented below as reference
• Single flow / Single core 64 byte UDP raw datapath switching performance with pktgen
• Latency
• OvS features are required too e.g. NSH, DPI, QoS, DPDK,
• Yet, some features are much slower to make it into OvS while other (e.g. Geneve, OVN) make it to main stream while not more stable or more advanced in standards…
• How does the community get to a more open process for new features? Performance improvements?
STANDARD-OVS DPDK-OVS LINUX-BRIDGE
Gbits / sec 1.159 9.9 1.04
Mpps 1.72 14.85 1.55
Source: http://openvswitch.org/support/ovscon2014/ OVS performance measurements and analysis (Madhu Challa, Noiro Networks)
OVS DPDK-OVS LINUX-BRIDGE NIC-NIC VM-OVS-OVS-VM
TCP 46 33 43 27 72.5
UDP 51 32 44 26.2 66.4
© 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change
20
AT&T is moving quickly to an SDN enabled AT&T Integrated Cloud
Performance of the AIC platform (DPDK, ODP, OVS, OVR) is critical to get to 75% by 2020
DPDK-ODP convergence is important for the deployment of 1000s of AT&T Integrated Clouds
Summary
© 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change
21
© 2015 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change 22
Top Related