Network Service Orchestration: A SurveyNathan F. Saraiva de Sousa1, Danny A. Lachos Perez2, Raphael V. Rosa3,

Mateus A. S. Santos4, Christian Esteve Rothenberg5

Abstract—Business models of network service providers areundergoing an evolving transformation fueled by vertical cus-tomer demands and technological advances such as 5G, Soft-ware Defined Networking (SDN), and Network Function Vir-tualization (NFV). Emerging scenarios call for agile networkservices consuming network, storage, and compute resourcesacross heterogeneous infrastructures and administrative domains.Coordinating resource control and service creation across inter-connected domains and diverse technologies becomes a grandchallenge. Research and development efforts are being devotedon enabling orchestration processes to automate, coordinate, andmanage the deployment and operation of network services. In thissurvey, we delve into the topic of Network Service Orchestration(NSO) by reviewing the historical background, relevant researchprojects, enabling technologies, and standardization activities. Wedefine key concepts and propose a taxonomy of NSO approachesand solutions to pave the way to the understanding of thediverse ongoing efforts towards the realization of multiple NSOapplication scenarios. Based on the analysis of the state ofaffairs, we finalize by discussing a series of open challengesand research opportunities, altogether contributing to a timelyand comprehensive survey on the vibrant and strategic topic ofnetwork service orchestration.

I. INTRODUCTION

Modern telecommunication infrastructures consist of a myr-iad of technologies from specialized domains such as radio,access, transport, core and (virtualized) data center networks.Designing, deploying and operating end-to-end services arecommonly manual and long processed performed via tradi-tional Operation Support Systems (OSS) resulting in long leadtimes (weeks or months) until effective service delivery [1].Moreover, the involved workflows are commonly hamperedby limited built-in infrastructure strongly coupled to physicaltopologies and hardware-specific constraints.

Technological advances under the flags of Software DefinedNetworking (SDN) [2] and Network Function Virtualization(NFV) [3] bring new ways in which network operators cancreate, deploy, and manage their services. SDN and NFV

*This work was supported by UNICAMP and Ericsson.1N. Saraiva is PhD. student at Electrical and Computer Engineer-

ing (FEEC), University of Campinas (UNICAMP), Campinas, SP, Brazilnsaraiva at dca.fee.unicamp.br

2D. Lachos is PhD. student at Electrical and Computer Engineer-ing (FEEC), University of Campinas (UNICAMP), Campinas, SP, Brazildlachosp at dca.fee.unicamp.br

3R. Rosa is PhD. student at Electrical and Computer Engineering (FEEC),University of Campinas (UNICAMP), Campinas, SP, Brazil rvrosa atdca.fee.unicamp.br

4M. A. S. Santos is an Experienced Researcher at Ericsson Research,Indaiatuba, SP, Brazil mateus.santos at ericsson.com

5C. Rothenberg is Assistant Professor at University ofCampinas (UNICAMP), Campinas, SP, Brazil chesteve atdca.fee.unicamp.br

introduce new means for efficient and flexible utilizationof their infrastructures through a software-centric serviceparadigm [4]. However, to realize this paradigm, there is aneed to model the end-to-end service and have the ability toabstract and automate the control of physical and virtual re-sources delivering the service. The coordinated set of activitiesbehind such process is commonly referred to as orchestration.In general, orchestration refers to the idea of automaticallyselecting and controlling multiple resources, services, andsystems to meet certain objectives (e.g. a customer requestinga specific network service). Altogether, the process shall betimely, consistent, secure, and lead to cost reduction due toautomation and virtualization.

Multiple stakeholders are involved in the development andstandardization of enabling technologies for network soft-warization and their embodiment into next generation net-works (e.g. 5G) based on SDN, NFV, and Orchestration build-ing blocks and reference architectures. The ecosystem includesStandards Developing Organizations (SDOs), as well as indus-try groups, open source projects, foundations, diverse user-leadgroups, and so on. Examples of these players include Euro-pean Telecommunications Standards Institute (ETSI), MetroEthernet Forum (MEF), Organization for the Advancementof Structured Information Standards (OASIS), Linux Foun-dation, and Open Networking Foundation (ONF). Similarly,several (academic and industrial) research and (commercial)development efforts in orchestration, SDN and NFV have beengoing in recent years, concretized in a number of collaborativeendeavors such as Open Source MANO (OSM) [5], Open-Stack [6], Open Network Automation Platform (ONAP) [7],5G-Exchange (5G-EX) [8], Central Office Re-architected asa Datacenter (CORD) [9], etc.Multiple stakeholders are in-volved in the development and standardization of enablingtechnologies for network softwarization and their embodimentinto next generation networks (e.g. 5G) based on SDN, NFV,and Orchestration building blocks and reference architectures.The ecosystem includes Standards Developing Organizations(SDOs), as well as industry groups, open source projects, foun-dations, diverse user-lead groups, and so on. Examples of theseplayers include ETSI, MEF, OASIS, Linux Foundation, andONF. Similarly, several (academic and industrial) research and(commercial) development efforts in orchestration, SDN andNFV have been going in recent years, concretized in a numberof collaborative endeavors such as OSM [5], OpenStack [6],ONAP [7], 5G-EX [8], CORD [9], etc.Multiple stakehold-ers are involved in the development and standardization ofenabling technologies for network softwarization and theirembodiment into next generation networks (e.g. 5G) based onSDN, NFV, and Orchestration building blocks and reference

architectures. The ecosystem includes Standards DevelopingOrganizations (SDOs), as well as industry groups, open sourceprojects, foundations, diverse user-lead groups, and so on.Examples of these players include ETSI, MEF, OASIS,Linux Foundation, and ONF. Similarly, several (academic andindustrial) research and (commercial) development efforts inorchestration, SDN and NFV have been going in recent years,concretized in a number of collaborative endeavors such asOSM [5], OpenStack [6], ONAP [7], 5G-EX [8], CORD [9],etc.

Unfortunately, broad understanding and practical definitionsof Network Service Orchestration (NSO) are still missing–not only across but also inside networking communities.The maturity of ongoing efforts varies largely with theoverall technical approach being very much fragmented andshowing little consolidation around an overarching notionof network service orchestration. For instance, the work byRotsos et al. [10] is the first notable attempt to survey therealm of network service orchestration. The authors provide ananalysis of the diverse standardization activities around NSOfrom an operator perspective. The article follows a top-downapproach, defining terminologies, requirements, and objectivesof a network service orchestrator.

In this survey, our main objectives are to provide a com-prehensive review of research, standardization and softwaredevelopment efforts around the overcharged term of NetworkService Orchestration. We present an in-depth and up-to-datestudy on network service orchestration covering some historyand context, related enabling technologies, standardization ac-tivities, actual solutions, open challenges and research opportu-nities. In contrast to [10], we propose a view of NSO also froma customer perspective and propose a taxonomy of the maincharacteristics and features of NSO approaches. We also coverhow recent open source platforms and research projects mapto the primary characteristics and technical implementationsto NSO realization.

Throughout the survey, we distinguish between two types ofdomains. First, administrative domains, which map to differentorganizations and therefore may exist within a single serviceprovider or cover a set of service providers. In one adminis-trative domain, multiple technology domains can exist basedthe type of technology in scope, for example, Cloud, SDN,NFV, or Legacy. Broadly speaking, we refer to NSO as theautomated coordination of resources and services embracingboth single-domain and multi-domain contexts.

Figure 1 presents a generic high-level reference modelfor multi-domain Network Service Orchestration, featuring aMulti-Domain Orchestrator (MDO) per administrative realmand including the notion of a Marketplace for business interac-tions. MDOs coordinate resources and services in a multipleadministrative domain scope covering multiple technologydomains [11]. The exchange of information, resources, andservices themselves are essential components of an end-to-end network service delivery. The MDO exposes the availableservices to the marketplace allowing service providers to sellnetwork services directly to their customers or other providersunder various possible resources consumption models (e.g.trading resources from each other). The MDO can be seen

as a single element with a possible split into two functionalcomponents: Service Orchestrator (SO) and Resource Orches-trator (RO). The SO orchestrates high-level services while theRO is responsible for managing resource and orchestratingworkflows across technology domains. The Domain Orchestra-tors (DOs) performs orchestration in each local domain actingon the underlying infrastructures and exposing resources andnetwork functions northbound to the MDO.

The survey is organized as follows. Section II presentsessential background and key technologies related to net-work service orchestration. Concepts, functions, scope, and ataxonomy of NSO are presented in Section III. Section IVfocuses on the standardization efforts whereas Section Vcovers major research projects around NSO. Section VI pro-vides an overview of open source and commercial solutions.Section VII presents some potential scenarios to illustrate theNSO in practice. The discussion in Section VIII points to aseries of open challenges and research opportunities. Finally,Section IX concludes the survey.

II. BACKGROUND

NSO foundations can be rooted back to three enablingtechnologies, namely Cloud Computing, Software DefinedNetworking, and Network Function Virtualization. This sec-tion presents a brief overview of these topics, as well as anintroduction to the historical background and definitions ofterm “orchestration”.

A. Cloud Computing

Cloud computing is a model for enabling ubiquitous, con-venient, on-demand network access to a shared pool of con-figurable computing resources (e.g., networks, storage, andservices) that can be rapidly and automatically provisionedand released with minimal effort [12]. Thereby, the resourcesare traded on demand, that is, the customer only pays whatto use. Cloud computing becomes one of relevant technologyfor the 5G networks mainly because it provides high data rate,high mobility, and centralized management [13].

The service models of cloud computing are generally cat-egorized into three classes: Software as a Service (SaaS),Platform as a Service (PaaS), and Infrastructure as a Service(IaaS). In a cloud IaaS, the infrastructure is offered as aservice to the customer. Each customer can have its virtualresources, such as compute, storage, and network. SaaSincludes applications such as Facebook, Google Apps, Twitter,and Microsot Office 365.

PaaS provides services according to a users applicationswithout installing or configuring the operating system. Thecustomers can develop and deploy their applications in thesame development environment. The PaaS model includesservices such as Microsoft Azure, Google App Engine, RedHatOpenShift, and Amazon Elastic Beanstalk.

In SaaS, in turn, the customer is able to use the providers’applications running on a cloud infrastructure [3]. The soft-wares are maintained and managed by a cloud provider. IaaSincludes applications, for example, OpenStack, CloudStack,Amazon CloudFormation, and Google Compute Engine.

Fig. 1. High-level reference model to illustrate the scope of Network Service Orchestration (NSO) in single-domain and multi-domain environment. TheNSO need to have an overview of entire environment to compose the service mainly if the service to use resources of different domains.

In a cloud environment, the notion of orchestration has alsobeen used for integrating basic services [14]. The Orchestra-tion in the cloud involves dynamically deploying, managingand maintaining resource and services across multiple hetero-geneous cloud platforms in order to meet the needs of clients.This procedure demands to automatize processes and create aworkflow. However, this is not a simple task.

B. Software Defined Networking (SDN)

SDN [2] is an evolving networking paradigm that attemptsto resolve the strongly vertical integration of current networkenvironments. To this end, SDN proposals decouple the con-trol plane (i.e. control logic) from the data plane (i.e. dataforwarding equipments). With this new architecture, routersand switches become simple forwarding network elementswhose control logic is provided by an external entity calledSDN controller or Network Operating System (NOS). TheSDN controller creates an abstract network view while hidingdetails of the underlying physical or virtual infrastructure.Running on the top of the SDN controller, software networkapplications can perform not only traditional functionalitiessuch as routing, load balancing, classification [15], or In-trusion Detection Systems, but also propose novel use casessuch as service orchestration across multi-domain and multi-technology in 5G networks [16]. Those applications togetherwith other industry and academy initiatives towards flexiblenetwork services over programmable resources are among themain drivers of SDN.

The communication between the SDN controller and theforwarding devices is done through Southbound Interfaces(SBIS), which allow decoupling the control and data planevia open communication protocols (i.e. well-defined APIs).Different SDN SBIS can be considered (e.g., ForCES [17],OVSDB [18], POF [19], etc.), with OpenFlow [20] [21] being

the most widely accepted solution available in commercial andopen source (hardware and software) devices.

Service orchestrators, OSSes and other network applicationscan be developed on top of high-level Northbound Interfaces(NBIS) offered by a SDN controller. Indeed, NBIS are crucialcomponents to control and monitor the network servicesorchestration. Unlike SBIS, where Openflow is a well-knownSDN standard protocol, NBIS are still an open issue withdifferent controllers offering a variety of NBIS (e.g., RESTfulAPIs [22], NVP NBAPI [23], [24], SDMN API [25], etc.).In addition, other type of high-level NBIS category are im-plemented as NOS management applications [10]. Examplesof this category include Virtual Tenant Networks, ALTO, andIntent-based networking (IBN).

The logically centralized SDN controller act in spirit ofcomputer operating systems that provide a high-level abstrac-tion for the management of computer resources (e.g., harddrive, CPU, memory) by playing the network operating systemrole for network management [26]. As such, it provides a setof services (base network services, management, orchestration)and common interfaces (North/South/East/West) to developerswho can implement different control applications and improvemanageability of networks. Moreover, such interfaces areused within the Management and Orchestration (MANO)framework to deploy end-to-end connectivity. As today, themost popular open source SDN controllers are Open NetworkOperating System (ONOS) [27] and OpendayLight [28].

In SDN, the concept of orchestration is vital to automatenetwork operations properly. SDN network domains needsingle-domain or multi-domain orchestration systems to co-ordinate end-to-end connectivity services through differentnetwork domains controlled by different SDN controller in-stances, which in turn must communicate directly with thephysical network [29].

C. Network Function Virtualization (NFV)

Traditionally, the telecommunication operators have basedtheir networks on a built-in infrastructure strongly coupledto physical topologies and proprietary devices, resulting innetwork services constrained to the network topology and thephysical location of the network appliances. As a consequence,it becomes hard for providers to offer new services with lowercost and more efficiency and agility [3]. Network FunctionVirtualization has been proposed to solve these problems [30]and change the mode networks are designed and operated bytaking a software-centric approached leveraging advances invirtualization technologies and general purpose processors.

According to ETSI Industry Specification Group (ISG)NFV [31], Network Function Virtualization is responsiblefor separating network functions from the hardware and of-fering them through virtualized services, decomposed intoVirtualized Network Function (VNF), on general purposeservers. With the virtualization of the network functions,NFV promises more flexible and faster network functiondeployment, as well as dynamic scaling of the VNFs towardsproviding finer settings. In NFV environment new servicesdo not require new hardware infrastructure, but simply thesoftware installation, i.e. to create VNFs.

Moreover, the NFV can address Network Functions (NFs)in the most appropriate location, providing better user trafficperformance. The network service can be decomposed in oneor more VNFs, and each one can be constituted in one ormore Virtual Machines (VMs). Each VNF is described bya Virtualized Network Function Descriptor (VNFD) whichdetails the behavioral and deployment information of a VNF.

VNFs can be connected or combined as building blocksto offer a full-scale network communication service. Thisconnection is known as service chain. Service chain provideslogical connectivity between the virtual devices of NFV ar-chitecture. It is worthwhile noting not only connectivity orderimportance, but also the logical environment interconnectionwith physical networks.

Within the scope of the ISG NFV [31], service chain isdefined as a graph of logical links connecting NFs towardsdescribing traffic flow between these network functions. Thisis equivalent to the Service Function Chaining (SFC) [32]defined by Service Function Chaining Working Group (IETFSFC WG) of the Internet Engineering Task Force (IETF). Anend-to-end network service may cover one or more NetworkFunction Forwarding Graph (NF-FG) which interconnect NFsand end points. Figure 2 describes two examples of end-to-endnetwork services. The first (green line) is composed of virtualCustomer Premises Equipment (VCPE) and virtual Firewall(VFW) VNFs and two endpoints (A1 and A2). The second(red line) is composed of VCPE and virtual Deep PacketInspection (VDPI) VNFs and two endpoints (B1 and B2).Note that NFV allows sharing a multi-tenant VNFs betweendifferent network services.

ETSI has developed a reference architectural frameworkand specifications in support of NFV management and orches-tration. The framework focuses on the support VNF operationacross different hypervisors and computing resources. It also

Fig. 2. Example of two end-to-end network services composed of two NFseach. NFV enables the reuse of VNFs, e.g., VCPE.

covers the orchestration and lifecycle management of physicaland virtual resources. According to [33], “the framework isdescribed at a functional level and it does not propose anyspecific implementation.” Figure 3 shows the ETSI NFV-Management and Orchestration (MANO) architectural frame-work with their main functional blocks [34]:

• Operation/Business Support System (OSS/BSS): blockresponsible for operation and business applications thatnetwork service providers use to provision and operatetheir network services. It is not tightly integrated into theNFV-MANO architecture.

• Element Management (EM): component responsible forthe network management functions FCAPS (Fault, Con-figuration, Accounting, Performance, and Security) of arunning VNF.

• VNF: functional block representing the Virtualised Net-work Function implemented on a physical server. Forinstance, Router VNF, Switch VNF, Firewall etc.

• NFV Infrastructure (NFVI): representing all the hardware(compute, storage, and networking) and software compo-nents where VNFs are deployed, managed and executed.

• Network Function Virtualization Orchestrator (NFVO): itis the primary component, in charge of the orchestrationof NFVI resources across multiple Virtualized Infras-tructure Managers (VIMs) and lifecycle management ofnetwork services.

• VNF Manager (VNFM): performs configuration andVNF lifecycle management (e.g., instantiation, update,query, scaling, termination) on its domain.

• VIM: block provides controlling and managing the NFVIresources as well the interaction of a VNF with hardwareresources. For example, OpenStack as cloud platform andOpenDaylight and ONOS as SDN controllers.

The NFV-MANO functional block performs all thevirtualization-specific management, coordination, and automa-tion tasks in the NFV architecture including the componentsNFVO, VNFM, VIM, NFV Service, VNF Catalogue, NFV

Fig. 3. The NFV-MANO architectural framework. Adapted from [34]

Instance, and NFVI Resource.The NFV-MANO reference architecture does not specify

anything about SDN in its architecture instead it assumesthat necessary transport infrastructure is already establishedand ready to be used. However, in [35], ETSI identifies usecases and the most common options for using SDN in anNFV architectural framework. It also points proof of conceptsand recommendations to be fulfilled by the organizations thatintend to perform such integration. [36] provides a recent in-depth survey on NFV state of affairs.

D. Orchestration: Historical Overview

The academic community and industry generally requiresome time to define the real meaning, reach and context of theconcepts related to new technology trends as is the case withthe term Orchestration. The term orchestration is used in manydifferent areas, such as multimedia, music, service-orientedarchitecture, business processes, SDN, and, more recently, inNFV.

From an end-user perspective, orchestration reminds a sym-phony orchestra where a set of instruments play togetheraccording to an arrangement. The music is arranged and splitin small part, after assigns to different musical instruments.When, who, and what will be played, as well as the conductingare essential parts towards achieving the desired effect.

One of the first works in the Information and Communi-cation Technology (ICT) area that cite the term orchestration[37] relate it with coordination and control of multiple mediatraffics. It distinguishes the orchestration from synchronizationand defines an architecture where the orchestration acts in dif-ferent layers. In the same scope, [38] relates the orchestrationwith multimedia data. In this work, the orchestration is associ-ated with multimedia presentation lifecycle management. Thelifecycle management involves the coordination of stages thatconstitute all orchestration process.

The use of orchestration is also widely discussed in thescope of web services. In this context, orchestration andautomation are considered separate processes. The work in[39] defines orchestration like an executable process thatcan interact both internal and external services and must bedynamic, flexible, and adaptable the changes. It emphasizes

that orchestration describes how web services can act witheach other at the message level, including the business logicand execution order of the activities.

More recently in 2009, [40] provides an overview of defini-tion and design of Management and Service-aware NetworkingArchitectures (MANA) for the Future Internet (FI). One of thepillars for the FI pointed by the article is Orchestration. In theenvisioned architecture, the orchestration function coordinatesthe integrated behavior and operations to dynamically adaptand optimize resources in response to changing context fol-lowing business objectives and policies.

Orchestration is generally related to service automation incloud [41] and NFV environments. In spite of that, its conceptsare not clearly defined in the scope of NFV yet [42] [43].

E. Orchestration: Definitions

Various communities differ with respect to the meaning,assumptions and scope of orchestration functions. Thus, itis helpful begin by reviewing the community understandingto get the main concepts and significance. To this end, weoverview the leading organizations and efforts defining theterm Orchestration in diverse contexts.

A couple of years ago, the term orchestration was adoptedby ETSI in the scope of NFV. In ETSI NFV, the meaning oforchestration is implied, leading to a vague distinction betweenorchestration and management. Thus, its meaning may justbe inferred from the NFVO functions (both resources andservices layers). Similarly, the Internet Engineering Task Force(IETF) comes up with an orchestration definition closelyaligned with ETSI.

National Institute of Standards and Technology (NIST) [44]was one of the first organizations to define the concept ofService Orchestration. According to NIST vision, orchestra-tion is a process related to the arrangement, coordination, andmanagement of virtualized infrastructure to provide differentcloud services to customers.

The ONF [45] has formally defined orchestration as usageand selection of resources by orchestrator for satisfying clientdemands according to the service level. The meaning of theorchestration is evident given a SDN Controller. ONF men-tioned that main functions of Orchestration are two-fold. First,orchestration implies to split heavy-loaded service requestsinto service components. Moreover, it distributes the afore-mentioned components among supported platforms, creatingan integrated end-to-end solution across multiple domains.

The ITU-T Recommendation Y.3300 [46] describes theframework of software defined networking. This recommen-dation defines that SDN functions are programming, orches-trating, controlling and managing network resources. Also,it mentions that orchestration provides automated controland management of network resources. Nevertheless, ITU-Tdoes not clarify the difference between SDN functions andorchestration, what causes some confusion.

According to 3GPP Technical Specification 28.801 [47],orchestration is responsible for interpreting and translating agiven service request into a configuration of resources (phys-ical and/or virtualized), as needed for service establishment.

The configuration of resources may use resource allocationpolicies or actual available resources.

In the 5G white paper issued by NGMN [48], there isan end-to-end management and orchestration entity whichcomposes the proposed architecture, and it is in charge oftranslating the service request (business models) into infras-tructure resources, beyond managing tasks such as resourcescaling and network functions geographic distribution. It isworthwhile noting this proposal is similar to the one presentedby ETSI NFVO.

The MEF [49] proposes Lifecycle Service Orchestration(LSO) as a reference architecture for multi-domain orches-tration. LSO, based on network-as-a-service principles, ex-tends the NFV-MANO architecture and creates new capabil-ities. The orchestration of LSO refers to ”automated servicemanagement across multiple operator networks that includefulfillment, control, performance, assurance, usage, security,analytics, and policy capabilities.”

In addition to all the above-mentioned leading organizations,there are some works in the literature which also definethe concept of orchestration. According to [50], orchestrationenables programmability for creating and deploying of end-to-end network services and dynamic network control through asingle interface. Thyagaturu et al. [51] address orchestration asthe coordination of network services and operations in a higherlayer, abstracting the underlying physical infrastructure. Thework in [52] makes a generic definition of orchestration asautomated management of complex systems, middleware, andservices.

From the definitions of orchestration presented, we canderive a clear relationship among orchestration, automation,and management. Although the three terms are lumped to-gether, it is necessary an understanding of the differencesbetween them as they are not the same thing. Automationdescribes a simple and technical task without the humanintervention, for example, launching a web server, stoppinga server. Management is responsible for maintaining andhealthiness of infrastructure. Its role consists of activities suchas alarms for event detection, monitoring, backups of criticalsystems, upgrades, and license management. Orchestration, inturn, is concerned with the execution of a workflow (processes)in a correct order. It controls the overall workflow process fromstarting the service until it ends. Its objective is to optimizeand automate the network service deployment.

Figure 4 illustrates the relationship among the three termscited. There is a certain hierarchic between them. The orches-tration is a high-level plane, below the management, and inthe bottom the automation. In our vision, the orchestration de-pends on tasks performed by management. Both managementand orchestration are based on the use of automation in theexecution of their tasks. Nevertheless, some activities are onlyperformed by a certain function, optimization, for instance, cannot be achieved by simple automation. The optimization is aresponsibility of orchestration. There is a difference betweenthem, but, if they work together in the execution of processes,the services deployments will succeed with further accuracy.

Fig. 4. Relationship among orchestration, management, and automation. Bothorchestration and management use automation in their processes.

III. NETWORK SERVICE ORCHESTRATION

A. Towards a Practical Definition

We refer to the Network Service Orchestration (NSO) whenapplied in the services deployment performed by telecommu-nication operators and service providers. We regard NSO notprecisely as a unique technology but a concept to understandnetwork services in detail, relying on multiple technologiesand paradigms to achieve such an overarching goal. In a nut-shell, network service orchestration comprises the semanticsof requested service, and thereby it coordinates specific actionsin order to fulfill the service requirements and to manage itsend-to-end lifecycle.

The entire orchestration process proposed by NSO involvesbusiness and operations that go beyond the delivery of networkservices as defined by ETSI. ETSI NFV-MANO is a platformfor management and orchestration required to provisioningVNFs in an NFV domain. The MANO is agnostic and thushas no insight of what is executed within a VNF, restrictingits responsibility to the VNF instantiation and lifecycle man-agement.

Based on Figure 1, the MDO understands the operatingcapabilities of the Network Service (NS) in a broad sense.When a customer demands an NS, firstly it requests the orderto a service provider or telecommunication operator throughBusiness-to-Business (B2) interface or a trading platform werefer to as Marketplace. After that, the MDO interacts withany MANO element or other elements (e.g., OSS/BSS, SDNControllers, Analytic Systems) to create the NS. Thus, a givenMANO does not know if the VNFs it is deploying is aload balancer, firewall, or gateway. Meanwhile, the DO justcoordinates and manages the orchestration process at a givendomain, connecting the involved elements such as networksystems, controllers, management software, and IT softwareplatforms.

The NSO works at a higher level in the control and manage-ment stack with interfaces to the OSS/BSS. During a networkservice creation, the orchestration process can exceed thedomains boundaries being necessary to use resources and/orservices of other providers or operators. Such resources com-prised of physical and virtual components. Thus, the NSO issupposed to provide service delivery both within single and/ormulti-domain environments (more details in Section III-B).In this sense, different organizations and telecommunicationenterprises have developed many open source projects, driving

Fig. 5. Overview of the relationships between NSO, NFV, SDN, and CloudComputing.

orchestration evolution towards open standards that it willpermit the implementation of products with a large scale ofintegration. Section VI addresses some of these projects.

In addition, the customers are demanding full informationregarding a given hired network service such as detailedpricing, real-time analytics, and a certain control over theservice. NSO can offer more information to the customers andput more control into their hand. Its objective is to understandthe service profoundly and to enable that providers/operatorsattend customer demands.

From an operator and service provider viewpoint, NSOenables to set up new end-to-end services in minutes, keepingthose services working and ensuring acceptable performancelevels. This process reduces OPEX and provides enhancedservices creating new market opportunities and raising the rev-enues. As well as, it opens up chances for different companiesbecome service providers or provide virtual network functions.

NSO is in charge of all network service lifecycle anddelivers an end-to-end connectivity service. To achieve so, or-chestration is supported by advances in cloud computing, andtechnologies such as SDN and NFV, which offer the abilityto reconfigure the network quickly as well as programmingthe forwarding and processing of the traffic. Figure 5 aims atillustrating the relationships between NSO, NFV, SDN, andCloud Computing.

Each one of these paradigms has different functions: highlevel orchestration for NSO, function programming for NFV,networking programming for SDN, and resource virtualizationfor cloud computing. They can work in an integrated patternto offer advantages such as agility, cost reduction, automation,softwarization and end-to-end connectivity, to enable novelservices and applications such as 5G networks.

After this analysis, we can identify the main NSO featuresas follows:

• High-level vision of the NS that permit an overview ofall involved domains, technological and administrative.

• Smart services deployment and provisioning. These arerelated to in-deep knowledge about the services, whatenable better make decisions.

• Single and multi-domain environment support that pro-vide deployment of end-to-end service independently ofgeographical location.

• Proper interaction with different MANO and non-MANOelements which leads to better executed workflows.

• New markets opportunities, offering enhanced servicesand reducing OPEX.

B. Single and Multi-Domain Orchestration

Orchestration in the single and multi-domain environment isdifferent. In a single domain, the orchestrator is in charge of allservices and resource availability within its domain as well ashas total control over those resources. A domain orchestratormanages the network service lifecycle and interacts with othercomponents not only to control VNFs, but also computing,storage, and networking resources. Its scope is limited byadministrative boundaries of the provider. As shown in Fig-ure 1, domain orchestrators can orchestrate heterogeneoustechnological domains such as SDN, NFV, Legacy, and Datacenter. Under single domain environment, it is noticeable thatthe domain orchestrator works as described by ETSI in [34].

However, in a multi-domain environment, local orches-trators do not know the resources and topologies used byother providers. So, multi-domain orchestration is more com-plex, since it is supposed to provide end-to-end services,which requires cross-domain information exchange features(cf. [53]). Currently, there is not a standard for informationexchange process in multi-domain environments, either multi-technology domains or multiple administrative domains. Thereare some multi-domain orchestration candidates, e.g., T-NOVAFP7 project [54], ONAP [7], Escape [55], and 5G-EX [16].All of them will be discussed later in this survey.

ETSI proposes some options regarding multi-domain or-chestration. Initially, ETSI NFV Release 2 presents two ar-chitectures to address multi-domain scenarios [34]. In the first,the NFVO is split into Network Service Orchestrator, managesthe network service, and Resource Orchestrator, provides anabstract resource present in the administrative domain. A usecase for this first option is illustrated in Figure 6(a). A NetworkOperator offer resources to different departments within thesame operator, likewise to a different network operator. Oneor more Data centers and VIMs represent an administrativedomain and provide an abstracted view of its resources. TheService Orchestrator and VNF Manager can or can not bepart of another domain. In this use case, a service can run onthe infrastructure provided and managed by another ServiceProvider.

The second architecture does not split the NFVO, butcreates a new reference point between NFVOs (See Fig-ure 6(b)) called Umbrella NFVO. This use case requiresthe composition of services towards deploying a high-levelnetwork service. Such service can include network serviceshosted and offered by different administrative domains. Eachdomain is responsible for orchestrating its resources and

network services. This approach has objectives similar tofirst, however, an administrative domain is also composed ofVNFMs (together with their related VNFs) and NFVO. TheNFVO provides standard NFVO functionalities, with a scopelimited to the network services, VNFs and resources that arepart of its domain.

More recently, the ETSI NFV Release 3 presented othersoptions to support network services across multiple admin-istrative domains [56]. In particular, the use case entitled“Network Services provided using multiple administrativedomains” proposes a multi-domain architecture using NFV-MANO. Such architecture introduces the new reference pointnamed “Or-Or” between NFVOs to enable communicationand interoperability. Differently of second option (Figure 6(b)),in this approach, there is a hierarchy between the domains. Inthe example shown in Figure 6(c), NFVO in AdministrativeDomain C is on-top, using network services offered by Ad-ministrative Domains A and B, as well as managing compositeNS lifecycle.

In the scope of this paper, end-to-end network services arecomposed of one or more network functions interconnectedby forwarding graphs. Such services might span multipleclouds and geographical locations. Given that, they requirecomplex workflow management, coordination, and synchro-nization between multiple involved domains (infrastructure en-tities), which are performed by one (or more) orchestrator(s).Examples of end-to-end services are virtual extensible LAN(VxLAN), video service delivery, and virtual private network.

C. Orchestrator Functions

Section III-A identifies the various areas of term orches-tration. Orchestration can be inserted in the context of cloud,NFV, management systems, web services and more recentlyin the deployment of end-to-end network service in large net-works with multiple technologies and administrative domains.In this scope, the orchestrator is the component responsiblefor automatic resource coordination and control, as well asservice provision to customers.

In the NFV context, ETSI NFV-MANO defines the or-chestrator with two main functions including resources orches-tration across multiple VIMs and network service orchestra-tion [57]. Network service orchestration functions provided bythe NFVO are listed below.

• Management of Network Services templates and VNFPackages. This includes validation of templates andpackages with the objective of verifying the artifacts’authenticity and integrity. Besides, the software imagesare cataloged in involved Points of Presence (POPs) usingthe support of VIM.

• Network Service instantiation and management;• Management of the instantiation of VNFMs and VNFs

(with support of VNFMs);• Validation and authorization of NFVI resource requests

from VNF managers (resources that impact NS);• Management of network service instances topology;• Policy management related to affinity, scaling (auto or

manual), fault tolerance, performance, and topology.

ETSI NFVO functions regarding Resource Orchestration isexpressed as follows:

• Validation and authorization of NFVI resource requestsfrom VNF Managers;

• NFVI resource management including compute, storageand network;

• Collect usage information of NFVI resources;Related to NSO, the orchestrator, in turn, has a more

comprehensive function: to decouple the high-level servicelayer (e.g., marketplace, network slicing) from underlyingmanagement and resources layer (e.g., VNFs, Controller,EMS), simplifying innovations and enhancing flexibility inboth contexts. The orchestrator allows complex functions to beimplemented in underlying technologies and infrastructures.For example, real-time analytics of network services canbe deployed through the orchestrator. Another example, theorchestrator can connect the traditional OSS/BSS to the virtu-alized infrastructure. The Figure 7 represents the significanceof the orchestrator in the context of network service.

The orchestrator creates an abstraction unified point, en-abling to abstract physical and virtual resources, transparentlyexposing them to service providers and other actors, includingmarketplace and other orchestrators. It gives service providersfurther control of their network services and enables develop-ers to create new services and functions.

To accomplish this, the orchestrator must be inserted ineach layer of telecommunication network stack, from theapplication layer down to the data plane. Therefore, differentorchestrators can exist in each plane, not being limited to asingle orchestrator [43]. Some of the existing orchestrationsolutions use an orchestrator logically centralized and consideronly “softwarized” networks (see Section VI). However, thisis impracticable for large and heterogeneous networks.

The orchestrator can be classified according to its functionin: Service Orchestration (SO), Resource Orchestration (RO),and Lifecycle Orchestration (LO). Figure 8 illustrates the threeprimary network service orchestrator functions.

The Service Orchestration is responsible for service com-position and decomposition. It can be taken as the upperlayer, focused in the interaction with other components suchas Marketplace and OSS / Business Support Systems (BSS).The Lifecycle Orchestration deals with the management ofworkflows, processes, and dependencies across service compo-nents. Besides, it maintains the services running according tothe contracted Service Level Agreement. Finally, the ResourceOrchestration is in charge of mapping service requests toresources, either virtual and/or physical. This mapping occursacross elements such as NFVO, Element Management Sys-tems (EMS), and SDN controllers.

Lifecycle is used to manage a network service with variousstates (created, provisioned, stopped, etc.). When some actionis applied to a network service (e.g., provision a networkservice), many activities may be needed to apply on thecomponents of this network service. Hence, a workflow isused to execute a bunch of tasks in correct order. Each stateof lifecycle can generate one or more activities on workflows.The Figure 9(a) depicts the relationship between lifecycle andworkflow of a Network Service.

(a) (b) (c)

Fig. 6. ETSI approaches for multiple administrative domains: (a) approach in which the orchestrator is split into two components (NSO and RO), (b) approachwith multiple orchestrators and a new reference point: Umbrella NFVO, (c) approach that introduces hierarchy and the new reference point Or-Or. Adaptedfrom [34] and [56].

Fig. 7. Strategic role of the Orchestrator as the glue between the actualservices and the underlying management of resources.

Fig. 8. Different orchestrator functions: Resource Orchestration, Service Or-chestration, and Lifecycle Orchestration. There is a relationship of dependencyand continuity between the functions.

Figure 9(b) presents an example to improve the real def-inition of lifecycle and workflow in the context of network

service. One of the states in the service lifecycle is the Created.In order to achieve such state is necessary to execute fourtasks: create Virtual Deployment Unit (VDU)1, create VDU2,configure network and run the application. Therefore, the stateonly is finished when all those activities are completed.

Service lifecycle automation will allow that requested ser-vice remains in a desired state of behavior during its life-time. With the automation, the system responds proactivelyto changes network and service conditions without humanintervention, getting resilience and faults tolerance. Thesefunctional aspects of an orchestrator to guarantee the state ofa network towawrds a service goal are also being referred toas Intent-based Networking (IBN), cf. [58].

D. TaxonomyWhile many aspects of orchestration are under active de-

velopment and commercial roll-outs, others are still in a pre-liminary maturity phase. This subsection enumerates centralconcepts and characteristics related to any NSO approach. Itbecomes very challenging trying to summarize all conceptsrelated to orchestration in a single work, a challenge exacer-bated by the fast evolving pace of so many moving pieces,from standards to enabling technologies. Figure 10 presentsthe proposed taxonomy as the result of extensive literatureresearch as well as practical experiences with a number oforchestration platforms and research projects.

We identify seven key aspects to characterize networkservice orchestration:

1) Service Models. Relates to the type of services unlockedby the NSO, which may offer new business and relation-ships and opportunities (e.g., VNF as a Service (VNF),Slice as a Service (SLaaS)).

2) Software: Identifies major software-related characteris-tics of the orchestration solutions, including specificitiesof the management and standard interfaces.

3) Resource: Refers to the type of underlying resources(e.g., network, compute, and storage) used for the net-work service deployment.

(a) (b)

Fig. 9. Difference between Lifecycle and Workflow: (a) Lifecycle – sequence of states and workflow – activities in correct order and (b) example of networkservice lifecycle.

Fig. 10. NSO Taxonomy with seven approach: Service Model, Software, Resource, Technology, Scope of Application, Architecture, and Standards StandardsDeveloping Organization (SDO).

4) Technology: Points to target technologies for NSO (e.g.,Cloud, SDN, NFV, and Legacy).

5) Scope: Considers the application domain in terms ofnetwork segments embraced by the network serviceorchestration (i.e., from access network to data centers).

6) Architecture: Unfolds into three relevant architecturaldimensions with relate to single- and multi-domainorchestration and functional organization.

7) SDO (Standards Development Organization): Relates tostandardization activities in scope of the NSO.

Additional sub-areas contribute to an in-depth analysis indifferent contexts, which are further discussed in the followingsections.

1) Service Models: This aspect corresponds to the differentservice models related to orchestration process. Each serviceis inserted in the context of cloud, SDN, and/or NFV. Cloud

computing offers three categories of services such as IaaS,PaaS and SaaS [59]. In IaaS, Cloud Service Provider (CSP)renders a virtual infrastructure to the customers. In PaaS,CSPs provide development environment as a service. Finally,SaaS is a service that furnishes applications hosted andmanaged in the cloud.

SDN and Network as a Service (NaaS) paradigms can begathered to provide end-to-end service provisioning. WhileSDN supply the orchestration of underlying network (switches,router, and links), the NaaS is responsible for private accessto the network and customer security [60].

The NFV, in turn, can offer new services including NFVIas a Service (NFVI), VNFaaS, SLaaS and Virtual NetworkPlatform as a Service (VNPaaS). The NFVIaaS providesjointly IaaS and NaaS tailored for NFV. VNFaaS is aservice that implements virtualized Network Functions to the

Enterprises and/or end customers. VNPaaS is a platformavailable by service providers allowing customers to createtheir own network services. The SLaaS is a concept that theslices are traded and used to build infrastructure services.

All these services can work in parallel to offer higher-levelservices. Each one acts in a specific area and offers featuresto customers, enterprises, or other providers.

2) Software: There are many software artifacts related toorchestration covering from a single cloud environment upto more complex scenarios involving multi-domain orchestra-tion. These solutions are outcomes of open source initiatives,research projects or commercial vendors.

Open source approaches significantly accelerate consensus,delivering high performance, peer-reviewed code that formsa basis for an ecosystem of solutions. Open source makesit possible to create a single unified orchestration abstraction.Thus, both research projects and commercial vendors leverageopen source technologies to accelerate and improve theirsolutions. Operators, such as Telefonica, China Mobile, AT&T,and NTT, appear committed to using open source as a way tospeed up their development of orchestration platforms [61].

The Open Source Initiative (OSI)1 defines licenses un-der Open Source Definition compliance, which allows codeand software to be freely used, shared and modified. Themore popular open source licenses are Apache License 2.0,Berkeley Software Distribution (BSD), GNU General PublicLicense (GPL), Mozilla Public License 2.0, and Eclipse PublicLicense. Namely, the most important orchestration projectsand frameworks (for instance, Aria, Cloudify, CORD, Gohan,Open Baton, Tacker, ONAP, SONATA, and T-NOVA) presenta widespread usage of Apache License 2.0.

Another topic related to open source is governance. In short,governance defines the processes, structures, and organiza-tions. It determines how power is exercised and distributedand how decisions are taken. Commonly, a governing boardis responsible for the budget, trademark/legal, marketing,compliance, and overall direction, while a technical steeringcommittee is responsible for technical guidance.

An open source orchestration project may be organizedas a single community (e.g. vendor-lead) or can be hosted(and eventually integrated with other projects) by a foundationentity [62]. A remarkable example is the Linux Foundation,which among multiple networking related projects is in chargeof ONAP, an open source platform aiming at the automation,design, orchestration, and management of SDN and NFVservices. Another noteworthy example of an orchestration opensource project under the Linux Foundation flagship aiming atdelivering a standard NFV/SDN platform for the industry isOpen Platform for NFV (OPNFV) [63].

NSO solutions need to perform management tasks such asremote device configuration, monitoring and fault manage-ment. Moreover, they require defining interface of commu-nication between various software components. For this, thereare diverse types of management and standard interfaces suchas Command Line Interface (CLI), Application ProgrammingInterface (API), and Graphical User Interface (GUI). The CLI

1http://opensource.org

just is used to execute commands directly in the softwareusing remote access via SSH or Telnet. The API enables theremote management and interconnection with other softwaresthrough specifics commands. The majority of solutions useREST-based API. GUI, in turn, offers a graphic interface thatmakes it easier its use.

3) Resource: During the creation of a network service,the resource orchestration is responsible for orchestrating theunderlying infrastructure. Such infrastructure is composed ofheterogeneous hardware and software for hosting and con-necting the network services. The resources include compute,storage, and network [64]. In essence, there are three types ofnetworks: packet, optical and wireless (e.g., Wi-Fi, wi-max,and mobile network).

Resources are shared and abstracted making use of virtu-alization techniques (e.g., para-virtualization [65], full virtu-alization [66], and containers [67]), defining virtual infras-tructures that can be used as physical ones. Sharing andmanagement of resources are much more complex in multi-domain scenarios. The NSO needs to know all the availableresources towards the efficient deployment of the NSs.

4) Technology: NSO involves complex workflows and dif-ferent technologies involved throughout orchestration process:cloud computing, SDN, NFV, and legacy.

The cloud computing paradigm provides resource virtualiza-tion and improves resource availability and usage by meansof orchestration and management procedures. This includesautomatic instantiation, migration and snapshot of VMs, High-Availability, and dynamic allocation of resources [30].

The SDN promotes control across network layers andlogical centralization of network infrastructure management.Its main functions is to connect the VNFs and the NFVI-POPs. In parallel, the NFV technology promotes the networkfunctions programming in order to enable elasticity, automa-tion, and resilience in cloud environments [10]. As illustratedin Figure 5, cloud computing, SDN and NFV are enablertechnologies to the NSO. The NSO must also handle legacytechnologies such as MPLS, BGP, SONET / SDH, and WDM.

5) Scope: Resources of operators under an orchestrationapplication domain can be part of access networks, aggregationnetworks, core networks and data centers [11]. The accessnetwork is the entry point which connects users to theirservice provider. It encompasses various technologies, i.e.,fixed access, radio access (Wi-Fi, LTE, radio), optical, andprovide connectivity to heterogeneous services such as mobilenetwork and Internet of Things (IOT). The core network isthe central part of a telecommunications network that connectslocal providers to each other. The aggregation network, in turn,connects the access network to core network. The data centeris the local where are localized the computing and storageresources.

The infrastructure is formed by heterogeneous technologiesthat may be owned by different infrastructure providers. Thenetwork service orchestration in this environment is a chal-lenging task. The NSO must have a view of resources andservices since access network until the data center to deployend-to-end network services. Besides, it is also important to

provide consistent and continuous service, independent of theunderlying infrastructure [11].

6) Architecture: An NSO architecture can be divided intothree sub-categories: (i) domain, (ii) organization, and (iii)functions. The domain refers to coverage of the orchestrationprocess in one or more administrative domains: single-domainand multi-domain. In each scenario, the orchestration has itspeculiarities.

Single-domain orchestration studies focus on verticalNFV/SDN orchestration within a same administrative domain.In our definition, an administrative domain can have multipletechnological domains, such as SDN, NFV, and Legacy.The taxonomy is aligned with ETSI NFV architecture thataddresses orchestration for NFV. The multi-domain orches-tration involves the instantiation of network service amongtwo or more administrative domains. It is composed of planes(or layers) with different functions and architecture topology.The multi-domain interfaces are not present in original ETSINFV architecture

The organization refers to the different architectural ar-rangements of a NSO solution. We identified three typesof organization: hierarchical, cascading and distributed. Thehierarchical approach assumes a high-level orchestrator thathas visibility of the entire other domains and capable ofconfiguring services across different domains. The serviceprovider facing the customer as a single entry point willmaintain relationships with other providers to complete therequested service. According to [44], the hierarchical approachis impractical because of scalability and trust constraints.Under the cascade model, the provider partially satisfies theservice request but complements the service by using resourcesfrom another provider. If this provider does not have all theresources, it also can request for another and so on (e.g., amobile network provider using a satellite provider). In thedistributed model, there is not a central actor, and providersrequest resource and services from each other on a peer-to-peer fashion.

Finally, functions,as discussed in Sec. III-C, refers to themain tasks developed by network service orchestrator: serviceorchestration, resource orchestration, and lifecycle orchestra-tion. These functions can be separated or together in thesame component of an orchestration framework. This decisiondepends on how the orchestrator was developed.

7) Standards Development Organization (SDO): SeveralStandards Development Organizations, including ETSI, MEF,IETF, and International Telecommunication Union (ITU), areactively working on a collection of standards in order to definereference architectures, protocols, and interfaces in scope ofthe orchestration domain. Besides, other organizations, aca-demic, vendors and industrial are working in parallel withdiverse goals. The main efforts within standardization bodieswill be outlined next.

IV. NSO AND STANDARDIZATIONInteroperability and standardization constitute essential fac-

tors of the success of a network service orchestration solution.An important design goal for any new networking paradigm re-lates to openness of interfaces, especially in order to overcome

interoperability issues [10]. Several standardization efforts aredelivering a diverse collection of norms and recommendationsto define an architecture and/or a framework that enables thenetwork service orchestration. This section presents the mainstandardization bodies at the NSO scope. Table I presents asummary of the main SDOs and organizations related to NSOstandardization, as well as the main outcomes produced todate.

A. ETSI

ETSI ISG NFV defines the MANO architectural frame-work to enable orchestration of VNFs on top of virtual-ized infrastructures. Since 2012, the group provides pre-standardization and specification documents in different ar-eas, including management and orchestration. NFVO takes afundamental role in NFV-MANO functional components, asdefined in [57] realizing: (i) the orchestration of infrastructureresources (including multiple VIMs), fulfilling the ResourceOrchestration functions; (ii) and the management of NetworkServices, fulfilling the network service orchestration functions.

Logically composing ETSI NFVO, NSO stipulates gen-eral workflows on network services (e.g., scaling, topol-ogy/performance management, automation), which conse-quently reach abstracted functionalities in other MANO com-ponents – lifecycle management of VNFs in coordination withVNFM and the consume of NFVI resources in accordancewith VIM operational tasks.

Currently, ETSI matures NFV in different areas, such asarchitecture, testing, evolution and ecosystem. Among ongoingtopics approached, network slicing report, multi-administrativedomain support [34] [56], and multi-site services (draftingstage) highlight important aspects of evolving the NFV archi-tectural framework, including possible new NSO functionali-ties. In the upcoming years, ETSI is expected to keep playinga driving role on NFV, what represents a path towards real-ization of concepts built upon the recommendations/reports, asattested by open source projects such as Open Source MANO.

B. MEF

Metro Ethernet Forum (MEF)’s Third Network [49] ap-proaches NaaS comprising agility, assurance and orchestrationas its main characteristics to broach LSO in their definedCarrier Ethernet 2.0 framework. LSO, as a primer component,provides network service lifecycle management when ap-proaching series of capabilities (e.g., control, performance, an-alytics) towards fulfilling network service level specifications.MEF’s LSO provides re-usable engineering specifications torealize end-to-end automated and orchestrated connectivityservices through common information models, open APIs,well-defined interface profiles, and attaining detailed businessprocess flows. Therefore, in LSO Service Providers orches-trate connectivity across all internal and external domains fromone or more network administrative domains.

A detailed LSO reference architecture [74] presents com-mon functional components and interfaces being exemplifiedin comparison with ETSI NFV framework and ONF SDNarchitecture. Internally, a Service Orchestration Functionality

TABLE INSO STANDARDIZATION OUTCOMES

SDO Working Group Scope Outcomes

Service Quality Metrics for NFV Orchestration [68]Management and Orchestration Framework [33]NFV ISG (Initial)

Multiparty Administrative domains [69]

VNF Architecture and SDN in NFV Architecture [70]Orchestration of virtualized resources [71]Functional requirements for Orchestrator [71]Lifecycle management of Network Services [71]

NFV ISG (Release 2)

Network Service Templates Specification [72]

Policy management [73]

ETSI

NFV ISG (Release 3)

NFV

Report on architecture options to support multiple administrativedomains [56]

MEF The Third Network NFV, LSOLifecycle Service Orchestration Vision [49]LSO Reference Architecture and Framework [74]

ONF Architecture andFramework SDNSDN Architecture [75]Mapping Orchestration Application to SDN [76]Definition of Orchestration [77]

IETF SFC SFC, NFV SFC Architecture [32]

White Paper: Next Generation Networks [78]NGMN Work Programme 5G Network

5G Network and Service Management and Orchestration [79]

TM Forum Project SDN, NFV ZOOM (Zero-touch Orchestration, Operations and Manage-ment) [80]

3GPP S5 5G Network (mobile) Management and orchestration for next generation network [47]

TOSCA version 1.0 [81]TOSCA for NFV Version 1.0 [82]OASIS TOSCA

Resource and ServiceModeling

TOSCA in YAML Version 1.2 [83]

Report on Standards Gap Analysis in 5G Network [84]Terms and definitions for 5G network [85]5G Network management and orchestration requirements [86]5G Network management and orchestration framework [87]

ITU-T SG 135G Network (IMT-2020)and networksoftwarization

Standardization and open source activities related to networksoftwarization [88]

ITU

ITU-R Mobile, radiodetermina-tion, amateur and relatedsatellite services

Framework and overall objectives of the 5G Network [89]

provides to LSO coordinated end-to-end management andcontrol of Layer 2 and Layer 3 Connectivity Services realizinglifecycle management supporting different capabilities. Be-sides, LSO defines APIs for essential functions such as serviceordering, configuration, fulfillment, assurance and billing. Arecent example of MEF’s use case conceptualization presentsan understanding of Software Defined Wide Area Network(WAN) (WAN) managed services in face of LSO referencearchitecture [90].

C. ONF

At ONF, the SDN architecture defines orchestration as TR-521 [77] states: “In the sense of feedback control, orches-tration is the defining characteristic of an SDN controller.

Orchestration is the selection of resources to satisfy servicedemands in an optimal way, where the available resources, theservice demands, and the optimization criteria are all subjectto change”.

Logically, ONF perceives the SDN controller jointly over-seeing service and resource-oriented models to orchestratenetwork services through intents on a client-server basis. Fromtop-to-bottom, a service-oriented perspective relates to invoca-tion and management of a service-oriented API to establishone or more service contexts and to fulfill client’s requestedservice attributes. Such requirements guide the SDN controllerin orchestrating and virtualizing underlying resources to buildmappings that satisfy the network service abstraction andrealization. While in a bottom-up view, a resource-orientedmodel consists of SDN controller exposing underlying re-

source contexts so clients might query information and requestservices on top of them. In accordance, resource alterationsmight imply in reallocation or exception handling of servicebehavior, which might be contained in policies specified byclient’s specific attributes in a service request.

Recursively, stacks of SDN controllers might coordinate ahierarchy of network service requests into resource allocationaccording to their visibility and control of underlying techno-logical and administrative network domains (e.g., Cross Stra-tum Orchestration [76]). Thus, SDN controllers might haveNorth-South and/or East/West relationships with each other. Atlast, a common ground for orchestration concepts is publishedby ONF as “Orchestration: A More Holistic View” [75],elucidating considerations of its capabilities, among them,employing policy to guide decisions and resources feedback,as well its analysis.

D. IETF/IRTF

At IETF and Internet Research Task Force (IRTF), differentworking and research groups address NSO from varyingangles. Traffic Engineering Architecture and Signaling (TEAS)working group characterizes protocols, methods, interfacesand mechanisms for centralized (e.g., PCE) and distributedpath computation (e.g., MPLS, GMPLS) of traffic engi-neered paths/tunnels delivering specific network metrics (e.g.,throughput, latency). Application-based Network Operations(ABNO) sets modular a modular control architecture, stan-dardized by IETF aggregating already standard components,such as Path Computation Element (PCE) to orchestrateconnectivity services. SFC Working Group (WG) defines adistributed architecture to enable network elements computeNF forwarding graphs realizing overlay paths. The list ofprotocols involved in NSO is by far not completed andmany new extensions to existing protocols and new ones areexpected due to the broad needs of interoperable networkservice orchestration solutions.

Conceptually, IETF establishes no direct relationship withorchestration, as it concerns majorly the development of pro-tocols. However, the development of protocol-related systems,information models and management interfaces by workinggroups can enable orchestration of services such as the layer 3VPN2. Even more, as the core network service of the internet,routing detains the capabilities to be managed by orchestrationinterfaces through the work being performed by the Interfaceto the Routing System (i2rs) working group. For instance, i2rsfacilitates “information, policies, and operational parametersto be injected into and retrieved (as read or by notification)from the routing system while retaining data consistency andcoherency across the routers and routing infrastructure, andamong multiple interactions with the routing system”. Assuch, RFCs developed by different IETF working groups sitin the scope of SDN, enablers of programmable networks, andtherefore, inheriting orchestration capabilities.

2https://datatracker.ietf.org/wg/l3vpn/about/

E. NGMNNext Generation Mobile Networks (NGMN) in [79] pro-

vides key requirements and high-level architecture principlesof Network and Service Management including Orchestrationfor 5G. Based on a series of user stories (e.g., slice creation,real-time provisioning, 5G end-to-end service management),the document establishes a common set of requirements.Among them self-healing, scalability, testing and automation,analysis, modeling, etc. Regarding orchestration functionali-ties, the presented user stories introduce components (e.g.,SDN controllers and ETSI NFVO), which execute actionsto perform actors goals. For instance, slice creation would beend-to-end service orchestration interpreting and translate ser-vice definitions into a configuration of resources (virtualizedor not) needed for service fulfillment.

As part of the initially envisioned 5G White Paper [78],NGMN provided business models and use cases based onadded values that 5G would bring for future mobile networks.In general, SDN and NFV components are listed as enablersfor operational sustainability that will drive cost/energy effi-ciency, flexibility and scalability, operations awareness, amongother factors for simplified network deployment, operation,and management. Such technology candidates highlight theimportance of orchestration capabilities besides the evolutionof radio access technologies towards 5G realization.

F. TM ForumTeleManagement Forum (TM Forum) is a global association

for digital businesses (e.g., service providers, telecom opera-tors, etc.) which provides industry best practices, standards andproof-of-concepts for the operational management systems,also known as Operations Support Systems (OSSs). One ofthe biggest TMForum achievements is the definition of atelecom business process (eTOM) and application (TAM)maps, including all activities related to an operator, fromthe services design to the runtime operation, consideringassurance, charging, and billing of the customer, among others.In order to accommodate the SDN/NFV impacts, the TMForum has created the Zero-Touch Orchestration, Operationsand Management (ZOOM) program, which intends to buildmore dynamic support systems, fostering service and businessagility.

As a related research project, SELFNET3 is, on one side, ac-tively following and aligning its architecture definition with theTM Forum ZOOM and FMO recommendations. Additionally,SELFNET, through one partner of the consortium that is an ac-tive member of TM Forum, is also going to actively contributeto the ZOOM working group with respect to the impact that theNFV/SDN paradigm has on the OSS information model (CFSCustomer Facing Service, RFS Resource Facing Services,LR Logical Resources, PR Physical Resources). Besides theZOOM working group, SELFNET will also contribute to theFMO working group by participating in the next generationOSS architecture, which includes the autonomic managementcapabilities to close the autonomic management loop: 1)Supervision 2) Autonomic 3) Orchestration/Actuation.

3https://selfnet-5g.eu/

G. 3GPP

Related to the ongoing specification “Study on managementand orchestration architecture of next generation networkand service” [91], 3GPP analyzes its existing architecturalmanagement mechanisms in contrast with next generationnetworks and services in order to recommend enhancements,for instance, to support network operational features (e.g., real-time, on-demand, automation) as evolution from Long TermEvolution (LTE) management. Among the item sets containedin the scope, the specification addresses: the scenario in whichthe applications are hosted close to the access network; end-to-end user services; and vertical applications, such as criticalcommunications. Another ongoing specification, “Telecommu-nication management; Study on management and orchestrationof network slicing for next generation network” [47], presentscomprehensive 3GPP views on network slicing associated withautomation, sharing, isolation/separation and related aspectsof ETSI NFV MANO. In both documents, use cases andrequirements cover single and multi-operator services takinginto consideration performance, fault tolerance and configura-tion aspects.

Similar to IETF, 3GPP establishes a relationship with or-chestration through management models for network slicing.Related to slicing, the 3GPP TS28 series of documents defines,among other specifications, a network resource model for 5Gnetworks. In a protocol and technology neutral way, suchmodels enable management interfaces for the lifecycle man-agement of 5G networks (e.g., core, access, and radio accesstechnologies). Closely related with NFV, in 3GPP the studyof management and virtualization aspects of 5G networkstakes places involving the characterization of performancemanagement and fault supervision. For instance, in a firststage, TS 28.545 defines the use cases and requirements forfault supervision of 5G networks and network slicing. Towardssinergy studies of 3GPP systems with NFV, there exists on-going work to elaborate further on the energy efficiencycontrol framework defined in TR 21.866 and identify potentialgaps with respect to existing management architectures, in-cluding self-organizing networks and NFV based architectures.Therefore, related to all the major benefits of introducingNFV paradigms into 3GPP, the management of 5G networksand network slices addresses orchestration in its essence,concerning mostly fault and performance.

H. OASIS

OASIS standardizes Topology and Orchestration Specifica-tion for Cloud Applications (TOSCA) focused on “Enhancingthe portability and operational management of cloud appli-cations and services across their entire lifecycle”. TOSCASimple Profile in YAML v1.0 was approved as standard in2016 in a rapidly growing ecosystem of open source communi-ties, vendors, researchers and cloud service providers. Lookingforward, TOSCA Technical Committee develops a SimpleProfile for NFV based on ETSI NFV recommendations.

Logically, TOSCA allows the expressiveness of serviceto resource mappings via flexible and compoundable datastructures, also providing methods for specifying workflows

and, therefore, enable lifecycle management tasks. In bothSimple and NFV Profiles, TOSCA models service behaviorsdefining components containing capabilities and requirements,and relationships among them. TOSCA realizes a compliantmodel of conformance and interoperability for NSOs, enhanc-ing the portability of network services.

TOSCA aims

I. ITU

International Telecommunications Union (ITU) is theUnited Nations specialized agency for information and com-munication technologies (ICTs). It develops technical stan-dards that ensure networks and technologies seamlessly in-terconnected. The Study Groups of ITUs Telecommunica-tion Standardization Sector (ITU-T) develops internationalstandards known as ITU-T Recommendations which act asdefining elements in the global infrastructure of ICTs [92].

The ITU is working on the definition of the framework andoverall objectives of the future 5G systems, named as IMT-2020 (International Mobile Telecommunications for 2020) sys-tems [89]. The documentation is detailed in RecommendationITU-R M.2083-0. It describes potential user and applicationtrends, growth in traffic, technological trends and spectrumimplications aiming to provide guidelines on the telecommu-nications for 2020 and beyond.

Besides, the Study Group 13 of ITU-T is developing areport on standards gap analysis [84] that describes the high-level view of the network architecture for IMT-2020 includingrequirements, gap analyses, and design principles of IMT-2020. Its objective is to give directions for developing stan-dards on network architecture in IMT-2020. In this report alsoincludes the study areas: end-to-end quality of service (QoS)framework, emerging network technologies, mobile front hauland back haul, and network softwarization. The report is basedon the related works in ITU-R and other SDOs.

V. RESEARCH PROJECTS

This section presents an overview of relevant NSO researchprojects and positions our taxonomy accordingly as summa-rized in Table II, providing a single vision of their scopeand status. The following subsections are identified by projectname and its duration.

A. T-NOVA (2014/01-2016/12)

The focus of the T-NOVA project [54] is to design andimplement an integrated management architecture for the au-tomated provision, configuration, monitoring and optimizationof network connectivity and Network Functions as a Service(NFaaS). Such architecture includes: (i) a micro-service basedNFV orchestration platform–called TeNOR [93], (ii) an in-frastructure virtualisation and management environment and(iii) an NFV Marketplace where a set of network services andfunctions can be created and published by service providersand, subsequently, acquired and instantiated on-demand bycustomers.

In the T-NOVA architecture, TeNOR is the highest-level in-frastructure management entity that supports multi-pop/multi-administration domain, transport network (i.e., MPLS, Optical,Carrier Ethernet, etc.) management between POPs, and datacenter cloud assets control. The TeNOR Orchestrator is splitinto two elements: (i) Network Service Orchestrator thatmanages the Network Service Lifecycle, and (ii) VirtualizedResource Orchestrator that orchestrates the underlying com-puting and network resources [94].

T-NOVA leverages cloud management architectures for theprovision of resources (compute and storage) and extends SDNfor efficient management of the network infrastructure [95].Its architecture is based on concepts from ETSI NFV modeland expands it with a marketplace layer and specific add-onfeatures. All software components produced during the projectare available as open source at github4.

B. UNIFY (2013/11-2016/04)FP7 Unify 5 project dedicated to approaching multiple

technology domains to orchestrate joint network services con-cerning compute, storage and networking. The primary focusset flexibility as its core concern, especially to bring methodsto automate and verify network services.

Unify overall architecture contains components in a hier-archical composition enabling recursiveness. At the bottom,a set of Controller Adapters (CAs) interface technology-specific domains (e.g., optical, radio, data center) to abstractsouthbound APIs for a typical model of information todefine software programmability over a network, computeand storage elements, such as virtualized container, SDNoptical controller and OpenStack. Overseeing CAs, ResourceOrchestrators (ROs) define ways to manage the abstractedcomponents of technology-domains specifically. For instance,while an RO for a SDN controller orchestrates network flows(e.g., allocating bandwidth and latency), a RO for a cloud or-chestrator would be concerned more over orchestrate networkjointly with compute and storage resources (e.g., allocatingmemory and disk). Moreover, managing one or more ROs,a global orchestrator performs network service orchestrationin multiple technological domains, understanding the servicedecomposition and outsourcing specific orchestration tasks toROs.

In the end, Unify was able to present a common model ofinformation to interconnect different technological domains,CAs, ROs and global orchestrator. Such YANG model wasnamed Virtualizer, and logically defined configurations fol-lowing the NETCONF protocol. Different demos showcasingjoint orchestration of compute and network resources werepresented, using the open source orchestrator ESCAPE 6, forinstance, modeling VNFs over data centers interconnected viaan SDN enabled network domain.

Besides, based on the ONF SDN architecture, Unify wasable to demonstrate methods to apply recursiveness across itsfunctional components in order to decompose network servicesto technological-specific domains.

4https://github.com/T-NOVA5http://www.fp7-unify.eu/6https://github.com/hsnlab/escape

C. 5GEx (03/2015-03/2018)

5GEx project aims agile exchange mechanisms for con-tracting, invoking and settling for the wholesale consumptionof resources and virtual network service across administrativedomains. Formed by a consortium of vendors, operators, anduniversities, 5GEx allows end-to-end network and serviceelements to mix in multi-vendor, heterogeneous technologyand resource environments. In such way, the project targetsbusiness relationships among administrative domains, includ-ing possible external service providers without infrastructureresources.

Architecturally, 5GEx addresses business-to-business (B2B)and business-to-customer (B2C) relationships across multi-administrative domain orchestrator that might interface dif-ferent technological domains. Basically, 5GEx extends ETSINFV MANO architecture with new functional componentsand interfaces. Among its main components, the project de-fines modules for: topology abstraction; topology distribution;resource repository; Service Level Agreement (SLA) man-ager; policy database; resource monitoring; service catalog;and an inter-provider NFVO. 5GEx currently utilizes outcomeresources mostly from the projects Unify and T-NOVA, es-pecially joining their open source components into alreadyprototyped demonstrations.

D. SONATA (07/2015-12/2017)

With 15 partners representing the telecommunication oper-ators, service providers, academic institutes (among others),the Service Programming and Orchestration for VirtualisedSoftware Networks (SONATA) project [96] addresses two sig-nificant technological challenges envisioned in 5G networks:(i) flexible programmability and (ii) deployment optimizationof software networks for complex services/applications. Todo so, SONATA provides an integrated development andoperational process for supporting network function chainingand orchestration [97].

The major components of the SONATA architecture consistof two parts: (i) the SONATA Software Development Kit (SDK)that supports functionalities and tools for the development andvalidation of VNFs and NS and (ii) the SONATA ServicePlatform, which offers the functionalities to orchestrate andmanage network services during their lifecycles with a MANOframework and interact with the underlying virtual infrastruc-ture through Virtual Infrastructure Managers (VIM) and WANInfrastructure Managers (WIM) [98].

The project describes the use cases envisioned for theSONATA framework and the requirements extracted fromthem. These use cases encompass a wide range of networkservices including NFVIaaS, VNFaaS, vContent DeliveryNetwork (CDN), and personal security. One of the use casesconsists of hierarchical service providers simulating one multi-domain scenario. In this scenario, Service Programming andOrchestration for Virtualised Software Networks (SONATA)does not address the business aspects, only the technicalapproaches are in scope. SONATA intends to cover aspectsin the cloud, SDN and NFV domains [99].

Moreover, the project proposes to interact and manage withnot only VNFs also support legacy [100]. Besides, it describestechnical requirements for integrating network slicing in theSONATA platform. The SONATA framework complies withthe ETSI NFV-MANO architecture [100]. The results ofthe project are shared with the community through a publicrepository7.

E. 5G-Transformer (06/2017-11/2019)

The 5G-Transformer Project [101] consists of a group of18 companies including mobile operators, vendors, and uni-versities. The objective of the project is to transform currentsmobile transport network into a Mobile Transport and Com-puting Platform (MTP) based on SDN, NFV, orchestration,and analytics, which brings the Network Slicing paradigm intomobile transport networks. The project will support a varietyof vertical industries use cases such as automotive, healthcare,and media/entertainment.

Likewise, 5G-Transformer defines three new components tothe proposed architecture: (i) vertical slicer as a logical entrypoint to create network slices, (ii) Service Orchestrator forend-to-end service orchestration and computing resources, and(iii) Mobile Transport and Computing Platform for integratefronthaul and backhaul networks. The Service Orchestrator isthe main decision point of the system and interacts with othersSOs to the end-to-end service (de)composition of virtualresources and orchestrate the resources even across multipleadministrative domains. Its function is similar to our definitionof NSO. The project architecture is still ongoing and is notclear its organization (hierarchical, cascade, or distributed).Still in an early stage, the project aims to produce open sourceartifacts, and deliverables in alignment with standardizationbodies such as 3GPP and MGMN [102].

F. VITAL (02/2015-07/2017)

The H2020 VITAL project [103] addresses the integrationof Terrestrial and Satellite networks through the applicabilityof two key networking paradigms, SDN and NFV. The mainVITAL outcomes consist of (i) the virtualization and abstrac-tion of satellite network functions and (ii) supporting multi-domain service/resource orchestration capabilities for a hybridcombination of satellite and terrestrial networks [104].

The VITAL overall architecture stands in line with themain directions established by the ETSI ISG NFV [33], withadditional concepts extended to the satellite communicationdomains and network service orchestration deployed acrossdifferent administrative domains. This architecture includes,among other, functional entities (e.g., NFVO, VNFM, SO,Federation Layer) for the provision and management of theNS lifecycle. In addition, a physical network infrastructureblock with virtualization support includes SDN and non-SDN(legacy) based network elements for flexible and scalableinfrastructure management.

Implementing the relevant parts of the VITAL architec-ture, X–MANO [105] is a cross-domain network service

7https://github.com/sonata-nfv/

orchestration framework. It supports different orchestrationarchitectures such as hierarchical, cascading (or recursive) andpeer-to-peer. Moreover, it introduces an information modeland a programmable network service in order to enableconfidentiality and network service lifecycle programmability,respectively.

G. Other Research Efforts

Further architectural proposals and research contributionscan be found in the recent literature.

Recent research works have addressed the definition ofNFV/SDN architectures. Vilalta et al. [106] present andNFV/SDN architecture for delivery of 5G services acrossmulti technological and administrative domains. The solutionis different from the NFV reference architecture. It con-sists of four main functional blocks: Virtualized FunctionsOrchestrator (VF-O), SDN IT and Network Orchestrator,Cloud/Fog Orchestrator and SDN Orchestrator. The VF-Ois the main component orchestrating generalized virtualizedfunctions such as NFV and IoT. Giotis et al. [107] proposea modular architecture that enables policy-based managementof Virtualized Network Functions. The proposed architecturecan handle the lifecycle of VNFs and instantiate applicationsas service chains. The work also offers an Information Modeltowards map the VNF functions and capabilities.

The work in [108] proposes a novel network slicing manage-ment and orchestration framework. The proposed frameworkautomates service network design, deployment, configuration,activation, and lifecycle management in a multi-domain envi-ronment. It can manage resources of the same type such asNFV, SDN and Physical Network Function (PNF), belongingto different organizational domains and belonging to the samenetwork domain such as access, core, and transport.

Finally, there still exists a large set of NSO related projectssponsored by the European Union Horizon 2020 researchand innovation programme in the 5G Infrastructure PublicPrivate Partnership phases 1, 2, and 38. In different mannersthose projects relate to orchestration detaining common rela-tionship with open source projects such as the Open SourceMANO (OSM) initiative, further described. To quote some ofthem, for instance: 5GCity9 aims an infrastructure businessmodel for the 5G city networks; 5G-MEDIA10 works tointegrate media-industry applications with the underlying 5Gprogrammable service platform based on SDN/NFV technolo-gies; and Sat5G11 intends to establish a plug-and-play satelliteinfrastructure integrated with 5G connectivity aiming unservedand underserved areas. Such myriad of projects wa