Building Blocks of Mayan: Componentizing the eScience Workflows Through Software-Defined Service...

Building Blocks of Mayan:Componentizing the eScience Workflows Through

Software-Defined Service Composition

Pradeeban Kathiravelu*, Tihana Galinac Grbac+, Luís Veiga*

*INESC-ID Lisboa & Instituto Superior Técnico, Universidade de Lisboa, Portugal+University of Rijeka, Croatia

23rd IEEE International Conference on Web Services (ICWS 2016)June 27 - July 2, 2016, San Francisco, USA.

Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 1 / 28

Overview

1 Introduction

2 Mayan Approach

3 Evaluation

4 Conclusion


Introduction

Introduction

eScience workflowsComputation-intensive.Execute on highly distributed networks.

Complex service compositions aggregating web servicesTo automate scientific and enterprise business processes.


Introduction

Motivation

Increasing demand forData quality and Quality of Service (QoS).Better Performance (Shorter completion times and higher throughput).Geo-distribution (workflows and compositions).

Need for additional control and flexibility.Exploring Trade-off: Efficiency vs. Accuracy.Leveraging Software-Defined Approaches (from SDN).


Introduction

Goals

Scalable Distributed Executions.High Scalability.Better orchestration.Data Quality Assurance.

Multi-Tenanted Environments.Isolation Guarantees.Differentiated Quality of Service (QoS).


Introduction

Contributions

Support for,Adaptive execution of scientific workflows.Flexible service composition.Reliable large-scale service composition.Efficient selection of service instances.


Mayan Approach

Mayan

Extensible SDN approach for cloud-scale service composition

Driven by:Loose couplingMessage-oriented Middleware (MOM)Availability of a logically centralized control plane

Leveraging OpenDaylight SDN controller as the core.Modular, as OSGi bundles.Additional advanced features.

State of executions and transactions stored in the controller distributeddata tree.Clustered and federated deployments.


Mayan Approach

Services as the building blocks of Mayan

Prototypical Example:


Mayan Approach



Mayan Approach

Multiple Implementations and Deployments of a Service


Mayan Approach



Mayan Approach

Services as the building blocks of Mayan



Mayan Approach

Too many requests on the fly?



Mayan Approach

Alternative Deployment/Implementation



Mayan Approach

Mayan Services Registry: Modelling Language


Mayan Approach

Service Composition Representation

<Service3,(<Service1, Input1>, <Service2, Input2>)>


Mayan Approach

Alternative Implementations and Deployments


Mayan Approach

Mayan Higher Level Deployment Architecture:Multi-Domain Workflows


Mayan Approach

Connecting Services View with the Network View


Evaluation

Evaluation System Configurations

Evaluation Approach:Smaller physical deployments in a cluster.Larger deployments as simulations and emulations (Mininet).

Evaluated Deployment:Service Composition Implementations.

Web services frameworks.Apache Hadoop MapReduce.Hazelcast In-Memory Data Grid.

OpenDaylight SDN Controller.


Evaluation

Preliminary Assessments

A workflow performing distributed data cleaning andconsolidation [PK 2015].

A distributed web service composition.vs.Mayan approach with the extended SDN architecture.


Evaluation

Speedup and Horizontal Scalability

No negative scalability in larger distributions.100% more positive scalability for larger deployments.


Evaluation

Memory consumption in the Service Nodes

Initial coordination overhead in memory for smaller deployments.Minimal overhead for larger deployments.


Conclusion

Related Work

MapReduce for efficient service compositions [SD 2014].

But we should not forget the registry!

Palantir: SDN for MapReduce performance with the network proximitydata [ZY 2014].A multi-domain deployment of SDN for communitynetworks [PK 2016].


Conclusion

Conclusion

SDN-based approach that enables large scale flexibility withperformance

Components in eScience workflows as building blocks of a distributedplatform.Service composition with web services and distributed executionframeworks.Multi-tenanted multi-domain executions.


Conclusion

Conclusion

SDN-based approach that enables large scale flexibility withperformance

Components in eScience workflows as building blocks of a distributedplatform.Service composition with web services and distributed executionframeworks.Multi-tenanted multi-domain executions.

Future WorkMayan should further be deployed and evaluated on physicalgeo-distributed nodes.Extending Software-defined service composition for the networkfunctions in service composition of middlebox actions.

Load balancing.Firewalls.

Adapting as an NFV framework for service function chaining.


Conclusion

References

PK 2015 Kathiravelu, Pradeeban, Helena Galhardas, and Luís Veiga. "∂u∂u Multi-Tenanted Framework: DistributedNear Duplicate Detection for Big Data." On the Move to Meaningful Internet Systems: OTM 2015Conferences. Springer International Publishing, 2015.

SD 2014 Deng, Shuiguang, et al. "Top-Automatic Service Composition: A Parallel Method for Large-Scale ServiceSets." Automation Science and Engineering, IEEE Transactions on 11.3 (2014): 891-905.

ZY 2014 Yu, Ze, et al. "Palantir: Reseizing network proximity in large-scale distributed computing frameworks usingsdn." 2014 IEEE 7th International Conference on Cloud Computing (CLOUD). IEEE, 2014.

PK 2016 Kathiravelu, Pradeeban, and Luıs Veiga. "CHIEF: Controller Farm for Clouds of Software-DefinedCommunity Networks." Software Defined Systems (SDS), 2016 IEEE International Symposium on. IEEE,2016.

Thank you!Questions?


Building Blocks of Mayan: Componentizing the eScience Workflows Through Software-Defined Service...

Software

Transcript of Building Blocks of Mayan: Componentizing the eScience Workflows Through Software-Defined Service...