Interoperability and scalability with microservices in science
-
Upload
ola-spjuth -
Category
Science
-
view
458 -
download
1
Transcript of Interoperability and scalability with microservices in science
![Page 1: Interoperability and scalability with microservices in science](https://reader035.fdocuments.us/reader035/viewer/2022062523/58738bae1a28ab272d8b6c51/html5/thumbnails/1.jpg)
Interoperability and scalability with microservices
Ola Spjuth <[email protected]>Science for Life Laboratory
Uppsala University
![Page 2: Interoperability and scalability with microservices in science](https://reader035.fdocuments.us/reader035/viewer/2022062523/58738bae1a28ab272d8b6c51/html5/thumbnails/2.jpg)
Service-Oriented Architectures (SOA) in the life sciences
• Standardize– Agree on e.g. interfaces, data formats,
protocols etc.• Decompose and compartmentalize
– Experts (scientists) should provide services – do one thing and do it well
– Achieve interoperability by exposing data and tools as Web services
• Integrate– Users should access and integrate
remote services
API
Scientist
service
Scientist
consume
![Page 3: Interoperability and scalability with microservices in science](https://reader035.fdocuments.us/reader035/viewer/2022062523/58738bae1a28ab272d8b6c51/html5/thumbnails/3.jpg)
Service-Oriented Architectures (SOA) in the life sciences, ~2005
Scientist
downtime
API changed
Not maintained
Difficult to sustain,unreliable solutions
APIAPIAPI
![Page 4: Interoperability and scalability with microservices in science](https://reader035.fdocuments.us/reader035/viewer/2022062523/58738bae1a28ab272d8b6c51/html5/thumbnails/4.jpg)
Cloud Computing
• Cloud computing offers advantages over contemporary e-infrastructures in the life sciences– On-demand elastic resources and services– No up-front costs, pay-per-use
• A lot of businesses (and software development) moving into the cloud– Vibrant ecosystem of frameworks and tools, including for
big data• High potential for science
![Page 5: Interoperability and scalability with microservices in science](https://reader035.fdocuments.us/reader035/viewer/2022062523/58738bae1a28ab272d8b6c51/html5/thumbnails/5.jpg)
5
Virtual Machines and Containers
Virtual machines• Package entire systems (heavy)• Completely isolated• Suitable in cloud environments
Containers:• Share OS• Smaller, faster, portable• Docker!
![Page 6: Interoperability and scalability with microservices in science](https://reader035.fdocuments.us/reader035/viewer/2022062523/58738bae1a28ab272d8b6c51/html5/thumbnails/6.jpg)
MicroServices
• Similar to Web services: Decompose functionality into smaller, loosely coupled services communicating via API– “Do one thing and do it well”
• Preferably smaller, light-weight and fast to instantiate on demand• Easy to replace, language-agnostic
– Suitable for loosely coupled teams (which we have in science)– Portable - easy to deploy and scale– Maximize agility for developers
• Suitable to deploy as containers in cloud environments
![Page 7: Interoperability and scalability with microservices in science](https://reader035.fdocuments.us/reader035/viewer/2022062523/58738bae1a28ab272d8b6c51/html5/thumbnails/7.jpg)
7
Scaling microservices
http://martinfowler.com/articles/microservices.html
![Page 8: Interoperability and scalability with microservices in science](https://reader035.fdocuments.us/reader035/viewer/2022062523/58738bae1a28ab272d8b6c51/html5/thumbnails/8.jpg)
8
Shippingcontainers?
![Page 9: Interoperability and scalability with microservices in science](https://reader035.fdocuments.us/reader035/viewer/2022062523/58738bae1a28ab272d8b6c51/html5/thumbnails/9.jpg)
9
Orchestrating containers
![Page 10: Interoperability and scalability with microservices in science](https://reader035.fdocuments.us/reader035/viewer/2022062523/58738bae1a28ab272d8b6c51/html5/thumbnails/10.jpg)
Kubernetes: Orchestrating containers
• A declarative language for launching containers
• Start, stop, update, and manage a cluster of machines running containers in a consistent and maintainable way
• Origin: Google• Suitable for microservices
Containers
Scheduled and packed containers on nodes
![Page 11: Interoperability and scalability with microservices in science](https://reader035.fdocuments.us/reader035/viewer/2022062523/58738bae1a28ab272d8b6c51/html5/thumbnails/11.jpg)
11
Virtual Research Environment (VRE)
• Virtual (online) environments for research– Easy and user-friendly access to computational resources, tools and
data, commonly for a scientific domain
• Multi-tenant VRE – log into shared system• Private VRE
– Deploy on your favorite cloud provider
![Page 12: Interoperability and scalability with microservices in science](https://reader035.fdocuments.us/reader035/viewer/2022062523/58738bae1a28ab272d8b6c51/html5/thumbnails/12.jpg)
• Horizon 2020-project, €8 M, 2015-2018– “standardized e-infrastructure for the processing, analysis and information-mining
of the massive amount of medical molecular phenotyping and genotyping data generated by metabolomics applications.”
• Enable users to provision their own virtual infrastructure (VRE)– Public cloud, private cloud, local servers– Easy access to compatible tools exposed as microservices– Will in minutes set up and configure a complete data-center (compute
nodes, storage, networks, DNS, firewall etc)– Can achieve high-availability, scalability and fault tolerance
• Use modern and established tools and frameworks supported by industry– Reduce risk and improve sustainability
• Offer an agile and scalable environment to use, and a straightforward platform to extend
http://phenomenal-h2020.eu/
![Page 13: Interoperability and scalability with microservices in science](https://reader035.fdocuments.us/reader035/viewer/2022062523/58738bae1a28ab272d8b6c51/html5/thumbnails/13.jpg)
Users should not see this…
![Page 14: Interoperability and scalability with microservices in science](https://reader035.fdocuments.us/reader035/viewer/2022062523/58738bae1a28ab272d8b6c51/html5/thumbnails/14.jpg)
![Page 15: Interoperability and scalability with microservices in science](https://reader035.fdocuments.us/reader035/viewer/2022062523/58738bae1a28ab272d8b6c51/html5/thumbnails/15.jpg)
Deployment and user access
Launch on reference installation
Launch on public cloudPrivate VRE
![Page 16: Interoperability and scalability with microservices in science](https://reader035.fdocuments.us/reader035/viewer/2022062523/58738bae1a28ab272d8b6c51/html5/thumbnails/16.jpg)
In-house deployment scenarios
MRC-NIHR Phenome Centre
• Medium-sized IT-infrastructure
• Dedicated IT-personnel
• Users: ICL staff
Hospital environment
• Dedicated server
• No IT-personnel• User: Clinical
researcher
Private VRE
![Page 17: Interoperability and scalability with microservices in science](https://reader035.fdocuments.us/reader035/viewer/2022062523/58738bae1a28ab272d8b6c51/html5/thumbnails/17.jpg)
Build and test tools, images, infrastructure
Docker Hub
PhenoMeNalJenkins
PhenoMeNalContainer Hub
Development: Container lifecycle
Source code repositories
![Page 18: Interoperability and scalability with microservices in science](https://reader035.fdocuments.us/reader035/viewer/2022062523/58738bae1a28ab272d8b6c51/html5/thumbnails/18.jpg)
Two proof of concepts so far
Kultima group Pablo Moreno
![Page 19: Interoperability and scalability with microservices in science](https://reader035.fdocuments.us/reader035/viewer/2022062523/58738bae1a28ab272d8b6c51/html5/thumbnails/19.jpg)
19
Implications
• Improve sustainability– Not dependent on specific data centers
• Improve reliability and security– Users can run their own service environments (VREs) within isolated
environments– High-availability and fault tolerance
• Scalability– Deploy in elastic environments
• Agile development– Automate “from develop to deploy”
• Agile science – Simple access to discoverable, scalable tools on elastic compute resources with
no up-front costs
• NB: Many problems of interoperability remains!– Data– APIs– etc.
![Page 20: Interoperability and scalability with microservices in science](https://reader035.fdocuments.us/reader035/viewer/2022062523/58738bae1a28ab272d8b6c51/html5/thumbnails/20.jpg)
20
Ongoing research on VREs
Datafederation
Computefederation
Privacypreservation
Workflows
Big Dataframeworks
Data management and modeling
![Page 21: Interoperability and scalability with microservices in science](https://reader035.fdocuments.us/reader035/viewer/2022062523/58738bae1a28ab272d8b6c51/html5/thumbnails/21.jpg)
21
Acknowledgements
Pharmb.ioWesley SchaalMaris LapinsJonathan AlvarssonArvid BergSamuel LampaMarco CapucciniMartin DahlöValentin GeorgievAnders LarssonPolina GeorgievStaffan ArvidssonLaeeq Ahmed
AstraZenecaLars CarlssonErnst Ahlberg-Helgee
University ViennaDavid KreilMaciej Kańduła
SNIC CloudSalman ToorAndreas Hellander
Caramba.clinicKim KultimaStephanie HermanPayam Emami
ToxHQ team