SC1 Workshop 2 Technical overview
-
Upload
bigdataeurope -
Category
Technology
-
view
760 -
download
0
Transcript of SC1 Workshop 2 Technical overview
![Page 1: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/1.jpg)
SC1 Health WorkshopTechnical overview4 Oct 2016
![Page 2: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/2.jpg)
Platform goals
◎ Low total cost of ownership
◎ Simple to get started with Big Data
◎ Cater for widely varying use cases
◎ Embrace emerging Big Data technologies
◎ Simple integration with custom components
![Page 3: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/3.jpg)
Key actors
![Page 4: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/4.jpg)
Big Data is
◎ Volumeo Quantity of data
◎ Velocityo Speed at which data is provided
◎ Varietyo Different formats/models in which data is provided
◎ Veracityo Accuracy/truthfulness of the data
Why did we need all this?
![Page 5: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/5.jpg)
Platform architecture
![Page 6: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/6.jpg)
Platform architecture
![Page 7: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/7.jpg)
Platform architecture
![Page 8: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/8.jpg)
Semantic Big Data
ongoing research!
◎ Semantic Data Lakeo from data swamp to data lakeo query contents in the data lake
◎ SANSA stacko Big Data analytics on semantic graph
![Page 9: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/9.jpg)
Support layer
◎ Swarm UIo Launch, install and manage pipelines
◎ Pipeline daemon & monitoro Determine order in which steps are
executedo eg: Upload files before running
computations
◎ Integrator UIo Present dashboards in a unified interface
![Page 10: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/10.jpg)
Platform architecture
![Page 11: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/11.jpg)
Key actors
![Page 12: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/12.jpg)
Platform installation
![Page 13: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/13.jpg)
Platform installation
◎ Manual installation guide
◎Using Docker Machineo On local machine (VirtualBox)o In the cloud (AWS, DigitalOcean, Azure)o Bare metal
◎ Screencast
![Page 14: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/14.jpg)
Platform development
![Page 15: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/15.jpg)
◎ High level pictureo docker-compose.yml describes pipeline topology
◎ Common componentso extend template image with your code
◎ New componentso build a Docker image for your componento this is your own little Virtual Machine for your component
◎ Sharingo publish topology as git repositoryo publish new components on docker hub
Platform development
![Page 16: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/16.jpg)
Platform development
![Page 17: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/17.jpg)
Deployment
![Page 18: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/18.jpg)
Swarm UI
![Page 19: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/19.jpg)
Swarm UI
![Page 20: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/20.jpg)
Deployment
![Page 21: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/21.jpg)
Swarm UI
![Page 22: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/22.jpg)
Swarm UI
![Page 23: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/23.jpg)
Integrator UI
![Page 24: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/24.jpg)
Workflow UI
![Page 25: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/25.jpg)
More monitoring
This topic is ongoing
◎Custom User Interfaces
◎System output logs
◎Monitor network wire format (and visualise)?
◎Monitor node load (and autoscheduling)?
![Page 26: SC1 Workshop 2 Technical overview](https://reader035.fdocuments.us/reader035/viewer/2022081514/58816d851a28abf7478b4f55/html5/thumbnails/26.jpg)
Concluding remarks
◎Used in practice◎Easy to get started◎ Improving as we speak