Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS...
Transcript of Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS...
![Page 1: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/1.jpg)
![Page 2: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/2.jpg)
Index
![Page 3: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/3.jpg)
1 Presentation
![Page 4: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/4.jpg)
JORGE LÓPEZ-MALLA@jorgelopezmalla
After working with traditionalprocessing methods, I started todo some R&S Big Data projects
and I fell in love with the Big Dataworld. Currently I’m doing some
awesome Big Data projects and tools at Stratio.
SKILLS
Presentation
![Page 5: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/5.jpg)
MARCOS PEÑATE @marcosmi5
Coming from the Dark Side I found a place to grow and innovate at Stratio.
I am passionate about astrophysics, a compulsive SciFi consumer and I really
enjoy automating and Dockerizing everything around me!
SKILLS
Presentation
![Page 6: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/6.jpg)
Our Company
![Page 7: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/7.jpg)
Our product
![Page 8: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/8.jpg)
2 Use Case
![Page 9: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/9.jpg)
DC/OS
➢
➢
➢
![Page 10: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/10.jpg)
DC/OS
![Page 11: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/11.jpg)
➢
➢
➢
➢
➢
DC/OS Use Case: preconditions
![Page 12: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/12.jpg)
3 Spark, Kerberos & Mesos
![Page 13: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/13.jpg)
➢
➢
➢
Spark <3 Mesos
![Page 14: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/14.jpg)
Mesos Master
Agent-2
Executor-1 Executor-2
Task-0 Task-1Task-2 Task-3
HDFS
Agent-3
Executor-0 Executor-1
Agent-1
Executor-0
Main {…..….sc = new SparkContext()rdd1 = sc.textFile.map()rdd1.saveAs….}
Main {…..….sc = new SparkContext()rdd1 = sc.textFile.map()rdd1.saveAs….} Spark dispatcher
Driver
Driver
Mesos - Cluster
![Page 15: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/15.jpg)
Adding Kerberos to the equation
➢
➢
➢
➢
○
○
○
![Page 17: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/17.jpg)
tgstgt
read.format(parquet).path(“/user1/b”).option(“user”, “user2”)read.format(parquet).path(“b”).option(“user”, “user2”)
toDF.saveAsParquet(“b”)textFile(“a”)saveAsTextFile(“a”)
Driver Executor
sparkUser
user1
user2
HDFS
tgttgs
Stratio’s Solution
![Page 18: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/18.jpg)
4 Dynamic authentication
![Page 19: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/19.jpg)
➢
➢
➢
KMS
![Page 20: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/20.jpg)
Dynamic authentication
➢
➢
![Page 21: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/21.jpg)
DCS cluster
Spark - KMS
![Page 22: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/22.jpg)
Spark - KMS
![Page 23: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/23.jpg)
DCS cluster
Spark - KMS Dynamic Authentication
![Page 24: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/24.jpg)
★
★
★
Spark - KMS Dynamic Authentication
![Page 25: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/25.jpg)
5 Mutual TLS
Long handshakes are always bad!!
![Page 26: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/26.jpg)
Mutual TLS Datastores
![Page 27: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/27.jpg)
Mutual TLS Datastores
![Page 28: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/28.jpg)
Mutual TLS Datastores
![Page 29: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/29.jpg)
Mutual TLS Datastores
![Page 30: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/30.jpg)
Mutual TLS Datastores
![Page 31: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/31.jpg)
Spark - Mutual TLS Datastores
![Page 32: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/32.jpg)
6 Postgres
![Page 33: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/33.jpg)
Postgres
➢
➢
➢
![Page 34: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/34.jpg)
➢
➢
➢
Postgres
![Page 35: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/35.jpg)
7 Network isolation
![Page 36: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/36.jpg)
Are you sure that your cluster is secure?
DCS cluster
![Page 37: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/37.jpg)
➢
➢
➢
SDN
![Page 38: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/38.jpg)
DCS cluster
kafkakafka
Are you sure that your cluster is secure?
![Page 39: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/39.jpg)
7 Live Demo
![Page 40: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/40.jpg)
8 Q&A
![Page 41: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/41.jpg)
![Page 42: Index [events.static.linuxfound.org] · awesome Big Data projects and tools at Stratio. SKILLS Presentation. MARCOS PEÑATE @marcosmi5 Coming from the Dark Side I found a place to](https://reader035.fdocuments.us/reader035/viewer/2022070917/5fb78d695de43a00e90fc41f/html5/thumbnails/42.jpg)