HDFS on Kubernetes—Lessons Learned with Kimoon Kim
-
Upload
databricks -
Category
Data & Analytics
-
view
1.968 -
download
1
Transcript of HDFS on Kubernetes—Lessons Learned with Kimoon Kim
![Page 2: HDFS on Kubernetes—Lessons Learned with Kimoon Kim](https://reader033.fdocuments.us/reader033/viewer/2022050613/5a1527f57f8b9a65768b4707/html5/thumbnails/2.jpg)
Outline1. Kubernetes intro
2. Big Data on Kubernetes
3. Demo
4. Problems we fixed -- HDFS data locality
![Page 3: HDFS on Kubernetes—Lessons Learned with Kimoon Kim](https://reader033.fdocuments.us/reader033/viewer/2022050613/5a1527f57f8b9a65768b4707/html5/thumbnails/3.jpg)
KubernetesNew open-source cluster manager.
Runs programs in Linux containers.
1000+ contributors and 40,000+ commits.
![Page 4: HDFS on Kubernetes—Lessons Learned with Kimoon Kim](https://reader033.fdocuments.us/reader033/viewer/2022050613/5a1527f57f8b9a65768b4707/html5/thumbnails/4.jpg)
“My app was running fine until someone installed their software”
![Page 5: HDFS on Kubernetes—Lessons Learned with Kimoon Kim](https://reader033.fdocuments.us/reader033/viewer/2022050613/5a1527f57f8b9a65768b4707/html5/thumbnails/5.jpg)
More isolation is goodKubernetes provides each program with:
• a lightweight virtual file system
– an independent set of S/W packages
• a virtual network interface
– a unique virtual IP address
– an entire range of ports
• etc
![Page 6: HDFS on Kubernetes—Lessons Learned with Kimoon Kim](https://reader033.fdocuments.us/reader033/viewer/2022050613/5a1527f57f8b9a65768b4707/html5/thumbnails/6.jpg)
Kubernetes architecture
node A node B
Pod 1 Pod 2 Pod 3
10.0.0.2
196.0.0.5 196.0.0.6
10.0.0.3 10.0.1.2
Pod, a unit of scheduling and isolation.
• runs a user program in a primary container• holds isolation layers like an virtual IP in an infra container
![Page 7: HDFS on Kubernetes—Lessons Learned with Kimoon Kim](https://reader033.fdocuments.us/reader033/viewer/2022050613/5a1527f57f8b9a65768b4707/html5/thumbnails/7.jpg)
Big Data on Kubernetesgithub.com/apache-spark-on-k8s• Google, Haiwen, Hyperpilot, Intel, Palantir, Pepperdata, Red Hat,
and growing.• patching up Spark Driver and Executor code to work on
Kubernetes.
Yesterday’s talk: spark-summit.org/2017/events/apache-spark-on-kubernetes/
![Page 8: HDFS on Kubernetes—Lessons Learned with Kimoon Kim](https://reader033.fdocuments.us/reader033/viewer/2022050613/5a1527f57f8b9a65768b4707/html5/thumbnails/8.jpg)
Spark on Kubernetes
node A node B
Driver Pod Executor Pod 1 Executor Pod 2
10.0.0.2
196.0.0.5 196.0.0.6
10.0.0.3 10.0.1.2
Client
ClientDriver Pod Executor Pod 1 Executor Pod 2
10.0.0.4 10.0.0.5 10.0.1.3
Job 1Job 2
![Page 9: HDFS on Kubernetes—Lessons Learned with Kimoon Kim](https://reader033.fdocuments.us/reader033/viewer/2022050613/5a1527f57f8b9a65768b4707/html5/thumbnails/9.jpg)
What about storage?Spark often stores data on HDFS.
How can Spark on Kubernetes access HDFS data?
![Page 10: HDFS on Kubernetes—Lessons Learned with Kimoon Kim](https://reader033.fdocuments.us/reader033/viewer/2022050613/5a1527f57f8b9a65768b4707/html5/thumbnails/10.jpg)
Hadoop Distributed File SystemNamenode• runs on a central cluster node.• maintains file system metadata.
Datanodes• on every cluster node.• read and write file data on local disks.
![Page 11: HDFS on Kubernetes—Lessons Learned with Kimoon Kim](https://reader033.fdocuments.us/reader033/viewer/2022050613/5a1527f57f8b9a65768b4707/html5/thumbnails/11.jpg)
HDFS on Kubernetes
node A node B
Driver Pod Executor Pod 1 Executor Pod 2
10.0.0.2
196.0.0.5 196.0.0.6
10.0.0.3 10.0.1.2
Client
ClientDriver Pod Executor Pod 1 Executor Pod 2
10.0.0.4 10.0.0.5 10.0.1.3
Job 1Job 2
Namenode Pod Datanode Pod 1 Datanode Pod 2
HDFS
hadoop.fs.defaultFS hdfs://hdfs-namenode-0.hdfs-namenode.default.svc.cluster.local:8020
![Page 12: HDFS on Kubernetes—Lessons Learned with Kimoon Kim](https://reader033.fdocuments.us/reader033/viewer/2022050613/5a1527f57f8b9a65768b4707/html5/thumbnails/12.jpg)
Demo1. Label cluster nodes2. Stand up HDFS3. Launch a Spark job4. Check Spark job output
![Page 13: HDFS on Kubernetes—Lessons Learned with Kimoon Kim](https://reader033.fdocuments.us/reader033/viewer/2022050613/5a1527f57f8b9a65768b4707/html5/thumbnails/13.jpg)
What about data locality?• Read data from local disks when possible• Remote reads can be slow if network is
slow
• Implemented in HDFS daemons, integrated with app frameworks like Spark
Client 2
node B
Client 1
node A
Datanode 1 Datanode 2
SLOWFAST
![Page 14: HDFS on Kubernetes—Lessons Learned with Kimoon Kim](https://reader033.fdocuments.us/reader033/viewer/2022050613/5a1527f57f8b9a65768b4707/html5/thumbnails/14.jpg)
HDFS data locality in YARNSpark driver sends tasks to right executors with tasks’ HDFS data.
Driver
Executor 1 Executor 2
/fileA /fileB
Read /fileA
Read /fileB
(/fileA → Datanode 1 → 196.0.0.5) == (Executor 1 →196.0.0.5)
Datanode 1 Datanode 2
node A196.0.0.5
node B196.0.0.6
Job 1
HDFS
YARN example
![Page 15: HDFS on Kubernetes—Lessons Learned with Kimoon Kim](https://reader033.fdocuments.us/reader033/viewer/2022050613/5a1527f57f8b9a65768b4707/html5/thumbnails/15.jpg)
Hmm, how do I find right executors in Kubernetes...
(/fileA → Datanode 1 → 196.0.0.5) != (Executor 1 → 10.0.0.3)
Executor Pod 2
10.0.1.2
Driver Pod Executor Pod 1
10.0.0.2 10.0.0.3
Read /fileB
Read /fileA
/fileA /fileBDatanode Pod 1
node A196.0.0.5
node B196.0.0.6
Kubernetes example
Datanode Pod 2
![Page 16: HDFS on Kubernetes—Lessons Learned with Kimoon Kim](https://reader033.fdocuments.us/reader033/viewer/2022050613/5a1527f57f8b9a65768b4707/html5/thumbnails/16.jpg)
Fix Spark Driver code1. Ask Kubernetes master to find the cluster node where
the executor pod is running.2. Get the node IP.3. Compare with the datanode IPs.
![Page 17: HDFS on Kubernetes—Lessons Learned with Kimoon Kim](https://reader033.fdocuments.us/reader033/viewer/2022050613/5a1527f57f8b9a65768b4707/html5/thumbnails/17.jpg)
Rescued data locality
Executor Pod 2
10.0.1.2
Driver Executor Pod 1
10.0.0.2 10.0.0.3
(/fileA → Datanode 1 → 196.0.0.5) == (Executor 1 →10.0.0.3 → 196.0.0.5)
Read /fileA
Read /fileB
/fileA /fileB
node A196.0.0.5
node B196.0.0.6
Datanode Pod 1 Datanode Pod 2
![Page 18: HDFS on Kubernetes—Lessons Learned with Kimoon Kim](https://reader033.fdocuments.us/reader033/viewer/2022050613/5a1527f57f8b9a65768b4707/html5/thumbnails/18.jpg)
Rescued data locality!with data locality fix
- duration: 10 minuteswithout data locality fix
- duration: 25 minutes
![Page 19: HDFS on Kubernetes—Lessons Learned with Kimoon Kim](https://reader033.fdocuments.us/reader033/viewer/2022050613/5a1527f57f8b9a65768b4707/html5/thumbnails/19.jpg)
![Page 20: HDFS on Kubernetes—Lessons Learned with Kimoon Kim](https://reader033.fdocuments.us/reader033/viewer/2022050613/5a1527f57f8b9a65768b4707/html5/thumbnails/20.jpg)
RecapGot HDFS up and running.Basic data locality works.
Open problems:• Remaining data locality issues -- rack locality, node preference, etc• Kerberos support• Namenode High Availability
![Page 21: HDFS on Kubernetes—Lessons Learned with Kimoon Kim](https://reader033.fdocuments.us/reader033/viewer/2022050613/5a1527f57f8b9a65768b4707/html5/thumbnails/21.jpg)
Join us!
● github.com/apache-spark-on-k8s● pepperdata.com/careers/
More questions?
● Come to Pepperdata booth #101● Mail [email protected]