A Full Stack Niels de Vos journey to reach Gluster...
Transcript of A Full Stack Niels de Vos journey to reach Gluster...
![Page 1: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/1.jpg)
A Full Stack journey to reach Efficient Container Storage Cloning
A Full Stack journey to reach Efficient Container Storage Cloning
Niels de VosGluster MaintainerEngineer at Red [email protected]
Niels de VosGluster MaintainerEngineer at Red [email protected]
@nixpanic@nixpanic
![Page 2: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/2.jpg)
AgendaAgenda
• Brief Introduction into Kubernetes Storage
• KubeVirt storage requirements
• Containerized-Data-Importer and Cloning
• Linux filesystem support for copy_file_range()
• Passing copy_file_range() on to Gluster
• Brief Introduction into Kubernetes Storage
• KubeVirt storage requirements
• Containerized-Data-Importer and Cloning
• Linux filesystem support for copy_file_range()
• Passing copy_file_range() on to Gluster
![Page 3: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/3.jpg)
Start of the trip with Kubernetes and KubeVirt
Start of the trip with Kubernetes and KubeVirt
![Page 4: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/4.jpg)
Brief Introduction into Kubernetes StorageBrief Introduction into Kubernetes Storage
• Dynamic provisioned persistent storage
• PersistenVolumeClaim and StorageClass
• Storage provisioners
• Internal
• External Storage project
• Container Storage Interface
• Dynamic provisioned persistent storage
• PersistenVolumeClaim and StorageClass
• Storage provisioners
• Internal
• External Storage project
• Container Storage Interface
![Page 5: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/5.jpg)
StorageClass ExampleStorageClass Example
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: glusterfile
provisioner: gluster.org/glusterfile
parameters:
resturl: "http://127.0.0.1:8081"
restuser: "admin"
restsecretnamespace: "default"
restsecretname: "heketi-secret"
#clusterid: "454811fcedbec6316bc10e591a57b472"
volumetype: "replicate:3"
volumeoptions: "features.shard enable"
volumenameprefix: "dept-dev"
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: glusterfile
provisioner: gluster.org/glusterfile
parameters:
resturl: "http://127.0.0.1:8081"
restuser: "admin"
restsecretnamespace: "default"
restsecretname: "heketi-secret"
#clusterid: "454811fcedbec6316bc10e591a57b472"
volumetype: "replicate:3"
volumeoptions: "features.shard enable"
volumenameprefix: "dept-dev"
![Page 6: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/6.jpg)
PersistentVolumeClaim ExamplePersistentVolumeClaim Example
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: my-persistent-data
spec:
storageClassName: "glusterfile"
accessModes:
- ReadWriteMany
resources:
requests:
storage: 4Gi
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: my-persistent-data
spec:
storageClassName: "glusterfile"
accessModes:
- ReadWriteMany
resources:
requests:
storage: 4Gi
![Page 7: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/7.jpg)
KubeVirt storage requirementsKubeVirt storage requirements
• Virtual Machines in containers managed by Kubernetes
• Disk-image on a PersistentVolume
• Cloning new VMs from a Template
• Scalable and resilient
• Virtual Machines in containers managed by Kubernetes
• Disk-image on a PersistentVolume
• Cloning new VMs from a Template
• Scalable and resilient
![Page 8: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/8.jpg)
Brief stop at the Containerized-Data-Importer
Brief stop at the Containerized-Data-Importer
![Page 9: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/9.jpg)
Containerized-Data-ImporterContainerized-Data-Importer
• DataVolume (CRD) as an extension to PVCs
• Pre-populating new disk-images
• Download image from the network
• Delay ‘readiness’ until the PVC is provisioned
• DataVolume (CRD) as an extension to PVCs
• Pre-populating new disk-images
• Download image from the network
• Delay ‘readiness’ until the PVC is provisioned
![Page 10: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/10.jpg)
Containerized-Data-ImporterContainerized-Data-Importer
![Page 11: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/11.jpg)
CDI ExampleCDI Example
apiVersion: cdi.kubevirt.io/v1alpha1
kind: DataVolume
metadata:
name: "example-import-dv"
spec:
source:
http:
url: "https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img" # Or S3
secretRef: "" # Optional
pvc:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "64Mi"
apiVersion: cdi.kubevirt.io/v1alpha1
kind: DataVolume
metadata:
name: "example-import-dv"
spec:
source:
http:
url: "https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img" # Or S3
secretRef: "" # Optional
pvc:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "64Mi"
![Page 12: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/12.jpg)
CDI and CloningCDI and Cloning
• Supports cloning of PVCs by Annotations
• CloneRequest is passed in the PVC
• CloneOf is returned by the provisioner
• Host-Assisted cloning as fallback
• Supports cloning of PVCs by Annotations
• CloneRequest is passed in the PVC
• CloneOf is returned by the provisioner
• Host-Assisted cloning as fallback
![Page 13: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/13.jpg)
CDI and Host-Assisted CloningCDI and Host-Assisted Cloning
![Page 14: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/14.jpg)
CDI Example of a DataVolume cloneCDI Example of a DataVolume clone
apiVersion: cdi.kubevirt.io/v1alpha1
kind: DataVolume
metadata:
name: "example-clone-dv"
spec:
source:
pvc:
name: source-pvc
namespace: example-ns
pvc:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "128Mi"
apiVersion: cdi.kubevirt.io/v1alpha1
kind: DataVolume
metadata:
name: "example-clone-dv"
spec:
source:
pvc:
name: source-pvc
namespace: example-ns
pvc:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "128Mi"
![Page 15: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/15.jpg)
CDI Example of a PVC with CloneRequestCDI Example of a PVC with CloneRequest
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: "target-pvc"
namespace: "target-ns"
labels:
app: Host-Assisted-Cloning
annotations:
k8s.io/CloneRequest: "source-ns/golden-pvc"
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: "target-pvc"
namespace: "target-ns"
labels:
app: Host-Assisted-Cloning
annotations:
k8s.io/CloneRequest: "source-ns/golden-pvc"
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
![Page 16: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/16.jpg)
Short detour passing Linux filesystemsShort detour passing Linux filesystems
![Page 17: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/17.jpg)
Linux filesystem support for copy_file_range()Linux filesystem support for copy_file_range()
• Linux VFS uses clone_file_range()
• XFS implements clone_file_range() with ‘reflinks’
• FUSE supports clone_file_range() and passes it on to the userspace process handling the filesystem
• Fallback in glibc when copy_file_range() is unsupported
• Linux VFS uses clone_file_range()
• XFS implements clone_file_range() with ‘reflinks’
• FUSE supports clone_file_range() and passes it on to the userspace process handling the filesystem
• Fallback in glibc when copy_file_range() is unsupported
![Page 18: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/18.jpg)
Linux filesystem support for copy_file_range()Linux filesystem support for copy_file_range()
• Fallback in glibc when copy_file_range() is unsupported
• Very similar to Host-Assisted cloning
• Fallback in glibc when copy_file_range() is unsupported
• Very similar to Host-Assisted cloning
![Page 19: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/19.jpg)
Reaching the destination on a solid brick road
Reaching the destination on a solid brick road
![Page 20: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/20.jpg)
Passing copy_file_range() on to GlusterPassing copy_file_range() on to Gluster
• Restricted to single mountpoint
• PV to be placed in a subdirectory
• Requires support in FUSE and the client-side mount process
• Network protocol additions from client to server
• Restricted to single mountpoint
• PV to be placed in a subdirectory
• Requires support in FUSE and the client-side mount process
• Network protocol additions from client to server
![Page 21: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/21.jpg)
Handling copy_file_range() in GlusterHandling copy_file_range() in Gluster
• Receive copy_file_range() calls from clients
• Bricks need to be on a filesystem with support for copy_file_range()
• Limitations with distributed Gluster Volumes
• Receive copy_file_range() calls from clients
• Bricks need to be on a filesystem with support for copy_file_range()
• Limitations with distributed Gluster Volumes
![Page 22: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/22.jpg)
Efficient Cloning with subdirectoriesEfficient Cloning with subdirectories
![Page 23: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/23.jpg)
ReferencesReferences
• kubevirt.io• Containerized-Data-Importer
• Talk by Fabian Deutsch, Wed. 14:15
• Kubernetes External Storage
• Gluster Subvol Provisioner (PR#1013)
• gluster.org
• FUSE clone_file_range() (Linux, libfuse)
• https://people.redhat.com/ndevos/talks/2018-10-OSSEU
• kubevirt.io• Containerized-Data-Importer
• Talk by Fabian Deutsch, Wed. 14:15
• Kubernetes External Storage
• Gluster Subvol Provisioner (PR#1013)
• gluster.org
• FUSE clone_file_range() (Linux, libfuse)
• https://people.redhat.com/ndevos/talks/2018-10-OSSEU
![Page 24: A Full Stack Niels de Vos journey to reach Gluster ...people.redhat.com/ndevos/talks/2018-10-OSSEU/full... · Brief Introduction into Kubernetes Storage • Dynamic provisioned persistent](https://reader034.fdocuments.us/reader034/viewer/2022050515/5f9fb3fe36298977453bd8ef/html5/thumbnails/24.jpg)