KubeCon EU 2016: Custom Volume Plugins
-
Upload
kubeacademy -
Category
Technology
-
view
218 -
download
0
Transcript of KubeCon EU 2016: Custom Volume Plugins
Just one thing
It's easy!
Easier than you might think
What's a
volume?
Persistentvs
Non persistent
User POV
spec: containers: - name: web image: nginx ports: - name: web containerPort: 80 volumeMounts: - name: www-root mountPath: "/usr/share/nginx/html"
volumes: - name: www-root flocker: datasetName: my-flocker-vol
... volumeMounts: - name: www-root mountPath: "/usr/share/nginx/html"
volumes: - name: www-root flocker: datasetName: my-flocker-vol
Available4 AWS EBS, GCE PD, Azure File
4 Git Repo
4 NFS
4 GlusterFS, Cinder
4 Flocker
4 Secrets
Is this
new?
HTTP API
/VolumeDriver.Create .Mount .Path .Unmount .Remove
Go interface{}
1. Add it to the API (pkg/api/{,v1/}types.go):
type VolumeSource struct { EmptyDir *EmptyDirVolumeSource `json:"emptyDir,omitempty"` Flocker *FlockerVolumeSource `json:"flocker,omitempty"` ...
2. Add it to the Kubelet (cmd/kubelet/app/plugins.go):
func ProbeVolumePlugins(pluginDir string) []volume.VolumePlugin { allPlugins := []volume.VolumePlugin{} allPlugins = append(allPlugins, empty_dir.ProbeVolumePlugins()...) allPlugins = append(allPlugins, git_repo.ProbeVolumePlugins()...) ...
3. Implement the volume plugin (pkg/volume/...):
type VolumePlugin interface { Init(host VolumeHost) error Name() string CanSupport(spec *Spec) bool
NewBuilder(spec *Spec, podRef *api.Pod, opts VolumeOptions) (Builder, error) NewCleaner(name string, podUID types.UID) (Cleaner, error)}
type VolumePlugin interface { Init(host VolumeHost) error Name() string CanSupport(spec *Spec) bool ...
... NewBuilder( spec *Spec, podRef *api.Pod, opts VolumeOptions, ) (Builder, error)
NewCleaner( name string, podUID types.UID, ) (Cleaner, error)}
Simplified example of NewBuilder
func (p *plg) NewBuilder(spec, pod, opts) (volume.Builder, error) { source, _ := p.getFlockerVolumeSource(spec) builder := flockerBuilder{ flocker: &flocker{ datasetName: source.DatasetName, pod: pod, ... }, ... } return &builder, nil}
4. Implement the builder
type Builder interface { Volume
SetUp(fsGroup *int64) error SetUpAt(dir string, fsGroup *int64) error GetAttributes() Attributes}
Simplified example of SetUpAt for git repo
I lied
func (b *gitRepoVolumeBuilder) SetUpAt(dir string, fsGroup *int64) error { if volumeutil.IsReady(b.getMetaDir()) { return nil }
wrapped, err := b.plugin.host.NewWrapperBuilder(b.volName, wrappedVolumeSpec, &b.pod, b.opts) if err != nil { return err } if err := wrapped.SetUpAt(dir, fsGroup); err != nil { return err }
args := []string{"clone", b.source}
if len(b.target) != 0 { args = append(args, b.target) } if output, err := b.execCommand("git", args, dir); err != nil { return fmt.Errorf("failed to exec 'git %s': %s: %v", strings.Join(args, " "), output, err) }
files, err := ioutil.ReadDir(dir) if err != nil { return err }
if len(b.revision) == 0 { // Done! volumeutil.SetReady(b.getMetaDir()) return nil }
var subdir string
switch { case b.target == ".": // if target dir is '.', use the current dir subdir = path.Join(dir) case len(files) == 1: // if target is not '.', use the generated folder subdir = path.Join(dir, files[0].Name()) default: // if target is not '.', but generated many files, it's wrong return fmt.Errorf("unexpected directory contents: %v", files) }
if output, err := b.execCommand("git", []string{"checkout", b.revision}, subdir); err != nil { return fmt.Errorf("failed to exec 'git checkout %s': %s: %v", b.revision, output, err) } if output, err := b.execCommand("git", []string{"reset", "--hard"}, subdir); err != nil { return fmt.Errorf("failed to exec 'git reset --hard': %s: %v", output, err) }
volumeutil.SetReady(b.getMetaDir()) return nil}
1. Check meta: was it called before? Is it ready?
2. Use empty dir to prepare the path
3. git clone it
4. Checkout the revision sent on the pod definition
5. Job done? Mark as ready
Important bit of empty_dir
switch ed.medium { case api.StorageMediumDefault: err = ed.setupDir(dir) case api.StorageMediumMemory: err = ed.setupTmpfs(dir, securityContext) default: err = fmt.Errorf("unknown storage medium %q", ed.medium) }
volume.SetVolumeOwnership(ed, fsGroup)
Another simplified example for Flocker
I liedagain
func (b flockerBuilder) SetUpAt(dir string, fsGroup *int64) error { if volumeutil.IsReady(b.getMetaDir()) { return nil }
if b.client == nil { c, err := b.newFlockerClient() if err != nil { return err } b.client = c }
datasetID, err := b.client.GetDatasetID(dir) if err != nil { return err }
s, err := b.client.GetDatasetState(datasetID) if err != nil { return fmt.Errorf("The volume '%s' is not available in Flocker. You need to create this manually with Flocker CLI before using it.", dir) }
primaryUUID, err := b.client.GetPrimaryUUID() if err != nil { return err }
if s.Primary != primaryUUID { if err := b.updateDatasetPrimary(datasetID, primaryUUID); err != nil { return err } }
b.flocker.path = s.Path volumeutil.SetReady(b.getMetaDir()) return nil}
1. Was it called? Is it ready?
2. Is the dataset ready? If not, not my problem mate
3. If primary UUIDs doesn't match (rescheduled pod), update them
4. Wait until ready
5. Mark as ready
5. Implement the Persistent Volume Plugin
type PersistentVolumePlugin interface { VolumePlugin
GetAccessModes() []api.PersistentVolumeAccessMode}
func (p *plg) GetAccessModes() []api.PersistentVolumeAccessMode { return []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, }}
http://bit.ly/kubecon
"Problems"
4 CLAs
4 Shippable & Jenkins
4 hack/ scripts
4 PR
Summary1. Add API
2. Enable the plugin on the Kubelet
3. Implement VolumePlugin interface: NewBuilder & NewCleaner
4. Implement the builder itself: SetUpAt
5. Persistent?
6. ❤
Just one (more) thing
We are hiring at
Thanks!@agonzalezro