Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… ·...
Transcript of Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… ·...
![Page 1: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/1.jpg)
Adventures & Challenges building an OpenStack public cloud
Walter Heukels, Koert van der Veer en Pim van Riezen
![Page 2: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/2.jpg)
The Sysadmin Experience
Walter Heukels
![Page 3: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/3.jpg)
About Me
• Walter Heukels – Senior Engineer at CloudVPS – Working on OpenStack Infra
![Page 4: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/4.jpg)
About OpenStack
• Free soJware (Apache License) • WriOen in Python
• Big project • Very flexible
![Page 5: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/5.jpg)
OpenStack AssumpSons
• Flexible, but it does have a philosophy – CaOle servers, not pets – Makes certain assumpSons
– We came up against some of those
• Don't go against the grain
![Page 6: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/6.jpg)
OpenStack AssumpSons
• Flexible, but it does have a philosophy – CaOle servers, not pets – Makes certain assumpSons
– We came up against some of those
• Don't go against the grain – Unless you really want to
![Page 7: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/7.jpg)
OpenStack Structure
• Lots of sub-‐projects – Nova – SwiJ – Quantum / Neutron
– Keystone – ...
• Distributed architecture
![Page 8: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/8.jpg)
Our Cloud
• Object Store – Since April 2013 – Very Cool
• Compute – Started free public beta – ZFS Storage – KVM VirtualisaSon
![Page 9: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/9.jpg)
Lessons Learned: Technical
• People are mostly running private clouds at the moment • Not much informaSon available on running a public cloud
• Examples – MulSple external networks
– Keystone (authenScaSon) performance
![Page 10: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/10.jpg)
Lessons Learned: ExpectaSons
• Customers make assumpSons • Especially our customers
• Examples: – IP spoof protecSon (“my VPN router doesn't work”)
– HA for VM's ..we're working on this!
![Page 11: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/11.jpg)
Lessons Learned: ExpectaSons
• Security groups?!? • Bitcoins!! • Some customers don't know what to expect “Will my
Wordpress site scale automaScally?”
![Page 12: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/12.jpg)
Lessons Learned: Debugging
• Distributed system • Race condiSons can occur • Hard to find the logging you need • Graph everything
![Page 13: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/13.jpg)
The Road Ahead
• New features – LBaaS – VPNaaS – Database as a Service – PaaS
![Page 14: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/14.jpg)
The Dev Experience
Koert van der Veer
![Page 15: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/15.jpg)
About Me
• Koert van der Veer – Senior Developer at CloudVPS – Working on OpenStack features
![Page 16: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/16.jpg)
My Role Before OpenStack
• Development responsible for every detail of cloud management system
• Large CompeStors are moving incredibly quickly – High pressure to add new features – No Sme to fix technical debt
![Page 17: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/17.jpg)
My Role With OpenStack
• Responsible for custom features only • Bugs are usually fixed by others • Large acSve community helps diagnosing problems • ContribuSng is very saSsfying and results in goodwill
![Page 18: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/18.jpg)
Our Work on OpenStack
• Core features (contributed) – ZFS block storage – SwiJ features – Bugfixes
• Deployment • Billing • Interfacing
![Page 19: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/19.jpg)
Development Tools
• Python with geventlet, kombu, sqlalchemy, etc. • DevStack • Unit tests • Tempest
• Grenade
![Page 20: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/20.jpg)
Development Environment
• ProducSon close to git head • Rapidly re-‐deployable testcluster • pip instell –e “.”
![Page 21: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/21.jpg)
Development Work Flow
• PreparaSon • Write code
• Review • Merge
• Maintain
![Page 22: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/22.jpg)
Step 1 -‐ PreparaSon
• Launchpad blueprints • Launchpad bugs • IRC • Mailinglist
![Page 23: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/23.jpg)
Step 2 -‐ Write Code
• Create feature branch • Write code
• Write unit tests • Run unit tests and staSc analysis • Commit
![Page 24: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/24.jpg)
Step 3 -‐ Review
• Submit to Gerrit • Jenkins tests • Other reviews • Core reviewer
approves
![Page 25: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/25.jpg)
Step 4 & 5 – Merge and Maintain
• Zuul reviews and audits code
• Jenkins merges code
![Page 26: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/26.jpg)
Case Study 1: Bug in Cinder-‐Rootwrap
• Bug detected (Jan 7th) • Bug fixed (Jan 9th) • SubmiOed to Gerrit (Jan 9th, Jan 10th) • Approved (Jan 14th) • Zuul rejected (Jan 15th) • SubmiOed to Gerrit (Jan 16th) • Approved (Jan 17th) • Zuul accepted (Jan 19th)
![Page 27: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/27.jpg)
Case Study 2: Custom Cinder Driver
• Goal • Challenges
– Bug in cinder-‐rootwrap – Feature completeness – Unit tests
• Progress
![Page 28: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/28.jpg)
Future Plans for ContribuSon
• High availability for VMs • Per-‐port IP spoofing control • Extra security msg queue • Requests?
![Page 29: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/29.jpg)
The Frontend Experience
Pim van Riezen
![Page 30: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/30.jpg)
About Me
• Pim van Riezen – Senior developer at CloudVPS – Working on OpenStack GUI
![Page 31: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/31.jpg)
Interfacing Goal
• Goal: “Make it easy to get started with a first VM”
![Page 32: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/32.jpg)
Interfacing Challenges
• Lots of dependencies: – Create a keypair – Create a private network – Create a NAT router – Create security groups and rules – Create Server
![Page 33: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/33.jpg)
Interfacing Challenges
• Decisions to make: – Networking – Key management
– …..
![Page 34: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/34.jpg)
Interfacing Challenges
• Security group abstracSon: – The double funcSon as membership tag and access rule grouping
confuses users
– It takes a lot of words to actually explain the concept
![Page 35: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/35.jpg)
Interfacing Challenges
• API documentaSon: – Hard to figure out what extensions are relevant – Different parts of an openstack cloud may be out of sync
– A lot of perculiar choices made in v1 APIs sSll leak through in v2
– Most command line tools also default to v1 APIs
– Lots of documentaSon lacks basic descripSons of parameters – CombinaSon of tracing command line client, making wild guesses, luck
![Page 36: Adventures*&*Challenges*building*an* OpenStackpubliccloud OpenStack P… · Adventures*&*Challenges*building*an* OpenStackpubliccloud ** Walter*Heukels, Koert*van*der*Veer* en*Pim*van*Riezen*](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f1810dd5aff7e357835dff4/html5/thumbnails/36.jpg)
Interfacing SoluSons
• SoluSons: – Comprehensive wizard
– Clear choices – Image metadata
– Predefined security groups