DIABLO VS HAVANAHOW OPENSTACK HAS MATURED

Joe Gordon

ABOUT ME Developer @

Full time upstream OpenStack developernova-coreATC since EssexInterested in making OpenStack scale and workcontact information

jog0 on freenodegithub.com/jogolaunchpad.net/~jogo

OPENSTACK FROM DIABLO TO HAVANAA BRIEF HISTORY OF THE INTEGRATED RELEASE

DIABLOSwiftNova

Nova-volumeNova-networkIdentity

GlanceKeystone*

ESSEXSwiftNova

Nova-volumeNova-networkIdentity

GlanceKeystoneHorizon

FOLSOMSwiftNova

Nova-volumeNova-network

GlanceKeystoneHorizonQuantum (Neutron)Cinder

GRIZZLYSwiftNova

Nova-networkGlanceKeystoneHorizonQuantum (Neutron)Cinder

Total number of OpenStack services: 32

HAVANASwift (4)Nova (6)

Nova-networkGlance (2)Keystone (1)Horizon (1)Neutron (6)Cinder (4)Ceilometer (4)Heat (4)

HAVANA FEATURES SUPPORTED BY DIABLOSwiftNova

Nova-networkGlanceKeystoneHorizonNeutron (Quantum)CinderCeilometerHeat

HAVANA FEATURES SUPPORTED BY DIABLOSwiftNova

Nova-networkGlanceKeystoneCinder

OPENSTACK HAS GROWN

GROWN OUTNEW SERVICESHeatCeilometerMore in pipeline

IcehouseTrove

IncubationIronicMarconiSavanna

GROWN UPExpand existing APIs

Split out projectsExtensions

DIABLO NOVA EXTENSIONS| Hosts | Host administration| Volumes | Volumes support| Quotas | Quotas management support| SimpleTenantUsage | Simple tenant usage extension| FlavorExtraData | Provide additional data for flavors| SecurityGroups | Security group support| FlavorExtraSpecs | Instance type (flavor) extra specs| Createserverext | Extended support to the Create Server v1.1 API| Multinic | Multiple network support| VSAs | Virtual Storage Arrays support| VolumeTypes | Volume types support| Keypairs | Keypair Support| Rescue | Instance rescue mode| VirtualInterfaces | Virtual interface support| Floating_ips | Floating IPs support

POST DIABLO NOVA EXTENSIONS AdminActions Agents Aggregates AssistedVolumeSnapshots AttachInterfaces AvailabilityZone BareMetalExtStatus BareMetalNodes BlockDeviceMappingV2Boot CellCapacities Cells Certificates Cloudpipe CloudpipeUpdate ConfigDrive ConsoleOutput Consoles Coverage DeferredDelete DiskConfig Evacuate ExtendedAvailabilityZone ExtendedFloatingIps ExtendedIps ExtendedIpsMac ExtendedQuotas ExtendedServerAttributes ExtendedServices ExtendedStatus ExtendedVIFNet ExtendedVolumes FixedIPs FlavorAccess FlavorDisabled FlavorManage FlavorRxtx FlavorSwap FloatingIpDns FloatingIpPools FloatingIps FloatingIpsBulk Fping HideServerAddresses Hypervisors ImageSize InstanceActions Migrations MultipleCreate NetworkAssociationSupport Networks OSInstanceUsageAuditLog OSTenantNetworks QuotaClasses SchedulerHints SecurityGroupDefaultRules ServerDiagnostics ServerPassword ServerStartStop ServerUsage Services Shelve UsedLimits UsedLimitsForAdmin UserData UserQuotas VolumeAttachmentUpdate

SO ONCE WE NORMALIZEFOR THE GROWTH IN

OPENSTACK WHAT ELSEHAVE WE DONE IN THEPAST FOUR RELEASES?

CHANGESDriversTesting and bugsPerformance and scalabilityForward facing internal changesProcess

MORE DRIVERSNova: baremetal, docker, hyperv, powervm

Cinder: coraid, emc, glusterfs, gpfs, nexenta, rbd, sheepdog,soldfire, ...

BETTER TESTING, FEWER BUGSUnit testsIntegration tests (Tempest)Lots of fixed bugs

UNIT TESTSPERCENT UNIT TESTS BY LOC

Project Diablo HavanaNova 43% 56%Glance 62% 62%Swift 65% 71%Keystone 41% 45%

TEMPESTTHE OPENSTACK INTEGRATION TEST SUITE

DIABLO TEMPESTOrignally named 'openstack-integration-tests'Just 107 tests*

extensionsflavorsfloating ipsimage metadataimageskeypairsserversvolumes

2,655 lines of python codeFirst release of Tempest

HAVANA TEMPEST1275* tests

1150 api testscompute, image, identity, network, object_storage,orchestration, volume102 test files

92 cli tests11 scenario tests18 thirdparty (boto) tests4 other

Parallel20-30 minutes to run in gate24,837 lines of python code

HAVANA API TESTS785 compute216 identity24 image17 network50 object_storage54 volume

HAVANA TEMPEST VS OPENSTACK DIABLOSame APIs as Diablo plus new onesDiablo has been 'end of life'edMany tests cover new features. Ignore thoseTempest not designed to be version independentGot 622 tests to run and 154 (27%) to pass.*

110 of which were in compute

HAVANA TEMPEST VS OPENSTACK DIABLORESULTS

Diablo has poor parameter validationInvalid key namesMin/max count for multiple creates

Changed return codes for negative casesReturn empty list and 200 instead of 400 when invalid filtervalue in nova list serversWe shouldn't have done this

Poor pagination in DiabloJSON only tests in Diablo, Havana tests XML tooMany false negatives in results due to slight changes in nowAPIs and metadata are used

CRITICAL BUGSProject Diablo Essex Folsom Grizzly HavanaNova 14 22 30 31 33Glance 5 15 30 13 8Keystone 31 11 9 8Cinder 10 3 12Neutron 27 14 24

CategoriesNon-gated featuresDependenciesRace conditionsPerformance

PERFORMANCE ANDSCALABILITY

PERFORMANCE AND SCALABILITYNova-scheduler

Active-activeNo RPC fanout

DBIndicesSmarter queriesFewer queries

service groupHeartbeat to DB every 10 seconds1k DB calls a second if have 10K nodes.Adjustable intervalAlternatives memcache and zookeeper

PERFORMANCE AND SCALABILITYMore efficient image work flow

Glance, Nova, CinderMove large files around less

PKI TokensFewer round trips to keystone

PERFORMANCE AND SCALABILITYCASE STUDY FROM HP PUBLIC CLOUD

Diablo40,836 LOC changed to 239 files in nova2,265 LOC to manage DB+100s of hours of DB tuning to make it work with morethen just a few nodes

HavanaDB just worked1,167 LOC changed in 35 filesMost of that is in patches in review today

Thanks to Thom Leggett for these statistics

FORWARD FACING INTERNAL CHANGESSplit out projectsOsloRoad to live and continuous upgradesIterate on APIs

PROCESS

DRIVING PRINCIPLESNever break trunk

Master branch is always greenDevelopers are never blocked

TransparencyAutomate everythingEgalitarianBe strict. Reduce burden on reviewers

DIABLOAutomated Gate87 Contributors in September 2011 (release of Diablo)No GateNo Git or GerritBzr and Launchpad

HAVANA346 Contributors in October 2013 (release of Havana)

Scale development processIntegrated Gate

9 Projects30+ separate services1 Merge pipeline

New tools for an asynchronous worldTests not pass fail, rather classified with a race percentageFix that race condition we hit 0.5% of the timeDebugging is hardelastic-recheck

HOW WILL OPENSTACK KEEP MATURINGWHAT CHALLENGES ARE WE FACING TODAY?

More testingLive rolling upgradesLimiting factor to the whole system is reviewers (and not justcore)Improving quality while growing the project as a wholeScalability

THANK YOUQUESTIONS?

Powered by reveal.js