Percona presentation v2

28
Challenges and lessons learned with Openstack deployments and MySQL Sandro Mazzio+a Sr Director Product Management IaaS

description

 

Transcript of Percona presentation v2

Page 1: Percona presentation v2

Challenges  and  lessons  learned  with  Openstack  deployments  and  MySQL  

Sandro  Mazzio+a  Sr  Director  Product  Management  IaaS  

Page 2: Percona presentation v2

Who  I  am    

o  I  am  not  a  DB  expert  or  a  DBA  o  I  work  in  Product  Management  since  15+  years  o  I  work  at  eNovance  since  2  months  o  I  started  with  Openstack  in  July  2011  o  I’ve  been  launching  HP  Cloud  Services  DBaaS  offering  

Page 3: Percona presentation v2

San Francisco

Montréal

Paris

Bangalore

eNovance  loca>ons  and  figures  

People

 

   

Customers    

 

 Growth in Revenue  

Founded in

 

   

Page 4: Percona presentation v2

Value   Engineering  

Softw

are

Software Architecture : from centralized apps to distributed services Software Engineering : from Waterfall to Agile

Infra

stru

ctur

e

IT Infrastructure : from Mainframe to Cloud Cloud Infrastructure Engineering : from Ops to DevOps

Journey  to  the  cloud    

Client / Server Vertical Scaling

Hardware is key Centralized

Services / API Horizontal Scaling

Software is key Distributed

Expensive Hw Vertical Scaling

Monolithic Centralized

Cheap Hw Horizontal Scaling

Distributed

Page 5: Percona presentation v2

Market  &  opportunity      Large

Enterprises Software for Private Cloud: $ 10 bn by 2017

Telcos xSP Software for Public Cloud : $ 8 bn by 2017

Telecom Equipment Vendors NFV : $ 2bn by 2017

Not Scalable Expensive Not Cloud

Rich Set of Features No Versioning

Shared Infrastructure

Not Scalable Expensive Not Cloud

Containers Not Scalable Not Cloud

Migrate from Physical Appliance To Network Functions Virtualization

eNovance   Cloud  Solu/ons

Seamless  Build  &  Delivery   of  OpenStack  Cloud Infrastructures

Page 6: Percona presentation v2

Build a Cloud Deliver your Private or Public Cloud on Premise with a real Cloud Approach :

•  No Versioning •  Pay as you Grow •  AWS Features Match •  Enterprise Grade SLA •  24/7 Support

 

 

 

Use a Cloud  

Deliver Cloud-Ready applications on top of existing Cloud infrastructures : •  Elastic & Scalable •  Continuous Delivery •  Pay as you Grow •  Enterprise Grade SLA •  24/7 Support    

What  we  do  ?  

 

 

 

Page 7: Percona presentation v2

Openstack  &  eNovance  

 

 

 

q Gold  Member  

q   Top  Ten  Contributor  

q Share  DNA  Open  Cloud,  Open  Source  

Page 8: Percona presentation v2

Openstack  is  not  simple    

Page 9: Percona presentation v2

Openstack  is  not  cloud    (Subbu  Allamaraju  eBay,  July  25,  2013)  

All  key  requirements  for  producRon:  •  By  Enterprise  for  Private  Cloud  •  By  Service  Providers  for  Public  Cloud  

“infra  onboarding,  boostrapping,  remediaRon,  config  management,  patching,  packaging,  upgrades,  high  availability,  monitoring,  metrics,  user  support,  capacity  forecasRng  and  management,  billing  or  chargeback,  reclamaRon,  security,  firewalls,  DNS,  integraRon  with  other  internal  infrastructure  and  tools,  and  on  and  on  and  on”.  

Page 10: Percona presentation v2

Openstack  Upstream  does  not  address  HA    

Openstack  HA  Guide    

h+p://docs.openstack.org/high-­‐availability-­‐guide/content/

index.html  

ApplicaRon  on  top  Openstack  OS  Infra.  Plane:  

 Openstack  API    Message  Queues    DB  

VirtualizaRon  Layer  …  

Page 11: Percona presentation v2

eNovance  Openstack  Reference  Architecture  V1.0    

Ø Management  cluster  full  highly  available  Ø OpenStack  API,  schedulers  Ø  RabbitMQ,  Database,  Cache  Ø Network  services  

Ø Compute  nodes  with  KVM  and  Open-­‐vSwitch  with  GRE  tunnels  as  overlay  

Ø Swia  cluster  Ø Ceph  cluster,  used  as  backend  for  Nova,  

Glance  and  Cinder  

Page 12: Percona presentation v2

eNovance  Openstack  product    

•  Contains  all  core  projects  of  Openstack  

•  Supports  HA  in  a  Single  DC  mode  •  Based  on    •  eDeploy  (eNovance  Bare  Metal  Deployment  Opensource)  

•  Puppet  as  Soaware  ConfiguraRon  Management  

Page 13: Percona presentation v2

What  about  DB  in  our  product?  

DB  Requirements:  FuncRonal  &  Performance  requirements:    •  100  %  upRme  •  Performance  DegradaRon  is  ok  

 Business  Requirements:  

Opensource  

Galera  cluster  for  MySQL  

Page 14: Percona presentation v2

eNovance  Galera  for  MySQL  usage:  results    

Experience  from  ProducRon  Environment:  1  cluster  of  3  Management  Nodes  

•  No  Service  InterrupRon  •  Performance  DegradaRon  are  OK  •  Galera  performed  correctly  

Page 15: Percona presentation v2

On  the  topic  of  Mul>  Data  Center    

West 1

East

EMEA

ASIA

Goal  is  to  address:    

•  Disaster  Recovery  •  Local  Affinity  

Page 16: Percona presentation v2

Not  all  services  are  ready  for  Mul>-­‐DC    

•  Openstack  has  a  concept  of  AZ,  Host  AggregaRon  =  good  in  a  single  DC  .  Not  good  If  your  DC  are  separated  by  a  WAN…  

•  Nova  Cells  is  good  for  distribuRon  (but  nobody  else  is  taken  care)…  •  MulRple  Region  –  Keystone  in  every  region  or  one  overlay    

Galera  3  Galera  SynchronizaRon  over  

WAN  

Using  Heat  to  orchestrate  MulRple  Region  

Page 17: Percona presentation v2

Keystone    

Our  experience  working  at  large  customers  is  that:

•  Token  storage  is  a  nightmare  and •  and  do  not  scale  well  on  large  scale  

deployment  

•  Key  Value  Store  (memcache,  redis,  …)  -­‐>  MySQL  -­‐>  memcache  (but  we  have  a  bug)…  euh…

Page 18: Percona presentation v2

Everything  is  MySQL,  almost…    

VS  

Page 19: Percona presentation v2

Ceilometer,  we  use  mongoDB    

But  Requires  MongoDB  DBA,  Runbook,  skills,  

…  

Because  we  need  scalibility  and  because  of  the  nature  of  informaRon  to  store  

Page 20: Percona presentation v2

Openstack  is  not  cloud    (Subbu  Allamaraju  eBay,  July  25,  2013)  

All  key  requirements  for  producRon:  •  By  Enterprise  for  Private  Cloud  •  By  Service  Providers  for  Public  Cloud  

“infra  onboarding,  boostrapping,  remediaRon,  config  management,  patching,  packaging,  upgrades,  high  availability,  monitoring,  metrics,  user  support,  capacity  forecasRng  and  management,  billing  or  chargeback,  reclamaRon,  security,  firewalls,  DNS,  integraRon  with  other  internal  infrastructure  and  tools,  and  on  and  on  and  on”.  

Page 21: Percona presentation v2

Why  we  need  update  –  upgrade  –patching…  ?    

User  

CRM  

Billing  

Portal  

seongs  changes  

new  release  

bug  fixes   driver  firmware  update  

IntegraRon  mess  

Page 22: Percona presentation v2

What  is  our  vision    

Deliver  Openstack  and  app  on  top  of  it  using  Openstack  process  and  tools  ConRnuous  IntegraRon/Delivery  

Page 23: Percona presentation v2

Leverage  the  Openstack  process    

How  Openstack  handle  •  Contribu/on  of  400  developers  /  month? •  Developers  from  250  companies? •  10+  integrated  projects?

and  yet  release  a  new  version  •  every  6  months  •  with  400  new  features  •  without  major  inconsistencies?  

Page 24: Percona presentation v2

…  and  tools    

Everything  in

•  Process  descrip/on  and  code •  Infrastructure  used  to  handle  the  process •  Code  to  deploy  the  infrastructure

100  Rmes  be+er  than  documentaRon  of  a  manual  process  that  is  never  up  to  date...  

Reproducible  environments    

•  Dynamic  Provision  test  env  when  needed  by  Jenkins

•  Allow  dev  to  generate  a  test  env  for  debug  at  will

•  Store  validated  releases  and  customer  specific  env  (in  Openstack  Swi^)

Page 25: Percona presentation v2

Benefits      

Improve  Efficiency    

•  Nothing  is  local,  everything  is  shared •  Control  access  rights •  Contribute  model •  No  need  for  a  full  release  team •  Factorize  hardware  needs

Reproducible  Env    

•  Dynamically  provision  test  environments  when  needed  by  Jenkins

•  Allow  developers  to  generate  a  test  environment  for  debugging  at  will

•  Store  validated  releases  and  customer  specific  environments  (in  Swi^)

Page 26: Percona presentation v2

What  we  are  describing  is  Devops…    

•  Validate  code  changes  and  mulRple  deployments  environments  at  once  •  Accommodate  more  devs  working  on  the  same  projects  •  Limit  wait  Rmes    ⇒  Industrializes  development  and...  

Page 27: Percona presentation v2

Conclusion    

•  Openstack  is  not  easy.  •  It  is  ok  at  small  scale  in  a  single  DC  mode.    •  eNovance    •  has  a  product  to  address  Single  DC  HA  mode  deployment  •  is  applying  Openstack  CD/CI  to  Openstack  deployment  itself  •  is  working  on  the  MulR  DC  mode  

Page 28: Percona presentation v2

Ques>ons?  

 

   

Sandro  Mazzioaa

[email protected]

@smazzioaa