Consul  -­‐  Service  discovery  and  others  By  Walter  Liu  2014/11/11  

Similar  services  •  Consul  •  SkyDNS  •  Others  •  Zookeeper  •  etcd  •  Serf  

What’s  Consul?  •  Service  Discovery  •  DNS  •  HTTP  (JSON)  

•  Health  Checking  (Service  Discovery  need  this)  •  Key/Value  Store  •  dynamic  configuraMon  •  feature  flagging  •  coordinaMon  •  leader  elecMon  •  (Very  like  the  funcMonality  of  Zookeeper  we  used)  

•  MulM  Datacenter  •  Network  parMMon  

•  If  one  DC  off,  no  impact  to  another  DC.  •  Data  duplicate  to  another  DC  automaMcally.  

User  Scenarios  in  Our  Infra  •  Simplify  config  (even  with  SaltStack)  •  Dynamic  server  cluster  •  Faster  failure  recovery  •  Dynamic  cluster  configuraMon  •  Launching  a  node  in  a  container  cloud.  (Future  maybe?)  

Scenario:  Simplify  con>ig  •  ENV  (from  hostname  or  specified  in  Salt)  •  $TYPE=“prod”  •  $GROUP=“gg”  •  $DATACENTER=“mydc1”  

•  Config  could  be  •  HOST_POSTFIX  =  $GROUP+$TYPE+$DATACENTER  =  

•  “prod.gg.mydc1.consul”  •  Config  in  Group  

•  “db.prod.gg.mydc.consul”  •  Config  for  a  datacenter  

•  “redis.prod.mydc1.consul”  

•  So,  every  config  for  prod/stg/dev  could  be  the  same.  •  Do  config  in  service  provider  instead  of  every  client.  •  Service  provider  could  automaMcally  register  if  supported.  

Scenario:  Dynamic  server  cluster  •  Background  •  A  server  cluster  without  load  balancer  (Cannot  apply  LB)  •  An  applicaMon  that  use  this  service  

•  Sequence  •  Query  Consul  will  return  a  list  of  servers.  •  When  a  server  failed,  Consul  will  aware  that  very  quickly  and  remove  it  from  the  list.  

•  An  applicaMon  that  use  this  cluster  will  be  interrupted  and  need  to  re-­‐query  Consul.  Consul  will  return  the  new  list  of  servers.  

Scenario:  Faster  failure  recovery  •  Background  •  A  master/slave  DB  without  automaMc  failover  

•  Sequence  •  The  master  DB  is  dead.  •  Nagios  found  it.  •  OPS  aware  it  and  log-­‐in  slave  DB.  •  OPS  restart  slave  DB  as  master  and  run  script  to  register  new  DB  •  All  applicaMons  that  use  this  DB  will  change  to  new  DB.  

Scenario:  Dynamic  cluster  con>iguration  •  If  a  service  cluster  needs  dynamic/automaMc  change  configuraMon  during  run-­‐Mme.  

Scenario:  Launching  a  node  in  a  container  cloud.  •  Assume  the  container  image  of  the  service  is  ready.  •  Specify  correct  hostname  of  the  new  container.  •  Launch  the  new  container.  •  The  new  container  start  the  new  service.  •  If  it  is  a  simple  service  provider,  it  registers  itself  to  Consul.  •  If  it  uses  other  services,  it  queries  these  services  from  Consul.  

•  If  the  configuraMon  is  all  the  same,  the  new  node  is  ready.  

OTHERS  

Use  pip  +  virtualenv  •  Why,  •  Reduce  package  conflict.  (Use  egg  is  not  very  friendly.)  •  Some  Python  packages  doesn’t  have  RPM.  

•  Install  using  requirement.txt  (pip  install  –r  requirement.txt)  •  (?)  private  pip  repository  •  Below  for  dev  •  Use  virtualenvwrapper  to  be  more  convenient.  •  How  to  produce  requirement  more  easily?  

•  pip  freeze  >  stable-­‐req.txt  •  Pip  freeze  –r  dev-­‐req.txt  >  stable-­‐req.txt  #  Keep  old  comments  

Example:  requirement.txt  •  Beaker==1.3.1  •  Chameleon==2.11  •  DecoratorTools==1.7  •  DistuMls2==1.0a3  •  Django==1.5.1  •  M2Crypto==0.20.2  •  Magic-­‐file-­‐extensions==0.1  •  Mako==0.8.1  •  MarkupSafe==0.9.2  •  MySQL-­‐python==1.2.3c1  •  PasteDeploy==1.5.0  •  PyXML==0.8.4  •  Pygments==1.1.1  •  SOAPpy==0.11.6  •  SSSDConfig==1.11.6  •  Scrapy==0.18.4    

Store  SHOULD-­‐BE-­‐CHANGED  con>ig  in  environment  var  •  Why,  •  Leak  secrets  in  version  control  system.  •  Check-­‐in  always  config  to  version  control  system  accidentally.  

•  Example,  •  DB  username/password,  AWS  API  Keys  •  Secret-­‐token  (web  framework  like  Django)  •  URI  resource  string,  Used  service  host,  port  …..  

Explicitly  store  prod/stg/dev  settings  in  version  control  •  Why,  •  Check-­‐in  always  config  to  version  control  system  accidentally.  •  All  sevngs  in  all  environment  are  visible  to  everyone.  •  For  example,    

•  DEBUG=True  in  Django  •  Log  level  

•  How  (Django  for  example)  •  sevngs/  

•  prod.py  •  staging.py  •  dev.py  

•  ENV[“DJANGO_SETTINGS_MODULE”]  =  “sevngs.prod”  •  “sevngs.prod”  could  automaMcally  come  from  hostname.  •  If  this  env  variable  is  not  set,  service  will  print  log  and  exit.  

Consul  vs.  SkyDNS  •  Pros  •  Beyer  health  check  (SkyDNS  uses  heartbeat  +  TTL)  •  MulMple  datacenter  support  by  using  “region”.  •  A  beyer  web  view.  (hyp://demo.consul.io/ui/)  •  Beyer  RAFT  implementaMon  (unverified)  

•  Cons  •  lesser  contributors  (1/3)  •  More  immature  (0.41  now)  and  young.  

Problem  needs  attention  •  For  only  one  server  service  like  DB,  what  if  a  second  server  register  the  service  accidentally?  

•  Use  DNS+DNS  forwarding  approach  may  not  be  good  for  DNS  heavy  service.  (Like  Odin)  

Other  References  •  Service  registrator  for  Docker  •  hyps://github.com/progrium/registrator  

•  Set  env  by  Consul  •  hyps://github.com/hashicorp/envconsul