Docker in production service discovery with consul - road to opscon 2015
Consul - service discovery and others
Click here to load reader
-
Upload
walter-liu -
Category
Technology
-
view
539 -
download
2
Transcript of Consul - service discovery and others
Consul -‐ Service discovery and others By Walter Liu 2014/11/11
Similar services • Consul • SkyDNS • Others • Zookeeper • etcd • Serf
What’s Consul? • Service Discovery • DNS • HTTP (JSON)
• Health Checking (Service Discovery need this) • Key/Value Store • dynamic configuraMon • feature flagging • coordinaMon • leader elecMon • (Very like the funcMonality of Zookeeper we used)
• MulM Datacenter • Network parMMon
• If one DC off, no impact to another DC. • Data duplicate to another DC automaMcally.
User Scenarios in Our Infra • Simplify config (even with SaltStack) • Dynamic server cluster • Faster failure recovery • Dynamic cluster configuraMon • Launching a node in a container cloud. (Future maybe?)
Scenario: Simplify con>ig • ENV (from hostname or specified in Salt) • $TYPE=“prod” • $GROUP=“gg” • $DATACENTER=“mydc1”
• Config could be • HOST_POSTFIX = $GROUP+$TYPE+$DATACENTER =
• “prod.gg.mydc1.consul” • Config in Group
• “db.prod.gg.mydc.consul” • Config for a datacenter
• “redis.prod.mydc1.consul”
• So, every config for prod/stg/dev could be the same. • Do config in service provider instead of every client. • Service provider could automaMcally register if supported.
Scenario: Dynamic server cluster • Background • A server cluster without load balancer (Cannot apply LB) • An applicaMon that use this service
• Sequence • Query Consul will return a list of servers. • When a server failed, Consul will aware that very quickly and remove it from the list.
• An applicaMon that use this cluster will be interrupted and need to re-‐query Consul. Consul will return the new list of servers.
Scenario: Faster failure recovery • Background • A master/slave DB without automaMc failover
• Sequence • The master DB is dead. • Nagios found it. • OPS aware it and log-‐in slave DB. • OPS restart slave DB as master and run script to register new DB • All applicaMons that use this DB will change to new DB.
Scenario: Dynamic cluster con>iguration • If a service cluster needs dynamic/automaMc change configuraMon during run-‐Mme.
Scenario: Launching a node in a container cloud. • Assume the container image of the service is ready. • Specify correct hostname of the new container. • Launch the new container. • The new container start the new service. • If it is a simple service provider, it registers itself to Consul. • If it uses other services, it queries these services from Consul.
• If the configuraMon is all the same, the new node is ready.
OTHERS
Use pip + virtualenv • Why, • Reduce package conflict. (Use egg is not very friendly.) • Some Python packages doesn’t have RPM.
• Install using requirement.txt (pip install –r requirement.txt) • (?) private pip repository • Below for dev • Use virtualenvwrapper to be more convenient. • How to produce requirement more easily?
• pip freeze > stable-‐req.txt • Pip freeze –r dev-‐req.txt > stable-‐req.txt # Keep old comments
Example: requirement.txt • Beaker==1.3.1 • Chameleon==2.11 • DecoratorTools==1.7 • DistuMls2==1.0a3 • Django==1.5.1 • M2Crypto==0.20.2 • Magic-‐file-‐extensions==0.1 • Mako==0.8.1 • MarkupSafe==0.9.2 • MySQL-‐python==1.2.3c1 • PasteDeploy==1.5.0 • PyXML==0.8.4 • Pygments==1.1.1 • SOAPpy==0.11.6 • SSSDConfig==1.11.6 • Scrapy==0.18.4
Store SHOULD-‐BE-‐CHANGED con>ig in environment var • Why, • Leak secrets in version control system. • Check-‐in always config to version control system accidentally.
• Example, • DB username/password, AWS API Keys • Secret-‐token (web framework like Django) • URI resource string, Used service host, port …..
Explicitly store prod/stg/dev settings in version control • Why, • Check-‐in always config to version control system accidentally. • All sevngs in all environment are visible to everyone. • For example,
• DEBUG=True in Django • Log level
• How (Django for example) • sevngs/
• prod.py • staging.py • dev.py
• ENV[“DJANGO_SETTINGS_MODULE”] = “sevngs.prod” • “sevngs.prod” could automaMcally come from hostname. • If this env variable is not set, service will print log and exit.
Consul vs. SkyDNS • Pros • Beyer health check (SkyDNS uses heartbeat + TTL) • MulMple datacenter support by using “region”. • A beyer web view. (hyp://demo.consul.io/ui/) • Beyer RAFT implementaMon (unverified)
• Cons • lesser contributors (1/3) • More immature (0.41 now) and young.
Problem needs attention • For only one server service like DB, what if a second server register the service accidentally?
• Use DNS+DNS forwarding approach may not be good for DNS heavy service. (Like Odin)
Other References • Service registrator for Docker • hyps://github.com/progrium/registrator
• Set env by Consul • hyps://github.com/hashicorp/envconsul