Consul: Microservice Enabling Microservices and Reactive Programming
-
Upload
rick-hightower -
Category
Technology
-
view
1.489 -
download
2
Transcript of Consul: Microservice Enabling Microservices and Reactive Programming
![Page 1: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/1.jpg)
CONSULConsul at a glance.
!The microservices architecture service discovery engine
!Enabling reactive programming by providing a reliable list of nodes participating in a cluster
![Page 2: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/2.jpg)
Consul provides a consistent view of services and configuration
![Page 3: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/3.jpg)
CONSUL
• Service Discovery
• Health
• Config
![Page 4: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/4.jpg)
Consul uses a microservices architecture to provide reliable service discovery and health monitoring
![Page 5: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/5.jpg)
CONSUL• Consul provides service discovery
• Consul provides a consistent view of services
• Consul provides a consistent view of configuration
• Consul uses a microservice interface to a replicated view of your topology and its configuration
• Consul can monitor and change services topology based on health
![Page 6: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/6.jpg)
FEATURES• scalable distributed health checks
• minimal dc to dc communication
• event system
• domain model for managing topology of datacenters, server nodes, and services running on server nodes along with their configuration
![Page 7: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/7.jpg)
CONSUL LIKE
• DNS server plus Consistent Key/Value Store + ZooKeeper + Nagios + … = Consul
• All the bits you need in a coherent model available to provide service discovery and replicated config
![Page 8: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/8.jpg)
UI
![Page 9: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/9.jpg)
SERVICES IN CATALOG• available via DNS
• available via REST interface
• Consul is a microservice architecture
• HTTP
• JSON
![Page 10: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/10.jpg)
AGENT• Long running daemon on every member of Consul
cluster
• Client or server mode
• Client runs on server hosting services
• All nodes run an agent
• Stays in sync, interface with REST and DHCP
![Page 11: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/11.jpg)
CLIENT
• Agent that forwards request to server
• Mostly stateless
• Client does LAN gossip (low traffic)
![Page 12: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/12.jpg)
SERVER• Server also agent with more tasks
• RAFT quorum, who is leader/master
• Maintain cluster state
• Handles WAN gossip to other datacenters
• Forwards queries to leader/master
• Forward queries to other datacenters
![Page 13: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/13.jpg)
DATACENTER
• Datacenter - obvious?
• EC2 availability zone
• Networking environment
• Private, low latency, and high bandwidth
• Fast between nodes, no hops, little routing, high speed
![Page 14: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/14.jpg)
CONSENSUS• Agreement on who is the leader
• Transactional finite state machine
• Replicated Log, FSM, Peer Set (who gets the replicated log), Quorum (majority of peers agree), Committed Entry, Leader
• End Result: Consistent view of configuration and live services
![Page 15: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/15.jpg)
GOSSIP
• Consul is built on top of Serf,
• Serf is a full gossip protocol
• Serf provides membership, failure detection, and event broadcast mechanisms
• Clustering of server nodes
![Page 16: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/16.jpg)
LAN, WAN, AND RPC• LAN Gossip -
• LAN gossip pool,
• all agent nodes (client and server) on same local network or datacenter
• WAN Gossip -
• WAN gossip pool
• Only agent servers
• Servers per datacenter (3 to 5 for redundancy per DC)
• WAN gossip is used to communicate over public internet or wide area network
• RPC - Remote Procedure Call. Request / response mechanism allowing a client to make a request of a server.
![Page 18: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/18.jpg)
AGENT RESPONSIBILITIES• Maintains its
• set of service
• check registrations
• health information
• Updates
• health checks
• local state
![Page 19: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/19.jpg)
CATALOG• Backs Consuls service discovery
• Catalog is the service catalog
• Aggregates information submitted by agents.
• Maintains topology view of cluster,:
• Services available,
• Nodes which run those services (Node = Client Agent),
• Health information,
• Services and checks in catalog have less information than agent view
• Catalog is maintained only by server nodes (server agents).
• Catalog is replicated via Consul Consensus (RAFT, Serf)
![Page 20: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/20.jpg)
AGENTS VS. CATALOG• Local agent state (agent client node) is different than catalog state (agent server
node)
• Local Agent notifies Catalog of changes. Updates are synced right away.
• Agents check to make sure its view of the world matches the catalog, and if not the catalog is updated
• Example: Registers a new service check with agent,
• Agent notifies catalog that this service check exists
• When a check is removed from the agent, it is removed from catalog
![Page 21: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/21.jpg)
AGENTS VERSUS CATALOG 2
• If Agents run health checks, and status changes, then update is sent to catalog
• Agent is the authority of that node, and the services that exist on that node
• Scope
• Agent = Server or Virtual Machine
• Catalog = single datacenter, local area network, EC2 availability zone
• Changes are also synchronized periodically every minute to ten minutes
![Page 22: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/22.jpg)
CONSUL COMMAND LINE
-server for server
mode
![Page 23: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/23.jpg)
STARTING UP A NEW SERVER
• Expect min amount of servers for bootstrap
$ consul agent -server -data-dir=“/opt/git/dc1" -bootstrap-expect 5
![Page 24: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/24.jpg)
HTTP / JSON API ENDPOINTS• kv - key value
• agent - api for dealing with agent
• catalog - dealing with datacenter catalog
• health - show health checks
• sessions, events, acl, status, etc.
![Page 25: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/25.jpg)
CONFIG FILES
• config files can be stored locally and used to bootstrap config of consul
![Page 26: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/26.jpg)
SERVICE DEFINITION
Default check is dead man switch or tll
![Page 27: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/27.jpg)
HEALTH CHECKS• Dead man switch,
• time to live, you have to dial in with status update every X
• HTTP ping every N time
• HTTP Code 200 - 299 = PASS
• HTTP Code 429 = WARN
• anything else = FAIL
• Run a script every N:
• process 0 = pass,
• process 1 = warn,
• anything else = FAIL
![Page 28: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/28.jpg)
SCRIPT CHECK
![Page 29: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/29.jpg)
HTTP CHECK
![Page 30: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/30.jpg)
DEAD MAN SWITCH CHECK
![Page 31: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/31.jpg)
RPC PROTOCOL
• All command line options available via TCP/MsgPack
• MsgPack = binary JSON more or less
![Page 32: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/32.jpg)
REGISTERING EXTERNAL SERVICES
External registry plus HTTP health check = integration of service that has no idea that Consul exists = Legacy integration of services = service discovery for legacy
services
![Page 33: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/33.jpg)
WHAT WE HAVE DONE WITH CONSUL
• wrote java microservices lib that uses consul for service discovery
• Blog: Using Consul from QBit, the Java, JSON, WebSocket and REST microservice library, for health monitoring, clustering and service discovery
• used consul to implement clustered, replicated event bus for qbit
• Create general purpose service discovery that uses it
• Used general purpose service discovery to build distributed/clustered stats service to implement things like client application id rate limiting at high-speeds
![Page 34: Consul: Microservice Enabling Microservices and Reactive Programming](https://reader030.fdocuments.us/reader030/viewer/2022020721/55a68dfb1a28abb97d8b4779/html5/thumbnails/34.jpg)
THANK YOUhttp://www.linkedin.com/in/
rickhigh/en