Node collaboration - sharing information between your systems

Node collaboration

Michael Richardson @m_richo

Sharing information between your systems

Agenda1.Chef Search2.Puppet Exported

Resources3.PuppetDB query4.Confd + etcd

Chef Server

Stores many useful things including• Policies applied to nodes• Metadata that describes each

node (ohai)• And more…

Chef ServerThis data is searchable!

Chef ServerThis data is searchable!

But how?

Chef Server

1. ‘search’ method within recipe.2. Search subcommand of ‘knife’3. /search or /search/INDEX endpoints of

Chef Server API

Chef ServerSearch within recipe – simple.

Chef ServerSearch within recipe – simple.

It’s that simple

Chef ServerSearch within recipe – more complex.

Chef Server

Chef Search is fantastic for

• Searching for nodes with particular attributes

• Searching for nodes with particular profiles

• Using search results to configure services on other nodes.

Chef ServerCommon use cases

• My application talks to a database. What’s its FQDN/IP?• How many memcache nodes are there for my application?

What are there FQDN/IPs?• Where should I send my log files? What’s the FQDN of the

syslog server?• I want to provide a white-list of IP’s that can request

information from a node.• What are the IP addresses of the webservers behind my load

balancer?• My Jenkins slaves need to connect to a Jenkins master.

What’s it’s IP?• What are the SSH Host keys for particular hosts.

Exported Resources

Definition:An exported resource declaration specifies a

desired state for a resource, does not manage the resource on the target system, and publishes the

resource for use by other nodes. Any node (including the node that exported it) can then

collect the exported resource and manage its own copy of it. *

* https://docs.puppetlabs.com/puppet/latest/reference/lang_exported.html

Exported Resources

Think of it as

• Node(s) “publish” resources to the Puppet Master

• Other Node(s) consume those resources

* https://docs.puppetlabs.com/puppet/latest/reference/lang_exported.html

Exported Resources

Common Examples• Backend servers configured in a load

balancer• Monitoring Servers updated with monitoring

clients (nagios_host / nagios_service)• Distribute public keys (ssh and openssl)

PuppetDB

Definition:collects data generated by Puppet. It enables

advanced Puppet features like the inventory service and exported resources, and can be the foundation for other applications that use Puppet’s

data. *

PuppetDB

PuppetDB stores:• The most recent facts from every node• The most recent catalog for every

node• Optionally, 14 days of event reports for

every node• Exported Resources

PuppetDB

PuppetDB stores:• The most recent facts from every node• The most recent catalog for every

node• Optionally, 14 days of event reports for

every node• Exported Resources

Useful information

PuppetDB

PuppetDB REST APIV3 API includes the following endpoints:• Facts• Resources• Nodes• Fact-names,• Metrics

• Reports• Events• And more

PuppetDB

PuppetDB

But this talk is about Node Collaboration.How can I make use of this data?

Awesome

PuppetDB

checkoutPuppetdbquery module

by Erik Dalén

https://forge.puppetlabs.com/dalen/puppetdbqueryhttps://github.com/dalen/puppet-puppetdbquery

https://forge.puppetlabs.com/dalen/puppetdbquery

https://forge.puppetlabs.com/dalen/puppetdbquery

https://github.com/dalen/puppet-puppetdbquery

https://github.com/dalen/puppet-puppetdbquery

PuppetDB

Puppetdbquery module

• Command line tools• Puppet functions to query

PuppetDB• Hiera backend to return query

results from PuppetDB

PuppetDB

Puppetdbquery functions

• query_nodes• query_facts

PuppetDB

query fact

PuppetDB

note: Output is abbreviated

array

hash

PuppetDB

PuppetDBquery• Works great and very similar to

Chef Search (searching for specific hosts, and use their facts for configuring other hosts).

• If you use Roles+Profiles pattern, this is a fantastic fit for profiles.

Chef Search and PuppetDB are great!

Chef Search and PuppetDB are great!but what if…

• You want your infrastructure to react more quickly than your Puppet agent / Chef-client run intervals (are you even running the agents continuously).



• You appreciate “desired state” AND “running state” aren’t always the same thing.




• You don’t want to run a Puppet Master or Chef Server (masterless Puppet / Chef-solo).




• You don’t want to run a Puppet Master or Chef Server (masterless Puppet / Chef-solo).

• You don’t use Puppet or Chef.


Confd

• “Lightweight configuration management tool”• https://github.com/kelseyhightower/confd• Written in go• Manage local application configuration files using

templates and data from etcd or consul.

https://github.com/kelseyhightower/confd



etcd

• Highly-available key/value store for shared configuration and service discovery

• https://github.com/coreos/etcd• Written in go• Typical cluster size or 3-9 peers.• CAP theorem

https://github.com/coreos/etcd

https://github.com/coreos/etcd

Confd + etcd

etcd1

Distributed etcd cluster

etcd2 etcd3 etcd4 etcd5

Hosts running applications and services configured with confd

Confd + etcd

• Hosts/Services that wish to share information post data to etcd.

• Confd is configured on other hosts to use this data to manage local configuration files and services.

Confd + etcd

Example1. App server runs Confd which is polling the following keys.

1. /myapp/db_ip2. /myapp/db_port3. /myapp/db_adapter4. /myapp/db_name

2. Once up and running DB Server, writes details to etcd cluster.3. Confd reads key/values. Uses template file to write new

application configuration file and reloads application.

etcd1

Distributed etcd cluster

etcd2 etcd3 etcd4 etcd5

App server DB server

21

Confd + etcd

Demo

A “working” demo is worth a thousand powerpoint slides…

Summary

• Share information between your servers.

• Let them discover each other.• Let your nodes collaborate with each

other.

Thank you

Node collaboration - sharing information between your systems

Technology

Transcript of Node collaboration - sharing information between your systems