Data Driven Infrastructure

Friday, August 23, 13

DataDrivenInfrastructure

http://git.io/UmUdKAFriday, August 23, 13

Let meTell youa story

What is a

I don’t have to think about IT

With the Cloud...

aaSPlatformSoftware

InfrastructureFriday, August 23, 13

Where is our cloud?

UtopiaFriday, August 23, 13

James Fryman

Automations Junkie

Are youCrazy?

Goals?Data?What

Machine

Parsable

There is SystemOne

Feedback

PuppetDBProvisioning

Configuring

Destroying

gPanel

Partsof the

System

provisioner

Home GrownFriday, August 23, 13

Home Grown

controller

Lots O’ Models

file { '/etc/facter/facts.d/quagga_manage_service': ensure => present, replace => false,} if $::quagga_manage_service { $service_state[ensure] = running $service_state[enable] = true} else { $service_state[ensure] = undef $service_state[enable] = undef} service { 'quagga': ensure => $service_state[ensure], enable => $service_state[enable],}

orchestrator

ChatOps

Feedbackof the

System

Metrics&Monitoring

begin JSON.parse(RestClient.get(url)).each do |cache| data["#{cache['target']}"] = 0 count = 0 cache["datapoints"].each do |point| unless (point[0].nil?) data["#{cache['target']}"] += point[0] count += 1 end end data["#{cache['target']}"] /= count data["total"] += data["#{cache['target']}"] endrescue ZeroDivisionError => e puts "UNKNOWN metric not in graphite!" exit EXIT_UNKNOWNrescue => e puts e.message exit EXIT_CRITICALend

eventlog

eventalertlog

eventalertlog error

Must self-correct

Feedback

PuppetDBProvisioning

Configuring

Destroying

gPanel

Deployable using text files

Modularity

haproxy::proxy { $es_proxy_name: proxy => 'listen', mode => 'http', ip => $::ipaddress_lo, port => '9200', config => { balance => 'roundrobin', }, }

haproxy::proxy::member { $es_proxy_name: hostname => $::ec2_local_ipv4, port => '9200', param => [ 'weight 1', 'maxconn 1000', 'check', ], }

collectd::plugin { [ 'cpu', 'load', 'memory', 'swap', 'irq', 'exec', 'entropy', ]: }

collectd::plugin { [ 'df', 'interface', 'protocols', 'disk', ]: config => true }

Level 4:Templates

Level 5:Data Driven

SelfAuthoritative

def self.collect_resources(resource) query = [ "and", ["=", "type", "File"], ["=", "exported", true], ["=", "tag", "nagios::object::#{resource}"], ["=", ["node", "active"], true] ] self.search('/v2/resources', query.to_json) end

# Seed the initial file with false, and enable # on next run with gh-nagios enable_pager <hostname>

file { '/etc/facter/facts.d/enable_pager.txt': ensure => file, owner => 'root', group => 'root', mode => '0644', content => 'enable_pager=false', replace => false,}

@@file { "/etc/nagios/objects/${type}.d/${filename}.cfg": content => template('nagios/etc/nagios/object.erb'), mode => '0444', group => 'nagios', notify => Service['nagios'], }

Refactoring

least to most

specificFriday, August 23, 13

frymanet.com

mysqlnginx rails

rubycommon admin

package repos

graduate to

paramsFriday, August 23, 13

class ntp::params { $defaults = { package => { version => ‘latest’, }, config => { servers => [‘pool.ntp.org’], }, }}

externalize

class ntp::params { $defaults = { package => { version => hiera(‘ntp_package_version’), }, config => { servers => hiera(‘ntp_servers’), }, }}

CloudFormation

"Resources": { "RendererServerGroup": { "Type": "AWS::AutoScaling::AutoScalingGroup", "Properties": { "AvailabilityZones": { "Fn::GetAZs": "" }, "LaunchConfigurationName": { "Ref": "LaunchConfig" }, "MinSize": "2", "MaxSize": "16", "Tags": [ { "Key": "Environment", "Value": "Production", "PropagateAtLaunch": "true" }, { "Key": "Role", "Value": "renderer", "PropagateAtLaunch": "true" } ] } },Friday, August 23, 13

Autoloading

# autoloader.ppclass nagios::autoload_helpers { $helpers = get_nagios_helpers()

case $::puppetversion { /^3/: { include $helpers } default: { nagios::autoload_helpers::import_shim { $helpers: } } }}

# autoloader.ppclass nagios::autoload_helpers { $helpers = get_nagios_helpers()

case $::puppetversion { /^3/: { include $helpers } default: { nagios::autoload_helpers::import_shim { $helpers: } } }}

module Puppet::Parser::Functions newfunction(:get_nagios_helpers, :type => :rvalue, :doc => "Grab all modules that have nagios helpers for import") do

module_path = File.expand_path('..', Puppet::Module.find('nagios',compiler.environment.to_s).path)

helpers = Dir["#{module_path}/**/nagios/helpers.pp"].map do |d| "#{d.split('/')[-4]}::nagios::helpers" done

helpers endend

Modeling

Be Dynamic

<%- if @comment -%>### <%= @comment %><%- end -%><%= @type %> <% if @label %><%= @label %><% end %> {<%- @config.sort.reverse.each do |key,value| -%> <%- if value.class == Array -%> <%- value.each do |element| -%> <%= key %>(<%= element %>); <%- end -%> <%- else -%> <%= key %>(<%= value %>); <%- end -%><%- end -%>};

Fencing Resources

# Seed the initial file with false, and enable # on next run with gh-nagios enable_pager <hostname>

file { '/etc/facter/facts.d/enable_pager.txt': ensure => file, owner => 'root', group => 'root', mode => '0644', content => 'enable_pager=false', replace => false,}

if $::enable_pager { @@file { "/etc/nagios/objects/${type}.d/${filename}.cfg": content => template('nagios/etc/nagios/object.erb'), mode => '0444', group => 'nagios', notify => Service['nagios'], }}

Buy it?

It’s About

Missing?What’s

Language

Predictive Analysis

Coming

Goals?Data?What

SystemsThinking

There is SystemOne

Machine

ParsableMust be aKnownQuantity

SelfAuthoritative

Must self-correctFriday, August 23, 13

jfryman

fryman@github.com

jfryman

fryman@github.com

what’s next?

Data Driven Infrastructure

Technology

Transcript of Data Driven Infrastructure

Building a Real-Time Event-Driven Enterprise Infrastructure · Agenda • PI for Enterprise Infrastructure • Utility Industry Use Cases – Operational Data – Non-Operational

Test-Driven Development of Infrastructure Code

Standards Driven Healthcare Information Integration Infrastructure

IT Strategy Board · 5/12/2014 · Network Virtualization and Security Implications ... Data science is driven more by intellectual infrastructure (human capital) and software infrastructure

Regulating Cross- Border Data Flows in a Data-Driven World · • National Security Concerns (e.g. cyber-attacks on critical infrastructure) ... • Ensuring data privacy and security

Zero to Test Driven Infrastructure

Data Driven, Risk-Based Infrastructure Planning€¦ · approach using risk -based, data-driven metrics to prioritize investments in order to enable the mission. Mission Dependency

Leveraging Data to Drive Innovation - Tower Hamlets...Ben Unsworth Data Solutions Architect • Ex local government ... Data-driven storytelling High-performance API infrastructure

ContainerCon - Test Driven Infrastructure

Databases & Data Infrastructure › files › 2018 › 07 › nsf-2013... · From Databases to Data Infrastructure Data-driven science creates new requirements for data: Data need

Enabling customer-centric, data-driven marketing and sales · Enabling customer-centric, data-driven marketing and sales “ Our data infrastructure is key to integrating and accessing

Test driven infrastructure avec Docker

Behaviour driven infrastructure

A GIS Data-Driven Process for Identifying Rail ... · - 2 - A GIS Data-Driven Process for Identifying Rail Infrastructure Improvements ABSTRACT The identification of infrastructure

Data-Driven Library Infrastructure

Data-Driven Innovations in the Public Cloud for …...a data-driven infrastructure that improved time to market. “We view ‘data-driven’ as saving lives—because we are making

Data Effectiveness: How to build a Data Driven and Reporting infrastructure

Orchestrate Event-Driven Infrastructure with SaltStack

Event driven infrastructure

Data-driven Transport Infrastructure Maintenance