San Antonio DevOps: Fluentd

download San Antonio DevOps: Fluentd

of 25

  • date post

    16-Jul-2015
  • Category

    Technology

  • view

    71
  • download

    0

Embed Size (px)

Transcript of San Antonio DevOps: Fluentd

  • Hart Hoover | @hhoover Josh OBrien | @OBrienCommaJosh

    @SADevOps

  • fluentd

  • Log all the thingsto all the things!

  • in JSON!

  • input&{&&&file&{&&&&&path&=>&"/var/log/http.log"&&&}&}&filter&{&&&grok&{&&&&&match&=>&[&"message",&"%{IP:client}&%{WORD:method}&%{URIPATHPARAM:request}&%{NUMBER:bytes}&%{NUMBER:duration}"&]&&&}&}

  • Fluentd Events

    Time Tag

    Record

    passed from source or

    adding a parsed time

    message routing in fluentd

    JSON

  • #&receive&events&via&HTTP&&&&type&http&&port&8888&&

  • #&read&logs&from&a&file&&&&type&tailpath&/var/log/httpd.log&format&apachetag&apache.access&&

  • #&DOCKER&OMG&&&&type&tail&&&format&json&&&path&/var/lib/docker/containers/ID/IDYjson.log&&&pos_file&/var/lib/docker/containers/ID/IDYjson.log.pos&&&tag&docker.container&&&rotate_wait&5&&&read_from_head&true&

  • #&save&alerts&to&a&file&&&&type&file&&path&/var/log/fluent/alerts&&

  • #&save&access&logs&to&MongoDB&&&&type&mongo&database&apache&collection&log&&&

  • #&Post&to&IRC&&&&type&irc&&&host&localhost&&&port&6667&&&channel&fluentd&&&nick&fluentd&&&user&fluentd&&&real&fluentd&&&message&notice:&%s&[%s]&%s&&&out_keys&tag,time,message&&&time_key&time&&&time_format&%Y/%m/%d&%H:%M:%S&&&tag_key&tag&

  • #&forward&other&logs&to&servers&&&&type&forward&&&&&&&&&host&192.168.0.11&&&&&&weight&20&&&&&&&&&&&&&host&192.168.0.12&&&&&&weight&60&&&&

  • #&Send&logs&to&ElasticSearch&&&&type&elasticsearch&&&logstash_format&true&&&host&localhost&&&port&9200&&&index_name&fluentd&

  • Client LibrariesRuby

    Python Java PHP

    Node.JS Scala

  • https://github.com/treasure-data/chef-td-agent

    Install with Chef

    Vagrantfile & kitchen.yml included!

  • https://github.com/dmytro/fluentd-cookbook

    Possibly Better

    https://gist.github.com/hhoover/4fceb09148a73f45136a

  • @SADevOpshttp://meetup.com/SanAntonioDevOps