Sumo Logic Confidential

Logging & Docker

Christian Beedgen, CTO & Co-Founder, Sumo Logic

Seattle Docker Meetup, October 13, 2015

Sumo Logic Confidential

$ whoami• Co-Founder & CTO, Sumo Logic

Cloud-based Machine Data Analytics ServiceApplications, Operations, Security

• Chief Architect, ArcSightMajor SIEM player in the enterprise spaceLog Management for security and compliance

Sumo Logic Confidential

December 2014, New York City

Sumo Logic Confidential

December 2014, New York City

Sumo Logic Confidential

December 2014, New York City

Sumo Logic Confidential

December 2014, New York City

Sumo Logic Confidential

December 2014, New York City

Sumo Logic Confidential

December 2014, New York City

Sumo Logic Confidential

December 2014, New York City

Sumo Logic Confidential

December 2014, New York City

Sumo Logic Confidential

December 2014, New York City

Sumo Logic Confidential

December 2014, New York City

Sumo Logic Confidential

December 2014, New York City

Sumo Logic Confidential

December 2014, New York City

Sumo Logic Confidential

December 2014, New York City

Sumo Logic Confidential

December 2014, New York City

Sumo Logic Confidential

December 2014, New York Cityhttp://www.slideshare.net/

raychaser/6-million-ways-to-log-in-docker-nyc-docker-meetup-12172014

Sumo Logic Confidential

Season 2Where Are We In Late 2015?

Sumo Logic Confidential

Basics• Logging in Docker as per 12factor.net

Sumo Logic Confidential

Basics• Logging in Docker as per 12factor.net

Sumo Logic Confidential

Basics• Logging in Docker as per 12factor.net

• Also, one process per container, plz!

Sumo Logic Confidential

Pre-Docker 1.6

• Docker simply collects stdout and stderr from a container• Wrapped in a bit of JSON and stored on disk

Sumo Logic Confidential

Pre-Docker 1.6

• Docker simply collects stdout and stderr from a container• Wrapped in a bit of JSON and stored on disk

Sumo Logic Confidential

Pre-Docker 1.6

• Docker simply collects stdout and stderr from a container• Wrapped in a bit of JSON and stored on disk

Sumo Logic Confidential

Pre-Docker 1.6

• Docker simply collects stdout and stderr from a container• Wrapped in a bit of JSON and stored on disk

Sumo Logic Confidential

Pre-Docker 1.6

• Early hardcore crowd would just collect /var/lib/docker/containers/**

• And then of course there’s the UX: docker logs• docker logs is using a daemon API for getting the logs• This leads to logspout – attach to API, forward to Syslog• https://github.com/gliderlabs/logspout

Sumo Logic Confidential

Docker 1.6 Introduced Log Drivers

• Hallelujah• Initially supports json-file, syslog, null• json-file – default, this is the old mechanism

– Continues to this day to be required for API access and docker logs• docker run -–log-driver syslog …

– Sends to local Syslog, no more writing to disk• docker run –-log-driver null

– STFU, basically

Sumo Logic Confidential

Docker 1.7 Introduces --log-opt

• Now we can pass parameters to the log drivers!• docker run \

--log-driver syslog \

--log-opt syslog-address=(udp|tcp)://… \

--log-opt syslog-facility=(kern|daemon|user|local0|…) \

--log-opt syslog-tag=“myapp”

• Forward directly to local Syslog aggregator, or to a cloud-based logging service

• Docker 1.7 also added support to log to journald

Sumo Logic Confidential

Docker 1.8, 1.9 - Even More Log Drivers

• Fluentd

• GELF

• AWS

Sumo Logic Confidential

Also in Docker 1.8 – Options For json-file

• json-file still the default, still required for docker logs and /logs API• Long standing problem – will eventually fill up your disk• Folks have been using logrotate hacks…• Now, json-file log driver can be configured:

• Basically, keep up to max-file files, roll current at max-size

Sumo Logic Confidential

Coming In Docker 1.9 – Log Tags• Many containers can share a single aggregator downstream the log driver• All this muxing creates a problem – which log from which container?• Basically, there is a loss of meta data• Log Tags enable to use of container meta data as part of each message• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"

• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:

Hello from Docker.

Sumo Logic Confidential

Coming In Docker 1.9 – Log Tags• Many containers can share a single aggregator downstream the log driver• All this muxing creates a problem – which log from which container?• Basically, there is a loss of meta data• Log Tags enable to use of container meta data as part of each message• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"

• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:

Hello from Docker.

Sumo Logic Confidential

Coming In Docker 1.9 – Log Tags• Many containers can share a single aggregator downstream the log driver• All this muxing creates a problem – which log from which container?• Basically, there is a loss of meta data• Log Tags enable to use of container meta data as part of each message• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"

• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:

Hello from Docker.

Sumo Logic Confidential

Coming In Docker 1.9 – Log Tags• Many containers can share a single aggregator downstream the log driver• All this muxing creates a problem – which log from which container?• Basically, there is a loss of meta data• Log Tags enable to use of container meta data as part of each message• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"

• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:

Hello from Docker.

Sumo Logic Confidential

What Is Sumo Working On?

• We have containerized our collectors– https://github.com/SumoLogic/sumologic-collector-docker– docker run -d -p 514:514 -p 514:514/udp \

--name="sumo-logic-collector" \

sumologic/collector:latest-syslog \

[Access ID] [Access key] – https://www.sumologic.com/2015/09/09/update-on-logging-with-docker/

Sumo Logic Confidential

What Is Sumo Working On?

• We are working towards our vision of Comprehensive Monitoring– https://www.sumologic.com/2015/06/16/comprehensive-monitoring-for-docker-more-than

-just-logs/

• We have released an initial App for Docker at DockerCon 2015