Cf logging&metics-presentation-meetup-06152015

13
TECHNOLOGY SOLUTIONS DELIVERED Cloud Foundry Logging & Metrics “Understanding the new Architecture and integrating with ELK”

Transcript of Cf logging&metics-presentation-meetup-06152015

TECHNOLOGY SOLUTIONS DELIVERED

TECHNOLOGY SOLUTIONS DELIVERED

Cloud Foundry Logging & Metrics“Understanding the new Architecture and integrating with ELK”

TECHNOLOGY SOLUTIONS DELIVERED

Agenda

• The new architecture• How To: Ship logs to ELK• Demo: Subscribing to the Firehose• Demo: Firehose-to-syslog• Demo: Sending Metrics to Doppler• Things to consider…• Q & A

TECHNOLOGY SOLUTIONS DELIVERED

The new architecture – 5 Basic Components

Sources

1

Cloud Foundry components and

apps that emit logs & metrics

Metron

2

Agents running on Cloud Foundry

VM’s that collect emitted

logs/metrics and forward them on to

Doppler

Doppler

3

Server(s) that collect (aggregate) logs & metric data from the Metron

agents

Traffic Controller

4

Handle subscription requests and stream logs/metrics to clients

Sinks

5

Clients – subscribe to receive

logs/metrics from Doppler.

Such as =>syslog drains, cf cli, …

TECHNOLOGY SOLUTIONS DELIVERED

Dropsonde EventsEvent Description

HttpStart Emitted when a client sends a request (or immediately when a server receives a request)

HttpStop Emitted when a client receives a response to its request (or when a server completes its handling and returns a response).

HttpStartStop An event representing the entire lifecycle of an http request

LogMessage An event containing a log entry (this is the normal log data available from cf cmd line or via the syslog daemon)

ValueMetric An event representing the value of a metric at a particular point in time

CounterEvent Represents the increment of a counter. It contains only the change in the value; it is the responsibility of downstream consumers to maintain the value of the counter.

Error Represents an error in the process that originated the event.

ContainerMetric An event representing resource usage (cpu/mem/disk) of an app in a container

TECHNOLOGY SOLUTIONS DELIVERED

How to send CF logs to ELK• Update your deployment manifest or find the appropriate template for your

release/deployment.• You need to populate the following in one of your templates or in your manifest.

properties: syslog_daemon_config: address: <<ip of your ELK Ingestor>> port: <<correct port of chosen transport. Usually 5514 for syslog 5515 for relp transport: <<syslog or relp>>

• You could put this in cf-properties.yml or create or use cf-syslog.yml

• Using bosh…target your deployment then perform a bosh deploy.• Open Kibana and see the wonder of CF logs steaming into your ELK

environment!

TECHNOLOGY SOLUTIONS DELIVERED

Demos

• Use NOAA client to subscribe for logs/metrics.

• Use Firehose-to-syslog to get metrics data from the Firehose into your ELK environment.

• Emit some new metrics and see them flow through to ELK.

TECHNOLOGY SOLUTIONS DELIVERED

Helpful links…• Main Loggregator Repository

• https://github.com/cloudfoundry/loggregator

• Main NOAA Client (GO) Repository• https://github.com/cloudfoundry/noaa

• Dropsonde• https://github.com/cloudfoundry/dropsonde• https://github.com/cloudfoundry/sonde-go

• Firehose-to-Syslog• https://github.com/cloudfoundry-community/firehose-to-syslog

• Cloud Credo – Great background and how to(s) for Firehose and ELK/Graphite integration• http://www.cloudcredo.com/cloud-foundry-logging-and-monitoring/• http://www.cloudcredo.com/the-basics/• http://www.cloudcredo.com/cloud-foundry-firehose-and-friends/• http://www.cloudcredo.com/how-to-integrate-elasticsearch-logstash-and-kibana-elk-with-cloud-foundry/• http://cloudcredo.com/how-to-integrate-graphite-with-cloud-foundry/

• ELK – bosh release(s)/deployments• https://github.com/cloudfoundry-community/logsearch-boshworkspace• https://github.com/logsearch/logsearch-boshrelease

• Stark & Wayne • https://blog.starkandwayne.com/2014/11/19/running-logsearch-on-bosh-lite-in-10-minutes/

TECHNOLOGY SOLUTIONS DELIVERED

ECS Team

TECHNOLOGY SOLUTIONS DELIVERED

About Us

Founded in 1999

Over 130 Consultants

HQ in Greenwood Village, CO

Delivering Technology Solutions

Solving Strategic Business

Challenges

TECHNOLOGY SOLUTIONS DELIVERED

Our Solutions

Cloud-NativeDevelopment

Collaboration & Portals

Custom Software Development

Enterprise Architecture

Enterprise Mobility Integration & API Management

Interactive Design UI/UX

Project Management & Governance

TECHNOLOGY SOLUTIONS DELIVERED

Our Partners

TECHNOLOGY SOLUTIONS DELIVERED

Our Clients

TECHNOLOGY SOLUTIONS DELIVERED

Stage Focus

Project Management & Governance

Ensure long-term project success through governance.

Discover Validate business challenges and develop a plan to address these challenges with a new solution.

Design Map the business challenges defined in discover to a technology solution.

Build Convert the design into a working, deployable solution. At the end of build, the solution is ready for the organization to use.

Launch Deploy the developed and tested solution.

Adopt Achieve client success and realize solution business value.

ECS Team ensures project success through its process of discover, design, build, launch and adopt,while offering project management & governance at the same time.

ECS Team Solution Framework