Monitora le performance della tua applicazione Python Flask con Elasticsearch e Kibana

PyCon Nove Florence20/04/2018

About me

MATTEO ZUCCON

@matteo_zuccon

zuccon.matteo@gmail.com

https://whiletrue.run

Solution Developer |

#PyconNove #PyconIT @pyconit

Agenda

• What is performance? Definition and meaning

• APM - Application Performance Management

• Open Source APM with and

• Monitor Flask Application

What is performance?

“total effectiveness of an application or computer system, including throughput ,

individual response time, and availability”

Something is successful in producing a desired result; success

What is performance? Monitoring

• Application Avg Response Time

• Application Load

• Monitoring CPU of servers

• Tracking and log errors

• Everything useful for you – depending from your use case and application

What is performance? Management and Monitoring

Performance matters!

NOT ONLY

monitoring

ALSO

management

APM - Application Performance Management

“In the fields of information technology and systems management, application performance management (APM) is

the monitoring and management of performance and availability of software applications.”

“the translation of IT metrics into business meaning”

Application performance managementor

application performance monitoring.

APM - Management and Monitoring

APM - Management and Monitoring

being more proactive

only being reactive when it comes to the performance

Application performance managementor

application performance monitoring.

APM - It seems easy

…monitoring and management of performance

APM – Components – 1 of 2

• Performance of individual web requests or transactions

• Usage and performance of all application dependencies like databases, web services, caching, etc

• Detailed transaction traces down to specific lines of code

• Code level performance profiling

• Basic server metrics like CPU, memory, etc

APM – Components – 2 of 2

• Evaluate performance management criteria

• Application aware and focused – understand application context

• Custom applications metrics created by the dev team or business

• Application log data and errors

• Real time monitoring

APM – Tools

• New Relic

• Glimpse

• Miniprofiler

• XRebel

• Stackify Prefix

• App Dynamics

• Scout Devtrace

• Rack trace

• Zend Z-Ray

APM – Tools – Cloud Providers

• X-Ray e Pinpoint

• Application Insights

• APM

APM – Literature

APM – Practices

APM – Not that easy- Where do I start?

A lot of components, tools, books, practices….

Elasticsearch and Kibana

Elasticsearch is a distributed, RESTful search and analytics engine, heart of the Elastic Stack, it centrally stores your data.

Kibana lets you visualize your Elasticsearch data and navigate the Elastic Stack

Open Source APM - Elastic

Already housing logs and system metrics in Elasticsearch?

Expand to application metrics with Elastic APM.

Four lines of code lets you see a bigger picture to quickly fix issues and feel good about the code you push.

(basic subscription – Free license)

Elasticsearch and doubling down on “open”: https://changelog.com/podcast/292

Elastic APM - Components

• Elasticsearch

• APM Server

• APM Agents

• Kibana APM UI

APM – Components - Elasticsearch

•Elasticsearch

• APM Server

• APM Agents

• Kibana APM UI

Centrally stores the application metrics

APM – Components – APM Server

• Elasticsearch

•APM Server

• APM Agents

• Kibana APM UI

Open source application written in Go which runs on your servers. It receives data from agents periodically.

The API is a simple JSON based HTTP API.

APM Server builds Elasticsearch documents from the data received from agents.

APM – Components – APM Agents

• Elasticsearch

• APM Server

•APM Agents

• Kibana APM UI

Hook into your application and start collecting performance metrics and errors automatically.

Collected data is buffered for a short period and sent on to APM Server.

APM – Components - Kibana

• Elasticsearch

• APM Server

• APM Agents

•Kibana APM UI dashboard to visualizethe APM metrics

APM – Components Schema

APM - Agents

• Python*

• NodeJS

• Ruby

• Java

• Go

• JavaScript

Version 1.8 – 2.0

Version 0.10+

* no support async frameworks like Twisted or Tornado

Elastic APM - Pros

• Open Source, free and easy to use

• APM Agents

• Fully integrated with Elasticsearch

• Built in Kibana Dashboard

APM – Python Flask - Configuration

pip install elastic-apm[flask]

For apm-server 6.2+, make sure you use version 2.0 or higher of elastic-apm

APM – Python Flask - Routes

Automatically track transactionsand capture uncaught exceptions within Flask.

APM – Python Flask - Dashboard

APM – Python Flask - Dashboard

List of requests with details

APM – Python Flask - Dashboard

Requests impact

APM – Python Flask - Dashboard

Response Times

APM – Python Flask - Dashboard

Request per minute by HTTP Code

APM – Elasticsearch Document

APM – Python Flask – Arbitrary Exception

Send additional events by raising an exception or logging a generic message

APM –Extra Data and ignore specific routes

Send additional data Ignore specific route

APM – Instrumenting custom code

Decorator or context manager Transaction sample

APM – Instrumenting custom code

Decorator or context manager Transaction sample

Transaction Sample Timeline

APM – Instrumenting custom code

Response Time Distribution

APM – Sanitizing data

Add processors

Custom Processor

APM –Tuning and Overhead considerations

• CPU time

• Memory

• Bandwidth use

• Elasticsearch storage

• Transaction Sample Rate:randomly sample only a subset of transactions

• Transaction Queue:reduce queue size and flush time

• Spans per transaction:transactions should have well below 100 spans

APM – Elastic - Python

• Agents ready to go (Flask and Django)

• Few lines of code

• Custom exceptions and messages

• Transactions and requests

• Exclude route and sanitize data

• APM Server public API

APM – Elastic - Python

Thank youQuestions?

@matteo_zuccon

zuccon.matteo@gmail.com

https://whiletrue.run

#PyconNove #PyconIT @pyconit

https://bit.ly/2JYGZtZ