HawkularSuccessor of the JBoss ON (RHQ)

JBoss User Group Koreatedwon

2017-11-29

Hawkular● Open source monitoring project● Next generation monitoring project of JBoss ON (RHQ since 2006)

○ Hawkular started end of 2014

● Apache License v2● Provide REST services● Providing a generic solutions to common problems.

○ From collecting rain sensors data and send an SMS on rain, ○ Monitor docker containers ○ Application Performance Monitoring,

● Adopted to the Middleware management solution in the ManageIQ project

For who ?● IoT entusiast who needs to collect metrics and possibly need to trigger alerts

● Operators who are looking for a solution to store metrics from statsD, collectd,

syslog…

● Developers of solutions who need long term timeseries database storage

● Users of ManageIQ who are looking for Middleware management

● Users of Kubernetes/Heapster who wants to store docker container metrics in a

long term timeseries database storage

Hawkular Projects● Hawkular Services & Alerts

○ flagship base project○ based on WildFly○ provides services

■ store metrics, ■ alert on metrics, ■ keep a graph view of an inventory

● Hawkular Metrics○ provide TimeSeries Database (TSDB) services○ Time Series Metrics Engine based on Cassandra for scalability○ offers a REST API to store and retrieve metrics.

● Hawkular APM○ Application Performance Management○ visualize which layer of an application time is spent○ deprecated in favor of CNCF's Jaeger

Red Hat Middleware Management with CloudForms● ManageIQ project● Middleware Manager for Red Hat Cloudforms Technology Preview

○ https://access.redhat.com/announcements/2820611

● Provide centralized management of all the JBoss middleware products from within the CloudForms user interface.

● Red Hat Middleware Management 7.0.TechPreview○ https://access.redhat.com/documentation/en-us/red_hat_middleware_management/7.0.techpr

eview/html-single/installing_red_hat_middleware_management_with_cloudforms/

Hawkular Services Quick Start

Hawkular Services Quick Start● Let’s run Hawkular

○ http://www.hawkular.org/hawkular-services/docs/quickstart-guide/

● Run Cassandra○ docker run --name hawkular-cassandra -e CASSANDRA_START_RPC=true -d

cassandra:3.0.9

● Run Hawkular Services○ docker run --name hawkular-services --link=hawkular-cassandra -e

CASSANDRA_NODES=hawkular-cassandra -p 8080:8080 hawkular/hawkular-services○ docker exec -it hawkular-services bash○ http://localhost:8080/

Hawkular Services Quick Start● STEP 1: ADD METRICS● STEP 2: READ THE METRICS● STEP 3: ADD ALERTING● STEP 4: TEST THE ALERT TRIGGER● Metrics REST API

○ http://www.hawkular.org/hawkular-metrics/docs/user-guide/ ○ http://www.hawkular.org//docs/rest/rest-metrics.html

Hawkular Services Quick Start● STEP 1: ADD METRICS

○ metrics_day_1.json => weather data collected every 30 min for the whole day of July 15th 2016.

○ curl -v -u jdoe:password -X POST http://localhost:8080/hawkular/metrics/gauges/temperature/raw -d @metrics_day_1.json -H "Content-Type: application/json" -H "Hawkular-Tenant: myTenant"

Hawkular Services Quick Start - Read Metrics● STEP 2: READ THE METRICS● from when to when we want the values, and ordered by time

○ curl -v -u jdoe:password -X GET "http://localhost:8080/hawkular/metrics/gauges/temperature/raw?start=1468578600000&end=1468594800001&order=ASC" -H "Content-Type: application/json" -H "Hawkular-Tenant: myTenant"

Hawkular Services Quick Start - Read Metrics

Hawkular Services Quick Start - Downsampled● STEP 2: READ THE METRICS● Want to know the minimum, maximum and average values for buckets that

would last 2 hours○ curl -v -u jdoe:password -X GET

"http://localhost:8080/hawkular/metrics/gauges/temperature/stats?bucketDuration=2h&start=1468533600000&end=1468618200001" -H "Content-Type: application/json" -H "Hawkular-Tenant: myTenant"

Hawkular Services Quick Start - Downsampled

Hawkular Services Quick Start● STEP 3: ADD ALERTING

○ curl -v -u jdoe:password -X POST http://localhost:8080/hawkular/alerts/import/all -d @trigger_definition.json -H "Content-Type: application/json" -H "Hawkular-Tenant: myTenant"

Hawkular Services Quick Start● STEP 4: TEST THE ALERT TRIGGER

○ clear console○ curl -v -u jdoe:password -X POST

http://localhost:8080/hawkular/metrics/gauges/temperature/raw -d @metrics_day_2.json -H "Content-Type: application/json" -H "Hawkular-Tenant: myTenant"

● In the logs of Hawkular Services, you should see 3 INFO messages showing that the conditions were met

Hawkular Alerts Quick Start with UI

Hawkular Alerts Quick Start with UI● You can

○ Add New Trigger○ Ack Alert○ Resolve Alert

● Run Hawkular Alerts UI○ docker stop hawkular-services○ docker stop hawkular-cassandra○ docker ps○ cd hawkular-alerts/examples/tutorial/lessons○ docker rm lessons_hawkularAlerts_1○ docker-compose up○ http://localhost:8080/hawkular/alerts/ui ○ ./tutorial-data.sh => generate random sample data every 5secs○ https://github.com/hawkular/hawkular-alerts/blob/master/examples/tutorial/lessons/lesson-04-ui.adoc

Hawkular Alert Lifecycle

https://www.slideshare.net/ponceballesteros/hawkular-alerting

Hawkular APM with Jaeger

Hawkular APM with Jaeger● Want to trace performance issue on distributed polyglot systems e.g. msa

○ Java○ Go○ Python○ Node.js○ C++

● Hawkular developed own contemporary distributed tracing systems ○ http://www.hawkular.org/blog/2016/09/19/hawkular-apm-on-msa.html○ for application performance management○ for distributed & micro-service applications

● e.g., Zipkin, Google Dapper

Hawkular APM with Jaeger● Hawkular deprecated Hawkular APM project

○ http://www.hawkular.org/blog/2016/09/19/hawkular-apm-on-msa.html

● Instead, Hawkular Team collaborate on the Jaeger project○ http://www.hawkular.org/blog/2017/04/19/hawkular-apm-jaeger.html

● Uber’s Yuri Shkuro and Red Hat’s Gary Brown, both core contributors to the Jaeger Distributed Tracing project

○ Hawkular Team contributing to OpenTracing standard

● Distributed tracing based on OpenTracing standard● OpenTracing is the vendor neutral open standard for distributed tracing

○ Adopted by CNCF (Cloud Native Computing Foundation)

● https://github.com/uber/jaeger ● http://opentracing.io/documentation/pages/quick-start ● https://eng.uber.com/distributed-tracing/ ● Run Jaeger

○ docker run -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 -p5775:5775/udp -p6831:6831/udp -p6832:6832/udp -p5778:5778 -p16686:16686 -p14268:14268 -p9411:9411 jaegertracing/all-in-one:latest

● Jaeger UI○ http://localhost:16686/

Let’s run Jaeger

Jaeger Example● Spring Boot App /chaining call => Spring Boot /hello => WildFly Swarm /hello

○ https://github.com/pavolloffay/opentracing-java-examples ○ http://www.hawkular.org/blog/2017/06/9/opentracing-spring-boot.html○ http://www.hawkular.org/blog/2017/07/opentracing-jaxrs.html

Jaeger Example

Jaeger Example

Jaeger - a Distributed Tracing System

References● http://www.hawkular.org/● http://www.hawkular.org/overview/ ● https://access.redhat.com/articles/2221611 ● http://www.hawkular.org/hawkular-services/docs/installation-guide/ ● http://www.hawkular.org/hawkular-services/docs/quickstart-guide/ ● http://www.hawkular.org/hawkular-services/docs/user-guide/inventory/ ● http://www.hawkular.org/hawkular-services/docs/user-guide/inventory/inventory-v3-details.html ● http://www.hawkular.org/hawkular-apm/ ● http://jaeger.readthedocs.io/en/latest/ ● http://www.hawkular.org/hawkular-metrics/docs/user-guide/● http://www.hawkular.org/hawkular-metrics/docs/user-guide/installation.html● http://www.hawkular.org/hawkular-metrics/docs/user-guide/configuration.html● https://www.slideshare.net/ponceballesteros/hawkular-alerting● http://www.hawkular.org/community/docs/developer-guide/alerts-v1.html ● http://www.hawkular.org/community/docs/developer-guide/alerts-v2.html● https://github.com/ManageIQ/manageiq-providers-hawkular/blob/master/GETTING_STARTED.md● https://github.com/hawkular/hawkular-alerts/tree/master/examples/hello-world● http://highscalability.com/blog/2010/4/27/paper-dapper-googles-large-scale-distributed-systems-tracing.html● http://www.hawkular.org/blog/2016/09/19/hawkular-apm-on-msa.html● https://eng.uber.com/distributed-tracing/● https://www.slideshare.net/YuriShkuro/distributed-tracing-at-uber-scale-creating-a-treasure-map-for-your-monitoring-data● http://www.hawkular.org/blog/2017/03/24/distributed-tracing-with-camel.html ● https://www.cncf.io/blog/2017/09/13/cncf-hosts-jaeger/ ● https://blog.openshift.com/openshift-commons-briefing-82-distributed-tracing-with-jaeger-prometheus-on-kubernetes/

Thank you