Log Monitoring at eBay kleinanzeigen (April 2013)
-
Upload
manuel-aldana -
Category
Technology
-
view
29 -
download
3
Transcript of Log Monitoring at eBay kleinanzeigen (April 2013)
![Page 1: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/1.jpg)
Log Monitoring für schnelles Feedback
bei eBay Kleinanzeigen
Manuel Aldana (eBay Kleinanzeigen)
![Page 3: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/3.jpg)
Change / Feedback
flickr.com/photos/tomas_sobek/4649690892/
![Page 4: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/4.jpg)
Change / Feedback
FeedbackReaktion
Change
BenutzerAktivität
(inhouse/extern)
Logs
Daten-Zustand(Persistence)
Operativ(I/O, CPU, Capacity)
![Page 5: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/5.jpg)
Log Typen
HTTPLogs (Apache,
Varnish)
ApplikationsLogs
JavaScriptLogs
FachlicheLogs
OSLogs
BackendLogs (DB, Search, ..)
- Log-Qualität- Sichtbarkeit ?
![Page 6: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/6.jpg)
Old-School Logs-Scanning
tail –f, cat, grep Hölle
![Page 7: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/7.jpg)
Post-Rollout Donkey
Pre-Rollout Post-Rollout
tsss....
![Page 8: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/8.jpg)
Feedback damals
Angry Mean Face by David Rader IICC license: flickr.com/photos/chexed/1546923312
Ops
Benutzer
Customer Support
![Page 9: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/9.jpg)
Risikominimierung
Pre-Rollout Post-Rollout
Verbessertes Log-Feedback
![Page 10: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/10.jpg)
Verbesserte Log-Infrastruktur
● Zentralisiert ● Aggregiert ● Nahe-Echtzeit● Suchbar● Sichtbar für alle!!
APP X APP Y APP Z
Zentralisiert Speicher + Suche
![Page 11: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/11.jpg)
Feedback heute
![Page 12: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/12.jpg)
Log Daten
● Log-Beschreibung● Version● Applikationstyp ● Stack-trace● User-Agent● URL● Cookies● Timestamp● Payload (POST, PUT)● Level: INFO,WARN,ERROR● Log-Source (z.B. MeineKlasse.java:78)
![Page 13: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/13.jpg)
Vorstellung Log-Tool...
![Page 14: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/14.jpg)
Log Incident-Code
![Page 15: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/15.jpg)
Implementierungsdetails
● log4j-Appender● graylog2-server (0.9.5)● MongoDB storage● Indexer per Groovy + Cron-Job● Solr Suche (3.2)● Inhouse frontend (Play framework 1.2.4,
google-charts)
![Page 16: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/16.jpg)
Custom
Reuse
![Page 17: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/17.jpg)
Custom
Reuse
![Page 18: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/18.jpg)
Log Stats Production
● ~40 "loggende" Server-Instanzen● ~600K Logs pro Tag● minutenaktuell● insg. 17M Logs bei 1 Monat Historie● Index Größe Solr: 5 GB● MongoDB Size: 25 GB (klein angesetzt)
![Page 19: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/19.jpg)
Alarming
● Monitoring != Alarming● Alarm on ERROR threshold● Nagios Plugin● Mail Event● Audio Event
![Page 20: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/20.jpg)
Ausblick
● Solr Upgrade● HTTP + SMTP-Logs nachziehen● Event-Basiertes tracking (statsd, graphite)● Alarming erweitern
![Page 21: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/21.jpg)
Technologie Alternativen
Logster
![Page 22: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/22.jpg)
Eure Anforderungen...
● Wie lange Release Zyklen?● Wie große Change-Sets?● Echtzeitanforderung wie hoch?● Kritikalität von Bugs?● Traffic wie hoch?● Log-Volumen wie hoch?
![Page 23: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/23.jpg)
Zusammenfassung
● Logs wichtige Informationsquelle und KEIN Abfallprodukt
● Erkenne DEINE verursachten Probleme schneller als andere
● Suboptimale Lösung besser als gar keine● Reuse wenn möglich, Anpassen falls nötig
![Page 24: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/24.jpg)
Pre-Rollout Post-Rollout
Produktion Mindset
tsss....
![Page 25: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/25.jpg)
Berlin DevOpsDays 201327./28.05.2012, Urania Berlinhttp://devopsdays.org/events/2013-berlin/
Call for proposalsopen until [email protected]
DevOpsDays - The conference that brings Dev & Ops together
![Page 26: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/26.jpg)
Nicht verzagen, sondern fragen...
![Page 27: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/27.jpg)
Surprise, we are hiring...
1) Senior Frontend Developer
2) Senior Android Developer
1st Contact: [email protected]
![Page 28: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/28.jpg)
Kurze Referenz Folien: Log-Tool
(da Live-Toolvorstellung nicht in Folien übernommen werden kann)
![Page 29: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/29.jpg)
Log Beschreibung
● For overview and Log Hotspots Classification● Normalization (e.g. cut id‘s)
![Page 30: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/30.jpg)
App Version
● Identify issues for new rollout-version
![Page 31: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/31.jpg)
Release Vergleich
![Page 32: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/32.jpg)
Log-Timestamp
● Time related patterns
![Page 33: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/33.jpg)
User-Agent
● Knowing your client... (web + API)
![Page 34: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/34.jpg)
● Reproktion per copy/paste URL + params● Payload Probleme identifizieren
Path + Payload
![Page 35: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/35.jpg)
Stack Trace
● Call-Trace nachvollziehen
![Page 36: Log Monitoring at eBay kleinanzeigen (April 2013)](https://reader030.fdocuments.us/reader030/viewer/2022032620/55cee053bb61eb3c678b45b0/html5/thumbnails/36.jpg)
Log-Source
● Schnell direkt im Code nachvollziehen