Logging for Containers
-
Upload
eduardo-silva-pereira -
Category
Software
-
view
49 -
download
0
Transcript of Logging for Containers
![Page 1: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/1.jpg)
Logging for Containers
Eduardo Silva @edsiper
![Page 2: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/2.jpg)
Logging
![Page 3: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/3.jpg)
Logging
● Monitoring
● Troubleshooting
● Statistics
It matters
![Page 4: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/4.jpg)
● Operating System
● Applications Unit
● Services
Logging Scenarios
Distribution Channels
![Page 5: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/5.jpg)
Containers
![Page 6: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/6.jpg)
Containers
Everything is about Isolation
![Page 7: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/7.jpg)
● File System
● Standard I/O interfaces (stdout / stderr)
● Over Network
Containers & Log Handlers
Distribution Channels
![Page 8: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/8.jpg)
● Json-file● Syslog● Journald● Fluentd● ...
Docker Implement drivers for different formats and distribution channels:
Docker Logging Drivers
Output / Destinations
![Page 9: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/9.jpg)
Structured Logs
![Page 10: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/10.jpg)
● Often based in Key-Value pairs
● Two minimum keys: time and message
Structured logging makes data processing easier
Structured Logs
Distribution Channels
![Page 11: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/11.jpg)
● JSON: readable format for structured data
● MessagePack: Binary serialization (json-like)
Structured logging makes data processing easier
Structured Logs
Common Formats
![Page 12: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/12.jpg)
Original Log Message
This is a test message
Structured Log Message
{ "container_id":"bfdd5b9...", "container_name":"/infallible_mayer", "source":"stdout", "log": "This is a test message"}
Structured Logs
Docker log example
![Page 13: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/13.jpg)
Microservices
![Page 14: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/14.jpg)
Monolithic● A service produces all
data about users access
Microservices● Many services produce
data about users access● Log needs to be collected
from many services.
Microservices
![Page 15: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/15.jpg)
● How to deal with different input formats ?
● Parse plain text is really expensive.
● Not all containers have permanent storage.
● Where to write the logs ?
Microservices
Logging Challenges
![Page 16: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/16.jpg)
Distributed Logging
![Page 17: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/17.jpg)
Distributed Logging
Architecture
![Page 18: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/18.jpg)
Collector● Retrieve raw logs: file system / network.● Parse log content.
Aggregator● Get data from multiple sources.● Convert incoming data into Streams.
Destination● Retrieve data streams from Aggregator.● Store formatted logs (records).
Distributed Logging
Workflow
![Page 19: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/19.jpg)
● Network Traffic
● CPU Load: parsing and formatting is expensive
● High Availability / Redundant aggregators
Scaling Logging
Items to consider
![Page 20: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/20.jpg)
Aggregation Patterns
![Page 21: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/21.jpg)
w/o source aggregation with source aggregation
Source Aggregation Patterns
![Page 22: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/22.jpg)
Pros● Simple Configuration
Cons● Fixed Aggregator endpoint address● Many network connections● High load on Aggregator
Aggregation Patterns
Without Source Aggregation
![Page 23: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/23.jpg)
Pros● Less connections● Lower load in aggregator● Less config in Containers
Cons● Need more resources (1 aggregate container
per host.
Aggregation Patterns
With Source Aggregation
![Page 24: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/24.jpg)
w/o destination aggregation With destination aggregation
Destination Aggregation Patterns
![Page 25: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/25.jpg)
Pros● Less Nodes● Simpler configuration
Cons● Storage side changes affects collector side● Worse performance: many small write requests
on storage
Aggregation Patterns
Without Destination Aggregation
![Page 26: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/26.jpg)
Pros● Collector side configuration is
free from storage side changes.● Better performance with fine
tune on destination side aggregator.
Cons● More Nodes.● More complex configuration.
Aggregation Patterns
With Destination Aggregation
![Page 27: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/27.jpg)
![Page 28: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/28.jpg)
● High Performance
● Built-in Reliability
● Structured Logs
● Pluggable Architecture
● More than 300 plugins! (input/filtering/output)
Open Source Data/Log Collector
![Page 29: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/29.jpg)
Architecture / Workflow
![Page 30: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/30.jpg)
● Docker Interoperability Native Docker logging driver to use Fluentd
● KubernetesFluentd as main aggregator (notes)
● OpenShiftFluentd as main aggregator
Full Collector/Aggregator for Containers
![Page 31: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/31.jpg)
Docker use case
![Page 32: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/32.jpg)
Docker use case
![Page 33: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/33.jpg)
Kubernetes use case
![Page 34: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/34.jpg)
Fluentd in the Real World
![Page 35: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/35.jpg)
Microsoft use case
![Page 36: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/36.jpg)
![Page 37: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/37.jpg)
We collect1.8M events per second !
![Page 38: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/38.jpg)
It’s a proud member of:
![Page 39: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/39.jpg)
NEWS!
![Page 40: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/40.jpg)
Fluentd is joining CNCF!
![Page 41: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/41.jpg)
![Page 42: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/42.jpg)
Let’s back to Logging
![Page 43: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/43.jpg)
![Page 44: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/44.jpg)
● Written in C
● High Performance
● Pluggable Architecture
● Built-in CPU / Memory metrics / Network TLS support
● Event-Driven
● Fluentd Compatible!
Fluent Bit
Lightweight log aggregator
![Page 45: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/45.jpg)
Fluent Bit
Architecture
![Page 46: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/46.jpg)
Messages Forwarding Performance
500K messages per second! (1 CPU core)
![Page 47: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/47.jpg)
Architecture
![Page 48: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/48.jpg)
Library Mode
![Page 49: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/49.jpg)
Library Mode
![Page 50: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/50.jpg)
Library Mode
![Page 51: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/51.jpg)
Library Mode
![Page 52: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/52.jpg)
Library Mode
![Page 53: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/53.jpg)
Library Mode
![Page 54: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/54.jpg)
Library Mode
![Page 55: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/55.jpg)
Library Mode
![Page 56: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/56.jpg)
Library Mode
![Page 57: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/57.jpg)
Library Mode
![Page 58: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/58.jpg)
Library Mode
![Page 59: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/59.jpg)
● Write plugins in Golang!
● Functional and upcomingfor v0.10 release.
● Do you want to contribute ?
Fluent Bit - Announcement
Golang Support!
![Page 60: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/60.jpg)
Fluent Bit - Announcement
Golang Support!
![Page 61: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/61.jpg)
● Write Go plugins!
● Built-in HTTP Monitoring
● Variables
● Filtering
Fluent Bit
Roadmap v0.10
![Page 62: Logging for Containers](https://reader030.fdocuments.us/reader030/viewer/2022021506/58a83fbc1a28ab30658b5ccf/html5/thumbnails/62.jpg)
Eduardo Silva @[email protected]
http://fluentd.org http://fluentbit.io
Thanks!
Logging and Open Source matters!