Treasure Data Summer Internship 2016
-
Upload
yuta-iwama -
Category
Engineering
-
view
3.288 -
download
3
Transcript of Treasure Data Summer Internship 2016
Internship Final ReportSep 30, 2016 Yuta Iwama
Who am I
• Yuma Iwama (@ganmacs)
• Master’s student, The University of Tokyo
• Research: Programming languages (My theme is extending language syntax)
• Group: Chiba Shigeru Group
What I did in summer intern
• Add features and enhancements to Fluentd v0.14.x
What I did in summer intern
• 6 features • Counter API (Not merged yet) • Data compression in buffer plugins and forward plugins • New out_file plugin only for <secondary> section • A CLI tool to read dumped event data • Log rotation • `filter_with_time` method in filter plugins
• 2 enhancements • Optimizing multiple filter calls • Add event size to options in a forward protocol
• Some small tasks
I’ll talk about
• Counter API • Data compression in buffer plugins and
forward plugins • New out_file plugin only for <secondary>
section • A CLI tool to read dumped event data • Log rotation • Optimizing multiple filter calls
I’ll talk about
• Counter API • Data compression in buffer plugins and
forward plugins • New out_file plugin only for <secondary>
section • A CLI tool to read dumped event data • Log rotation • Optimizing multiple filter calls
Data Compression in buffer plugins and
forward plugins
Current buffer plugins and forward plugins in Fluentd• Buffer plugins have data as a string (formatted
with MessagePack or user custom formats)
• Forward plugins send data as a string (format is same as buffer plugins)
• Although data is serialized with MessagePack, its footprint is large
• Current way consumes many memory resources and bandwidth of the network
New buffer plugins and forward plugins• String data in buffer plugins can be compressed
• Forward plugins can send and receive compressed data
• Things to be able to
• Save the bandwidth across the datacenter
• Accelerate the transfer speed and save the time
• Reduce memory consumptions and costs of IaaS (EC2 , etc.)
Implementation
• I used “zlib” in Ruby to implement a compression/decompression method
• It’s hard to work both compressed version and raw version (To solve this problem, I used `extend` in Ruby not to break existing interface)
New out_file plugin only for <secondary> section
Background
• Many users use out_file plugin to dump buffer with <secondary> sections when primary buffered output plugins are failing flush
• But out_file is too complex and has too many features for such purpose
• => We need simple out_file only for <secondary> section just to dump buffer
New plugin: secondary_out_file
• Only four attributes • directory: the directory dumped data saved • basename: the file name of dumped data (default value is
dump.bin) • append: the flushed data is appended to an existing file or
not (default false) • compress: The type of the file ( gzip or txt, default is txt)
• Users can use this plugin only to set directory<match > @type forward ... <secondary> type secondary_file directory log/secondary/ </secondary> </match>
A CLI tool which is used for reading dump data
Background
• Dumped data are created by secondary plugins (e.g. secondary_out_file) when primary plugins are failing flush
• We can't read dumped data because dumped data is binary format(MessagePack) in most case
• => Provide a CLI tool to read dumped data
fluent-binlog-reader
• fluent-buinlog-reader is bundled in Fluent • It reads dumped data and outputs readable
format • Users can use fluent’s formatter plugins as an
output format$ fluent-binlog-reader --help Usage: fluent-binlog-reader <command> [<args>]
Commands of fluent-binlog-reader: cat : Read files sequentially, writing them to standard output. head : Display the beginning of a text file. format : Display plugins that you can use. See 'fluent-binlog-reader <command> --help' for more information on a specific command.
fluent-binlog-reader
$ fluent-binlog-reader head packed.log 2016-08-12T17:24:18+09:00 packed.log {"message":"dummy"} 2016-08-12T17:24:18+09:00 packed.log {"message":"dummy"} 2016-08-12T17:24:18+09:00 packed.log {"message":"dummy"} 2016-08-12T17:24:18+09:00 packed.log {"message":"dummy"} 2016-08-12T17:24:18+09:00 packed.log {"message":"dummy"}
Default format is json format
Using a “csv formatter” to output dumped data
$ fluent-binlog-reader cat --formats=csv -e fields=message packed.log "dummy" ... "dummy"
Log rotation
Background
• Fluentd can’t do log rotation • As the file size of log increases, it becomes
difficult to handle the log file.
• => Fluentd supports log rotation to keep log files down to a manageable size
Log rotation
• Two options • log-rotate-age: The number of old log files
to keep • log-rotate-size: Maximum log file size
• Use serverengine log rotation (Fluentd uses serverengine logger to one’s log)
Optimise multiple filter calls
Background
• If users apply multiple filters to incoming events, Fluentd creates a lot of EventStream object and calls its add method
• => Removing useless instantiations of EventStream and the `add` method calls
Filter 11. Create an EventStream object (1 time) 2. Apply a filter to each event (5 times) 3. Add a filtered event to an EventStream object (5times)
[e1, e2, e3, e4, e5]
If 10 filters are applied 1. call 10 times 2. call 50 times 3. call 50 times
[e1’, e2’, e3’, e4’, e5’]
[e1x, e2x, e3x, e4x, e5x]
Filter n
Current filters
Filter 1
1. Create an EventStream object (1 time) 2. Apply each filters to each event (n * 5 times) 3. Add a filtered event to an EventStream object (5times)
[e1, e2, e3, e4, e5]
if 10 filters are applied 1. call 1 time 2. call 50 times 3. call 5 times
[e1x, e2x, e3x, e4x, e5x]
Filter n+
Constraint: These filters must not be implemented `filter_stream` method
Optimised case
Performance
Tool : ruby-prof ProductName: Mac OS X ProductVersion: 10.11.6 BuildVersion: 15G31 PROCESSOR: 2.7 GHz Intel Core i5 MEMORY: 8 GB 1867 MHz DDR3
Not optimized Optimised
0.063186 0.051646
1.2 times faster when it is using 10 filters and 1000 events per sec
I’ll talk about
• Counter API • Data compression in buffer plugins and
forward plugins • New out_file plugin only for <secondary>
section • A CLI tool to read dumped event data • Log rotation • Optimizing multiple filter calls
Counter API
Motivations
• To get metrics of Fluentd itself between processes
• To provide counter API to 3rd party plugins
• It is useful to implement counter plugins (e.g. fluent-plugin-datacounter and fluent-plugin-flowcounter)
What’s the counter
• The counter:
• A key-value store
• Used for storing the number of occurrences of a particular event in the specified time
• Provides API to users to operate its value(e.g. inc, reset , etc.)
• shared between processes
Counter
key value
key1 5
key2 1.2
key3 3
CounterProcess 1 inc(key1 => 2)
Process 2 reset(key2)
What’s the counter (cont.)
Counter
Counter
What’s the counter (cont.)
key value
key1 7
key2 1.2
key3 0
Process 1 inc(key1 => 2)
Process 2 reset(key2)
Implementation
• RPC server and client
• All operators should be thread safe
• Cleaning mutex objects for keys
Implementation
• RPC server and client
• All operators should be thread safe
• Cleaning mutex objects for keys
RPC server and client
• Because the counter is shared between processes. We need a server and clients (Store counter values in server and clients manipulate them by RPC )
• I designed RPC server and client for counter • I use cool.io to implement RPC server and client
• cool.io is providing a high-performance event framework for Ruby (https://coolio.github.io/)
API to operate counter values
• init: create new value • reset: reset a counter value • delete: delete a counter value • inc: increment or decrement a counter value • get: fetch a counter value
Implementation
• RPC server and client
• All operators should be thread safe
• Cleaning mutex objects for keys
All operations should be a thread safe
• Counter works in multi threads
• You need to get a lock per keys when you change a counter value
• Counter stores mutex objects in hash (key_name => mutex_object)
How an inc method works
key valuekey1 2
Counter server
client in worker1inc( key1 => 2)
1. Call an inc method
key value
key1 mutex obj
Mutex hash
How an inc method works
key valuekey1 2
Counter server
key value
key1 mutex obj
Mutex hashclient in worker1inc( key1 => 2)
1. Call an inc method 2. Get a lock for a mutex hash
locked
How an inc method works
key valuekey1 2
Counter server
key value
key1 locked
Mutex hashclient in worker1inc( key1 => 2)
1. Call an inc method 2. Get a lock for a mutex hash 3. Get a lock for a key
locked
How an inc method works
key valuekey1 2
Counter server
key value
key1 locked
Mutex hashclient in worker1inc( key1 => 2)
1. Call an inc method 2. Get a lock for a mutex hash 3. Get a lock for a key 4. Unlock a mutex hash
How an inc method works
key valuekey1 4
Counter server
key value
key1 locked
Mutex hashclient in worker1inc( key1 => 2)
1. Call an inc method 2. Get a lock for a mutex hash 3. Get a lock for a key 4. Unlock a mutex hash 5. Change a counter value
How an inc method works
key valuekey1 4
Counter server
key value
key1 unlock
Mutex hashclient in worker1inc( key1 => 2)
1. Call an inc method 2. Get a lock for a mutex hash 3. Get a lock for a key 4. Unlock a mutex hash 5. Change a counter value 6. Unlock a key lock
Implementation
• RPC server and client
• All operators should be thread safe
• Cleaning mutex objects for keys
Mutex objects for keys
• To avoid storing mutex objects for all keys, I implement a cleanup thread which removes unused key’s mutex object (like GC)
• This thread removes mutex objects which are not used for a certain period
Cleaning up a mutex hash
key valuekey1 2
Counter serverkey value
key1 mutex obj
Mutex hash
• If “key1” are not modified for a long period, “key1” may be unused after this
Cleaning up a mutex hash
key value
key1 mutex obj
Mutex hash
• If “key1” are not modified for a long period, “key1” may be unused after this
1. Start a cleaning thread (once in 15 min)
Cleaning up a mutex hash
key value
key1 mutex obj
Mutex hash
• If “key1” are not modified for a long period, “key1” may be unused after this
1. Start a cleaning thread (once in 15 min) 2. Get a lock for a mutex hash locked
Cleaning up a mutex hash
key value
Mutex hash
• If “key1” are not modified for a long period, “key1” may be unused after this
1. Start a cleaning thread (once in 15 min) 2. Get a lock for a mutex hash 3. Remove a mutex for an unused key
locked
Cleaning up a mutex hash
key value
Mutex hash
• If “key1” are not modified for a long period, “key1” may be unused after this
1. Start a cleaning thread (once in 15 min) 2. Get a lock for a mutex hash 3. Remove a mutex for an unused key 4. Try to get a lock for the same key
If this thread can’t get a lock restore a key-value
locked
Cleaning up a mutex hash
key value
Mutex hash
• If “key1” are not modified for a long period, “key1” may be unused after this
1. Start a cleaning thread (once in 15 min) 2. Get a lock for a mutex hash 3. Remove a mutex for an unused key 4. Try to get a lock for the same key 5. Unlock a mutex hash
Summary
• Add six features and two enhancements to Fluentd v0.14.x
• Counter API is not merged yet
• Other PRs have been merged
Impression of intern
• The hardest thing for me is to design about counter API(It takes over 1 week)
• I have learned about the development of middleware which is used by many people
• I want to became more careful to code written by myself (typo, description, comment etc.)