https://2016.opensource.hk/topics/fluentd-unified-logging-layer/

1. Fluentd:
Unified logging layer
June 25, 2016 / HKOScon
Masahiro Nakagawa

2. Who am I
• Masahiro Nakagawa
• github: @repeatedly
• Treasure Data, Inc.
• Senior Software Engineer
• Fluentd / td-agent developer
• Living at OSS :)
• D language - Phobos, a.k.a standard library, committer
• Fluentd - Main maintainer
• MessagePack / RPC
• etc…

3. Background

4. Before
✓ duplicated code for error handling...
✓ messy code for retrying mechanism...

5. After

6. Structured logging
Reliable forwarding
Pluggable architecture
http://fluentd.org/

7. Concept
/
Design

8. What’s Fluentd?
• Data collector for unified logging layer
• Streaming data transfer based on JSON /
MessagePack
• Robust core and plugins written in Ruby
• Rubygems based various plugins by the community
• www.fluentd.org/plugins
• Apache License, Version 2.0
• https://github.com/fluent/fluentd

9. Reliable data transfer
error retry
error retry retry
retry
Batch
Stream
Other stream
(micro-batch)

10. Core
Common concerns Use case specific
PLUGINSCORE
• Read Data
• Parse Data
• Buffer Data
• Write Data
• Format Data
• Divide & Conquer
• Buffering & Retries
• Error Handling
• Message Routing
• Parallelism

11. > logged time
Event structure(log message)
✓ Time
> for message routing
✓ Tag
> Actual content
> JSON / MessagePack
✓ Record
127.0.0.1 - - [11/Dec/2012:07:26:27] "GET / ...
127.0.0.1 - - [11/Dec/2012:07:26:30] "GET / ...
127.0.0.1 - - [11/Dec/2012:07:26:32] "GET / ...
127.0.0.1 - - [11/Dec/2012:07:26:40] "GET / ...
...
2012-12-11 07:26:27
apache.log
{
"host": "127.0.0.1",
"method": "GET",
...
}
Apache log Fluentd event
convert
time
tag
record

12. Architecture

13. Data processing pipeline
ParserInput Buffer Output FormatterFilter
“output-ish”“input-ish”

14. Input plugins
File tail (in_tail)
Syslog (in_syslog)
HTTP (in_http)
RDBMS (in_sql)
...
✓ Receive logs
✓ Or pull logs from data sources
✓ non-blocking
InputInput

15. Parser plugins
JSON
Regexp
Apache/Nginx
Syslog
CSV/TSV
etc.
✓ Parse into JSON
✓ Common formats out of the box
✓ Some inputs plugin depends on
Parser plugin
ParseParser

16. Filter plugins
grep
record_transformer
suppress
…
✓ Filter / Mutate record
✓ Record level and Stream level
✓ v0.12 and above
Filter

17. Buffer plugins
✓ Improve performance
✓ Provide reliability
✓ Provide thread-safetyMemory (buf_memory)
File (buf_file)
Buffer

18. Buffer internal
✓ Chunk = adjustable unit of data
✓ Buffer = Queue of chunks
chunk
chunk
chunk output
Input

19. Formatter plugins
✓ Format output
✓ Convert json into other format
✓ Some plugins depends on
Formatter plugins
JSON
CSV/TSV
“single value”
msgpack
Formatter

20. Output plugins
✓ Write to external systems
✓ Buffered & Non-buffered
✓ 300+ pluginsFile (out_file)
Amazon S3 (out_s3)
Kafka (out_kafka)
...
Output

21. Use
cases

22. Simple Forwarding

23. # logs from a file
<source>
@type tail
path /var/log/httpd.log
pos_file /tmp/pos_file
format apache2
tag backend.apache
</source>
# logs from client libraries
<source>
@type forward
port 24224
</source>
# store logs to MongoDB
<match backend.*>
@type mongo
database fluent
collection test
</match>

24. Less Simple Forwarding
- At-most-once / At-least-once
- HA (failover)
- Load-balancing

25. Container Logging

26. Treasure Data
Frontend
Job Queue
Worker
Hadoop
Presto
Fluentd
Applications push
metrics to Fluentd
(via local Fluentd)
Datadog
for realtime monitoring / alerting
Treasure Data
for historical analysis
Fluentd sums up data minutes
(partial aggregation)

27. Slideshare
http://engineering.slideshare.net/2014/04/skynet-project-monitor-scale-and-auto-heal-a-system-in-the-cloud/

28. Other companies…
https://www.fluentd.org/testimonials

29. Roadmap

30. History / Roadmap
• v0.10.0 (In Oct 2011)
• v0.12.0 (In Dec 2014) -> Current stable
• v0.14.0 (In May 2016)
• v0.14.x (some versions in 2016)
• v1 (4Q in 2016 / 1Q in 2017)

31. v0.10 (old stable)
> First stable release
> Mainly for log forwarding
> Only Input and Output
> No complex event handling support
> Only At-most-once semantics in forwarding
> Treasure Data provides td-agent 1 for v0.10

32. v0.12 (current stable)
> Event handling improvement
> Filter, Label, Error Stream
> New configuration format
> Add at-least-once semantics in forwarding
> Use require_ack_response parameter
> HTTP RPC based process management
> Treasure Data provides td-agent 2 for v0.12

33. • New Plugin APIs, Plugin Helpers & Plugin Storage
• Supervisor using ServerEngine
• Time with Nanosecond precision
• Windows support
v0.14 (Next stable)

34. Time with nanosecond
• Fluent::EventTime
• behaves as Integer (used as time in v0.12)
• has methods to get sub-second precision
• be serialized into msgpack using Ext type
• Fluentd core can handle both of Integer and
EventTime as time
• compatible with older versions and software in eco-
system (e.g., fluent-logger, Docker logging driver)

35. Windows support
• Fluentd and core plugin work on Windows
• several companies have already used
v0.14.0.pre version on production
• We will send a patch to popular plugins if
it doesn’t work on Windows
• Use HTTP RPC instead of signals
• Treasure Data will provide td-agent msi package

36. v0.14.x - v1
• v0.14.x (some versions in 2016)
• Symmetric multi processing model
• Counter API
• TLS/authentication/authorization support
(merging secure-forward into core)
• v1 (4Q in 2016 / 1Q in 2017)
• Stable version for new APIs/features
• fully compatible with v0.12.x and v0.14.x

37. Symmetric multi processing model
• 2 or more workers share a configuration file
• and share listening sockets via PluginHelper
• under a supervisor process (ServerEngine)
• Multi core scalability for huge traffic
• one input plugin for a tcp port, some filters and
one (or some) output plugin
• No more fluent-plugin-multiprocess

38. Worker
Supervisor
Worker Worker
Worker
Supervisor
Worker Worker
Supervisor Supervisor
Using fluent-plugin-multiprocess
v0.14

39. TLS/Authn/Authz support for forward plugin
• secure-forward will be merged into built-in forward
• TLS w/ at-least-one semantics
• Simple authentication/authorization w/ non-SSL
forwarding
• Authentication and Authorization providers
• Who can connect to input plugins?
What tags are permitted for clients?
• New plugin types (3rd party authors can write it)
• Mainly for in/out forward, but available from others

40. Ecosystem

41. Treasure Agent (td-agent)
> Treasure Data distribution of Fluentd
> include Ruby runtime, fluentd and popular plugins
> deb, rpm, dmg are supported
> Treasure Agent 2 is current stable
> fluentd v0.12 and Ruby 2.1
> Treasure Agent 3 will be released at fall
> fluentd v0.14 and Ruby 2.3
> msi will be supported for Windows

42. fluentd-ui
> Manage Fluentd instance via Web UI
> https://github.com/fluent/fluentd-ui

43. fluent-bit
> http://fluentbit.io/
> Log collector mainly for embedded systems
> Written in C
> Can communicate with Fluentd
> There are several plugins
> Treasure Data, Elasticsearch, MQTT, etc…

44. > Bulk Loader version of Fluentd
> Pluggable architecture
> JRuby, JVM languages
> High throughput by parallel processing
> Invented by Treasure Data
> Share your script as a plugin
> https://github.com/embulk
Slide: Fighting Against Chaotically SeparatedValues with Embulk

45. treasuredata.com
Enjoy logging!