1. Log everyting in JSON.
Treasuare Data, Inc.
Sadayuki Furuhashi

2. Self-introduction
> Sadayuki Furuhashi
twitter: @frsyuki
> Original author of Fluentd
> Treasure Data, Inc.
Software Architect; Founder
> open-source
MessagePack - efficient serialization format

3. 0. Why logging?
1. Why Fluentd? - Design of Fluentd
> Extensibility
> Unified log format
> Simplicity
2. Who uses Fluentd?
3. Future of Fluentd

4. 0. Why logging?
1. Why Fluentd? - Design of Fluentd
> Extensibility
> Unified log format
> Simplicity
2. Who uses Fluentd?
3. Future of Fluentd

5. 0. Why logging?
> Error notifications
> Performance monitoring
> User segment analysis
> Funnel analysis
> Heatmap analysis
> Market prediction
etc...

6. 0. Why logging? - Error notifications
Error!

7. 0. Why logging? - Performance monitor

8. 0. Why logging? - User segment analysis

9. 0. Why logging? - Funnel analysis
-28%!
-27%!

10. 0. Why logging? - Heatmap analysis

11. 0. Why logging? - Market prediction

12. 0. Why logging?
1. Why Fluentd? - Design of Fluentd
> Extensibility
> Unified log format
> Simplicity
2. Who uses Fluentd?
3. Future of Fluentd

13. 0. Why logging?
1. Why Fluentd? - Design of Fluentd
> Extensibility
> Unified log format
> Simplicity
2. Who uses Fluentd?
3. Future of Fluentd

14. Alerting
Nagios
Analysis
MongoDB
MySQL
Hadoop
log utilization
Archiving
Amazon S3

15. Access logs Alerting
Apache Nagios
App logs Analysis
Frontend log sources MongoDB
Backend MySQL
System logs Hadoop
syslogd log utilization
Archiving
Databases Amazon S3

16. Access logs Alerting
Apache Nagios
App logs Analysis
Frontend MongoDB
Backend MySQL
System logs Hadoop
syslogd
Archiving
Databases Amazon S3

17. Access logs rsync servers
Alerting
Apache Nagios
App logs Analysis
Frontend MongoDB
Backend MySQL
bash scripts
System logs Hadoop
syslogd
Archiving
Databases Amazon S3
perl scripts

18. Problems...
No unified method to collect logs
> Too many bash/perl scripts
Fragile for changes
Less reliable
> Mixed log formats
Old-fashioned “Human-readable” text logs
Not ready to analyze
> High latency
must wait a day for log rotation

19. Access logs Alerting
Apache Nagios
App logs Analysis
Frontend MongoDB
Backend MySQL
System logs Hadoop
syslogd
Archiving
Databases Amazon S3

20. Access logs Alerting
Apache Nagios
App logs Analysis
Frontend MongoDB
Backend MySQL
System logs Hadoop
syslogd
Archiving
filter / buffer / routing
Databases Amazon S3

21. Input Plugins Output Plugins
Buffer Plugins
Filter Plugins

22. Input Plugins Output Plugins
2012-02-04 01:33:51
JSON format
myapp.buylog {
“user”: ”me”,
“path”: “/buyItem”,
“price”: 150,
“referer”: “/landing”
}

23. Input Plugins Output Plugins
time
tag
2012-02-04 01:33:51
JSON format
myapp.buylog {
“user”: ”me”,
“path”: “/buyItem”,
“price”: 150,
“referer”: “/landing”
}
record

24. Why Fluentd?
> Extensibility - Plugin architecture
collect logs from various systems
forward logs to various systems
> Unified log format - JSON format
modern “Machine-readable” log format
immediately ready to analyze
> Reliable - HA configuration
> Easy to install - RPM/deb packages
deploy instantly to everywhere

26. Comparision with other log collectors:
> Scribe
Less extensible
No unified log format
No longer developped?
> Flume
Less simple
No unified log format
Little information about Flume-NG

27. 0. Why logging?
1. Why Fluentd? - Design of Fluentd
> Extensibility
> Unified log format
> Simplicity
2. Who uses Fluentd?
3. Future of Fluentd

28. 0. Why logging?
1. Why Fluentd? - Design of Fluentd
> Extensibility
> Unified log format
> Simplicity
2. Who uses Fluentd?
3. Future of Fluentd

29. NHN Japan COOKPAD NAVER
Crocos
http://www.quora.com/Who-uses-Fluentd-in-production

30. 0. Why logging?
1. Why Fluentd? - Design of Fluentd
> Extensibility
> Unified log format
> Simplicity
2. Who uses Fluentd?
3. Future of Fluentd

31. 0. Why logging?
1. Why Fluentd? - Design of Fluentd
> Extensibility
> Unified log format
> Simplicity
2. Who uses Fluentd?
3. Future of Fluentd

32. Future of Fluentd
> <filter>
> <match> in <source>
> <label>
> MessagePack for Ruby v5
> td-agent-lite
> Pub/Sub & Monitoring API
> New process model & Live restart
> Backward compatibility

33. <source> <match **>
type tail type forward
path /var/log/httpd.log host log.server
format apache </match>
tag not_filtered.apache
</source>
Mysterious tag
<match not_filetered.**>
type rewrite
remove_prefix not_filtered tag operations
<rule>
key status
pattern ^500$
ignore true
</rule>
</match>
Before

34. <source> <match **>
type tail type forward
path /var/log/httpd.log host log.server
format apache </match>
tag apache
</source>
<filter **> Filter plugins!
type rewrite
<rule>
key status
pattern ^500$
ignore true
</rule>
</match>
After (v11)

35. <source> <match **>
type tail type forward
path /var/log/httpd.log host log.server
format apache </match>
tag apache
<filter **>
type rewrite
<rule>
key status
pattern ^500$
<filter>/<match> in
ignore true <source>
</rule>
</match>
</source>
After (v11)

36. <source> <match **>
type tail type forward
path /var/log/httpd.log host log.server
tag apache </match>
</source>
I want to add flowcounter here...
Before

37. <source> <store>
type tail type forward
path /var/log/httpd.log host log.server
tag apache </store>
</source> </match>
<match flow.traffic>
type forward
host traffic.server
</match> Nested!
<match **>
type copy
<store>
type flowcounter
tag flow.traffic
</store>
Before

38. <source> <match **>
type tail type forward
path /var/log/httpd.log host log.server
tag apache </match>
</source>
<filter **>
type copy
<match>
type flowcounter
tag flow.traffic
<match>
type forward
host traffic.server
</match>
</match> Filtering pipeline
</match>
After (v11)

39. <source> # copy & label & forward
type forward <filter **>
</source> type copy
<match>
<filter **> type forward
type copy label alert
<match> host alerting.server
type file </match>
path /mnt/local_archive </filter>
</match>
</filter> # copy & label & forward
<filter **>
<label alert> type copy
<match **> <match>
... type forward
</match> label analysis
</label> host analysis.server
</match>
<label analysis> </filter>
...
</label>
After (v11)

40. MessagePack for Ruby v5
(tweets/sec)
40000
30000
20000
10000
0
Serialize Deserialize
msgpack v5 msgpack v4 yajl json

41. td-agent-lite
> in_tail + out_forward in “single” binary
statically linked ruby binary + scripts tied with the binary

42. New process model & Live restart
Old multiprocess model
detached
process
fork()
Supervisor Engine
detached
all data pass through process
the central process

43. New process model & Live restart
New multiprocess model
detached
process
Process
Supervisor Engine
Manager
detached
process
direct communication

44. New process model & Live restart
New multiprocess model
detached
process
Process
Supervisor Engine
Manager
detached
Live restart process
Process
Engine
Manager

45. Backward compatibility
Fluentd v11 includes 2 namespaces:
> Fluentd:: new code base
> Fluent:: old code base + wrapper classes
Checkout the repository for details:
> http://github.com/frsyuki/fluentd-v11

46. Conculution
Fluentd makes logging better
> Plugin architecture
> JSON format
> HA configuration
> RPM/deb package
Fluentd is under active development
Fluentd is suppored by many committers

47. contact: sales@treasure-data.com

50. ログ収集/解析に使っているツール

51. ログの保存先

52. Fluentdを導入するにあたっての障壁