Fluentd is an open source data collector to simplify and scale log management. In this talk, I gave a brief overview of Fluentd, focusing on the why: Why should you care about logging? Why do you want a unified logging layer? Why Fluentd?

1. Fluentd:
Towards Unified
Logging Layer
Kiyoto Tamura
@kiyototamura
Treasure Data, Inc.

3. • Not an ops engineer myself
• “marketing guy”
• My laptop isn’t very hipster
Why you shouldn’t listen to me

4. • Fluentd maintainer and
community manager
• Former programmer
• data nerd
Why you should listen to me

5. why Fluentd?

6. I was a data scientist
before it was cool to be
one.

7. http://cacm.acm.org/blogs/blog-cacm/169199-data-science-workflow-
overview-and-challenges/fulltext

8. Acquire Data (or
so you think)
WUT!? Invalid
UTF8?
Fix the encoding
issue…
Yell at the
engineers
Some columns
are missing!?
Run the
script…DIVISION
BY ZERO!!!

9. Hmm…

10. Logging.priority
=> :not_super_high

11. analytics.priority
=> :very_high

12. analytics.needs? :logs
=> true

13. outage.priority
=> :super_high

14. outage.needs? :logs
=> [“no”, “shit”]

18. Unified Logging Layer

19. overview

21. 127.0.0.1 - - [05/Feb/2012:17:11:55
+0000] "GET / HTTP/1.1" 200 140 "-"
"Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/535.19 (KHTML, like Gecko)
Chrome/18.0.1025.5 Safari/535.19"

22. {
"host": "127.0.0.1",
"user": "-",
"method": "GET",
"path": "/",
"code": "200",
"size": "140",
"referer": "-",
"agent": “Mozilla/5.0 (Windows…"
}

24. Parse as JSON!

25. ?

26. [“05/Feb/2012:17:11:55”,“web.access”,{
"host": "127.0.0.1",
"user": "-",
"method": "GET",
"path": "/",
"code": "200",
"size": "140",
"referer": "-",
"agent": “Mozilla/5.0 (Windows…"
}]

27. ?
web.mongodb
web.file
web.hdfs
web.s3
web.mysql

28. Fluentd Design Patterns
(credit: @tagomoris’s talk at
RedDotRubyConf)

29. Real World Use Cases

31. • Automate
monitoring!
• App and System
metrics
• JSON
everywhere

33. • 2000+ node
• ~1B events/day
• Forwarder-
Aggregator

34. Demo

35. Fluentd
Treasure
Data
Twitter API
#NED #ARG
Postgres +
d3.js
Collect Raw Tweets
Tokenize + Count
serve + visualize

36. {
“install”: “gem install fluentd”,
“website”: “www.fluentd.org”,
“github” : “fluent/fluentd”,
“twitter”: “@fluentd”
}