Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
zeromq
zeromq
Loading in …3
×
1 of 36

Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

5

Share

Download to read offline

Fluentd is an open source data collector to simplify and scale log management. In this talk, I gave a brief overview of Fluentd, focusing on the why: Why should you care about logging? Why do you want a unified logging layer? Why Fluentd?

More Related Content

You Might Also Like

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

  1. 1. Fluentd: Towards Unified Logging Layer Kiyoto Tamura @kiyototamura Treasure Data, Inc.
  2. 2. • Not an ops engineer myself • “marketing guy” • My laptop isn’t very hipster Why you shouldn’t listen to me
  3. 3. • Fluentd maintainer and community manager • Former programmer • data nerd Why you should listen to me
  4. 4. why Fluentd?
  5. 5. I was a data scientist before it was cool to be one.
  6. 6. http://cacm.acm.org/blogs/blog-cacm/169199-data-science-workflow- overview-and-challenges/fulltext
  7. 7. Acquire Data (or so you think) WUT!? Invalid UTF8? Fix the encoding issue… Yell at the engineers Some columns are missing!? Run the script…DIVISION BY ZERO!!!
  8. 8. Hmm…
  9. 9. Logging.priority => :not_super_high
  10. 10. analytics.priority => :very_high
  11. 11. analytics.needs? :logs => true
  12. 12. outage.priority => :super_high
  13. 13. outage.needs? :logs => [“no”, “shit”]
  14. 14. Unified Logging Layer
  15. 15. overview
  16. 16. 127.0.0.1 - - [05/Feb/2012:17:11:55 +0000] "GET / HTTP/1.1" 200 140 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/535.19"
  17. 17. { "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "140", "referer": "-", "agent": “Mozilla/5.0 (Windows…" }
  18. 18. Parse as JSON!
  19. 19. ?
  20. 20. [“05/Feb/2012:17:11:55”,“web.access”,{ "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "140", "referer": "-", "agent": “Mozilla/5.0 (Windows…" }]
  21. 21. ? web.mongodb web.file web.hdfs web.s3 web.mysql
  22. 22. Fluentd Design Patterns (credit: @tagomoris’s talk at RedDotRubyConf)
  23. 23. Real World Use Cases
  24. 24. • Automate monitoring! • App and System metrics • JSON everywhere
  25. 25. • 2000+ node • ~1B events/day • Forwarder- Aggregator
  26. 26. Demo
  27. 27. Fluentd Treasure Data Twitter API #NED #ARG Postgres + d3.js Collect Raw Tweets Tokenize + Count serve + visualize
  28. 28. { “install”: “gem install fluentd”, “website”: “www.fluentd.org”, “github” : “fluent/fluentd”, “twitter”: “@fluentd” }

×