3. Fluent
Structured logging
Pluggable architecture
Reliable forwarding
e Event Collector Service
Sadayuki Furuhashi
Treasure Data, Inc.
@frsyuki

4. Fluent
Structured logging
Pluggable architecture
Reliable forwarding
e Event Collector Service
Sadayuki Furuhashi
Treasure Data, Inc.
@frsyuki

6. log

7. log log log log

8. log log log log

9. log log log log

11. Fluent
Structured logging
Pluggable architecture
Reliable forwarding
e Event Collector Service

12. Fluent
Structured logging
Pluggable architecture
Reliable forwarding
e Event Collector Service
Sadayuki Furuhashi
Treasure Data, Inc.
@frsyuki

13. Fluent
Structured logging
Pluggable architecture
Reliable forwarding
e Event Collector Service
Sadayuki Furuhashi
Treasure Data, Inc.
@frsyuki

15. log log log log

16. log log log log

17. log log

18. log log

19. <match ...>
type mongo_backup
capped_size 100m
log
<store>
tcp
type tcp
host 192.168.0.13
...
</store>
</match>

20. log log log log

21. <match mongo.**>
type mongo
database fluent
collection test
log
# Following attibutes are optional
host fluenter
port 10000
# Other buffer configurations here
</match>

22. Mon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 }
Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { replSetGetStatus: 1 } ntoreturn:1 reslen:571 0ms
Mon Nov 14 23:36:22 [conn13] run command admin.$cmd { ismaster: 1 }
Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 reslen:234 0ms
Mon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 } Input
{
_id : ...,
time: Mon Nov 14 23:36:22,
log
key1 : “[conn13]”,
key2 : “command”,
key3 : ”admin.$cmd”,
key4 : {
“ismaster”: 1
},
value : “0ms”,
} Output

24. aggregate aggregate aggregate aggregate
log log log log
key1 key2 key3 shuffle
aggregate aggregate aggregate
aggregate

25. <source>
type tail
format /^(?<time>[^ ]* [^ ]* [^ ]* [^ ]*) (?<key1>[^ ]*) (?<key2>[^ ]*) (?<key3>[^ ]*)
(?<value1>[^ ]*)$/
time_format %a %b %e %H:%M:%S
path /var/log/something.log
tag aggr_hostneme
</source>
<metrics>
<server>
name one_key
name host1
partition_by m
host host1
each_key key1
port 24224
</metrics>
</server>
<metrics>
<server>
name two_keys
name host2
partition_by m
host host2
each_key key2,key3
port 24224
value_key value1
</server>
type float
...
</metrics>

26. Mon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 }
Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { replSetGetStatus: 1 } ntoreturn:1 reslen:571 0ms
Mon Nov 14 23:36:22 [conn13] run command admin.$cmd { ismaster: 1 }
Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 reslen:234 0ms
Mon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 } Input
{
_id : "399e94941cacf13eeb3f808e8ac00981",
delta1 delta2 delta3
name : one_key,
partition : "2011-11-14 19:17"
key : {
key1 : "PeriodicTask::Runner"
}, delta4 delta5
count : 30,
value : {
response : 1024
} delta6
} Output

27. key1 key3 shuffle
key2
aggregate aggregate aggregate
...
shard key1 shard key3
shard key2
aggregate aggregate
...

28. Fluent
Structured logging
Pluggable architecture
Reliable forwarding
e Event Collector Service
Sadayuki Furuhashi
Treasure Data, Inc.
@frsyuki