1. Norikra in action
Data/Stream Processing Meetup (2013/06/28)
TAGOMORI Satoshi (@tagomoris)
13年6月29日土曜日

2. TAGOMORI Satoshi (@tagomoris)
LINE corp.
Ruby, Perl, Node.js, Hadoop, ...
13年6月29日土曜日

3. 13年6月29日土曜日

4. System Overview
Web
Servers Fluentd
Cluster
Archive
Storage
(scribed)
Fluentd
Watchers
Graph
Tools
Notifications
(IRC)
Hadoop Cluster
(HDFS, YARN)
webhdfs
Huahin
Manager
hive
server
STREAM
Shib ShibUI
BATCH
SCHEDULED
BATCH
Norikra
13年6月29日土曜日

5. Stream query
Custom fluentd plugin: not so casual
enough
xQL: declarative language
streams processing
for optional data fields
no more schema management
connectivity with Fluentd
13年6月29日土曜日

6. Stream query:
vs stored data query
No more query wait time
Immediate result for time batch
No more storages
No more query execution management
Once register query, runs forever
13年6月29日土曜日

7. Esper
"Esper and Event Processing Language (EPL)
provide a highly scalable, memory-efficient, in-
memory computing, SQL-standard, minimal
latency, real-time streaming Big Data processing
engine for medium to high-velocity and high-
variety data."
http://esper.codehaus.org/
13年6月29日土曜日

8. Norikra:
is not for only Fluentd.
13年6月29日土曜日

9. Norikra query:
vs Fluentd custom plugin
SQL!!!
No more restart for new queries
register queries whenever we want
No more private plugins
No more fat Fluentd configurations
13年6月29日土曜日

10. Norikra
Full feature of Esper over JRuby
Simple RPC: msgpack-rpc-over-http
Simple RPC Server: mizuno (jetty + rack)
Simple Client Library: norikra-client
Just same code for cruby/jruby
13年6月29日土曜日

11. Norikra
Norikra Server (on JVM)
Esper Instance (Query Engine)
Type Definition
Manager
Output Event
Pool
Norikra Engine
RPC Server
mizuno (Jetty + Rack)
Rack RPC Handler
Norikra
Client
Norikra
Client
JRUBY
CRUBY
msgpack-rpc-over-http
13年6月29日土曜日

12. Norikra Query: target "sales"
goods_id:5 price:49.8 num:1 shop:"LINE"
goods_id:2 price:12.5 num:3 shop:"Cookpad"
goods_id:4 price:36.6 num:10 shop:"Cookpad"
SELECT shop, sum(price*num) AS amount
FROM sales.win:time_batch(10 minutes)
GROUP BY shop
goods_id:5 price:49.8 num:1 shop:"LINE"
goods_id:2 price:12.5 num:3 shop:"Cookpad" affiliate:"BiS"
SELECT affiliate, count(*) AS cnt
FROM sales.win:time_batch(1 hour)
GROUP BY affiliate
13年6月29日土曜日

13. Esper and Norikra
Esper:
queries for streams
stream: a set of field-type pairs of events
users need to know for field set variations
(or manage 'map subtypes' on your own)
Norikra:
queries for targets
target: virtual name of union of field set variation
users don't need to know for detail of target
13年6月29日土曜日

14. automated stream inheritance
of norikra's target
Base typedef
Query typedef
Data typedef
b_xxxxxxxxx
minimal fieldset definition:
name: 'string'
id: 'long'
valid: 'boolean'
action_type: 'string'
13年6月29日土曜日

15. automated stream inheritance
of norikra's target
Base typedef
Query typedef
Data typedef
b_xxxxxxxxx
event data fieldset
definition:
name: 'string'
id: 'long'
valid: 'boolean'
action_type: 'string'
product_code: 'string'
charge: 'integer'
shop_code: 'long'e_xxxxxxxx1
13年6月29日土曜日

16. automated stream inheritance
of norikra's target
Base typedef
Query typedef
Data typedef
b_xxxxxxxxx
e_xxxxxxxx1 e_xxxxxxxx2
event data fieldset definition:
name: 'string'
id: 'long'
valid: 'boolean'
action_type: 'string'
product_code: 'string'
charge: 'integer'
shop_code: 'long'
affiliate: 'string'
13年6月29日土曜日

17. automated stream inheritance
of norikra's target
Base typedef
Query typedef
Data typedef
b_xxxxxxxxx
e_xxxxxxxx1 e_xxxxxxxx2
new query:
SELECT count(*)
FROM target.win:time_batch(1min)
WHERE affiliate.length() > 0
13年6月29日土曜日

18. automated stream inheritance
of norikra's target
Base typedef
Query typedef
Data typedef
b_xxxxxxxxx
e_xxxxxxxx1 e_xxxxxxxx2'
event data fieldset
definition:
name: 'string'
id: 'long'
valid: 'boolean'
action_type: 'string'
affiliate: 'string'
q_xxxxxxxx0
new query:
SELECT count(*)
FROM target.win:time_batch(1min)
WHERE affiliate.length() > 0
13年6月29日土曜日

19. automated stream inheritance
of norikra's target
Base typedef
Query typedef
Data typedef
b_xxxxxxxxx
e_xxxxxxxx1 e_xxxxxxxx2'
q_xxxxxxxx0
Registered EPL:
SELECT count(*)
FROM q_xxxxxxxx0.win:time_batch(1min)
WHERE affiliate.length() > 0
13年6月29日土曜日

20. automated stream inheritance
of norikra's target
Base typedef
Query typedef
Data typedef
b_xxxxxxxxx
e_xxxxxxxx1' e_xxxxxxxx2'
q_xxxxxxxx0
e_xxxxxxxx3'
q_xxxxxxxx1
13年6月29日土曜日

21. Output data pooling
Output event data: pushed
Event pushing brings many problems
Pooling + fetch
typical usecase: aggregation
-> not so many outputs
13年6月29日土曜日

22. fluent-plugin-norikra
Fluentd plugin to use Norikra
Norikra server autostart
Automatically defined target
Pre-defined queries for each targets
13年6月29日土曜日

23. fluent-plugin-norikra
installation
`gem install fluent-plugin-norikra`
configuration
see DEMO
13年6月29日土曜日

24. Demo: bootstrap
rbenv shell jruby-1.7.4
gem install norikra
which norikra
rbenv shell 2.0.0-pxxx
gem install fluent-plugin-norikra
vi demo.conf
fluentd -c demo.conf
13年6月29日土曜日

25. Demo: query streams
some messages over fluent-cat
register queries with norikra-client
more messages over fluent-cat & norikra-client
13年6月29日土曜日

26. roadmap of norikra
Norikra is still UNDER DEVELOPMENT
Norikra feature updates (JOINs, etc)
Web GUI
query & target list management
save & restore metadata
Distributed & orchestrated nodes
13年6月29日土曜日

27. See also:
http://fluentd.org/
http://fluentd.org/plugin/
https://github.com/tagomoris/norikra
https://github.com/tagomoris/norikra-client
https://github.com/tagomoris/fluent-plugin-norikra
http://esper.codehaus.org/
"Fluentd: The ruby based middleware across the world"
http://www.slideshare.net/tagomoris/fluentd-in-tkrk10
"Log analysis system with Hadoop in livedoor 2013 Winter"
http://www.slideshare.net/tagomoris/log-analysis-with-hadoop-in-livedoor-2013
13年6月29日土曜日