The document summarizes a presentation about using Kafka, Streamliner, MemSQL and ZoomData for real-time analytics visualization. It shows an initial setup with one producer and queue feeding into Kafka, then adding a sink to an in-memory SQL database and real-time visualization consumer. It asks questions about ensuring the system is resilient, handles bad data and schema evolution, maintains consistency across visualization layers, and ability to scale throughput, concurrency and size.

1. Anton Gorshkov
Real-Time Analytics Visualized
Kafka  Streamliner  MemSQL  ZoomData
Please note that during the course of this presentation ZoomData products will be used and shown on the
screen. Goldman Sachs has an ownership interest in ZoomData, Inc. and may have other business relationships
with ZoomData, Inc. Nothing herein shall constitute an offer to sell or a solicitation of an offer to buy an interest in
any entity or product.
Learn more at GS.com/Engineering

2. >docker run kafka
>docker run memsql
>docker run zoomdata
Initial Set-Up
2 4-CPU / 16GB / 80GB SSD / Intel Xeon E5-2670 @ 2.5GHzX

3. Context

4. Start with one producer & a queue
Producer 1 Kafka

5. Add-a-Sink
Producer 1 Kafka In-Memory
SQL RDB
Consumer

6. Add a Real-Time Visualization
Producer 1 Kafka In-Memory
SQL RDB
RT VisConsumer

7. Enterprise
Grade?

8. Resilience
- Don’t lose data
- Be Up
- Deliver (at least once and in-order)

9. Bad Data

10. Schema
Evolution

11. How “I” is the BI?
- Is it a “view” or a “do work” layer?
- Data-at-Rest vs Data-at-Motion
- Pull vs Push
- Consistency to other vis layers

12. Will It Scale?
Throughput, Concurrency, Size
(and at what cost…)

13. Will it Scale?
Producer 1
Kafka In-Memory
SQL RDB
RT VisProducer 2
Producer n
Consumer

14. Audience Participation Time…
Kafka In-Memory
SQL RDB
RT Vis
(620) 487-2222
Consumer

15. Audience Participation Time…
send a text [Fruit] [Quantity]
(620) 487-2222
example: mango 540

16. Adaptability
Producer 1
Kafka
In-Memory
SQL RDB
RT Vis
Producer 2
Producer n
Elastic
Kibana
Consumer
Kafka
Connect

17. Representative Deployment
Direct Sink
Spark
Streaming
IMRDBM
S
Data Lake
Kafka
Source
DBs
Order Mgmt
Batch ETL
Cache
VertX

18. Learn more at GS.com/Engineering