Building a Central Nervous System for Data with Apache Kafka®, Anatoly Zelenin, IT Consultant https://www.meetup.com/Apache-Kafka-Germany-Munich/events/281819704/

1. Building a
Central Nervous System
for Data
with Apache Kafka

2. "Dieses Foto" von Unbekannter Autor ist lizenziert gemäß CC BY-ND

3. I want it ALL
and I want it NOW

5. Data at REST

6. Backend
Frontend
DB

7. Backend
Frontend
DB

8. Backend
Frontend
DB

9. Backend
Frontend
SAP Connector
External Services
DB

11. We have build a
distributed Monolith

12. We treated
„Inside Data“
as
„Outside Data“

14. Inside Data Outside Data
"Dieses Foto" von Unbekannter Autor ist lizenziert gemäß CC BY-ND

15. Inside Data
• Make the most out of the data
• Very specific to a certain Service
• Tightly coupled
Outside Data
"Dieses Foto" von Unbekannter Autor ist lizenziert gemäß CC BY-ND

16. "Dieses Foto" von Unbekannter Autor ist lizenziert gemäß CC BY-ND
Inside Data
• Make the most out of the data
• Very specific to a certain Service
• Tightly coupled
• Best for moving data
• Not specific to a certain Service
• Loosely coupled
Outside Data

17. Databases:
Perfect for
Inside Data

18. Databases:
Current State of
MY World

19. What about
Outside Data?

20. Meet Logs
"Dieses Foto" von Unbekannter Autor ist lizenziert gemäß CC BY-SA

21. A Log
[a]

22. A Log
[a] Log.append(a) -> void
Log.readFromOffset(int) -> [a]

23. A Log
H
0

24. A Log
H
E
Appending
0

25. A Log
H E
L
Appending
0 1

26. A Log
H E L
L
Appending
0 1 2

27. A Log
H E L L
O
Appending
0 1 2 3

28. A Log
H E L L O
0 1 2 3 4

29. A Log
H E L L O
Reading
0 1 2 3 4

30. Producer Producer

31. Producer Producer
Service 2 Service 1
Service 3

32. DB L O G S ?
vs

33. DB L O G S

34. Change Data
Capture
Inside Data
Outside Data

35. Aggregate
Outside Data
Inside Data

36. What about
Kafka?

37. Kafka
is a
Log

38. Kafka
is a
distributed
Log

39. Producer
Producer
Producer
🤯

40. Producer
Producer
Producer
Partition 1
Partition 2
Partition 3

41. Producer
Producer
Producer
Partition 1
Partition 2
Partition 3
Topic

42. Producer
Producer
Producer
Partition 1
Partition 2
Partition 3
Consumer
Consumer
Consumer
(*) Consumers in one
Consumer Group

43. Kafka
3 Tier App

44. Kafka
Mobile App
3 Tier App

45. Kafka
Mobile App
3 Tier App
Kafka
Streams

46. Kafka
Mobile App
3 Tier App
Kafka
Streams
Analytics

47. Kafka
Mobile App
3 Tier App
Kafka
Streams
Analytics
External APIs

48. Kafka
Mobile App
3 Tier App
Kafka
Connect
Databases
Files
Message
Queues
Kafka
Streams
Analytics
External APIs

49. Kafka
Mobile App
3 Tier App
Kafka
Connect
Kafka
Connect
Cloud
FTP
Databases
Databases
Files
Message
Queues
Kafka
Streams
Analytics
External APIs

50. Kafka
SAP
Mobile App
3 Tier App
Kafka
Connect
Kafka
Connect
Cloud
FTP
Databases
Databases
Files
Message
Queues
Kafka
Streams
Analytics
External APIs

52. Payload
R
e
l
i
a
b
i
l
t
y
P
e
r
f
o
r
m
a
n
c
e
Foundations

53. Hot off the Press!
Win a copy today
Stay up-to-date:
streamcommit.de