Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/39NIjLV. Akhilesh Gupta does a technical deep-dive into how Linkedin uses the Play/Akka Framework and a scalable distributed system to enable live interactions like likes/comments at massive scale at extremely low costs across multiple data centers. Filmed at qconlondon.com. Akhilesh Gupta is the technical lead for LinkedIn's Real-time delivery infrastructure and LinkedIn Messaging. He has been working on the revamp of LinkedIn’s offerings to instant, real-time experiences. Before this, he was the head of engineering for the Ride Experience program at Uber Technologies in San Francisco.

1. Streaming a million likes/second
Real-time interactions on Live Video
Akhilesh Gupta
Realtime Engineer, LinkedIn
2

2. InfoQ.com: News & Community Site
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
linkedin-play-akka-distributed-
systems/
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon London
www.qconlondon.com

4. tiny.cc/qconlive
3
Login with LinkedIn
credentials

5. Which was the largest live stream in the world?
(largest number of concurrent viewers)
?
4

6. INDIA VS NEW ZEALAND WORLD CUP
Largest Live Stream
>25.3M
viewers
© The Indian Express https://images.indianexpress.com/2019/06/13th-june_india-vs-new-zealand_759.jpg
5

7. 6

8. D i s t r i b u t e d S y s t e m s
Start small and add simple layers to your
architecture
7

9. 8

10. What is a Live Video?
9

11. What are real-time interactions
on Live Video?
10

12. Mobile
LINKEDIN LIVE
Likes/Comments
Concurrent Viewer Counts
11

13. LINKEDIN LIVE
Desktop
Likes/Comments
Concurrent Viewer Counts
12

14. Distribution of Likes
A
S
13

15. Distribution of Likes
A
S
Simple HTTP Request
14

16. Distribution of Likes
A
S
Simple HTTP Request
?
15

17. Challenge 1:The delivery pipe
16

18. The delivery pipe
Real-time
Delivery
A
S
Likes Backend (Publisher) 17

19. The delivery pipe
Real-time
Delivery
A
S
Likes Backend (Publisher)
Simple HTTP Request
18

20. The delivery pipe
Real-time
Delivery
A
S
Likes Backend (Publisher)
Persistent Connection
19

21. The delivery pipe
Real-time
Delivery
A
S
Likes Backend (Publisher)
Persistent Connection
20

22. Persistent Connection
Real-time
Delivery
21

23. TRY IT!
Login at linkedin.com
Persistent Connection
tiny.cc/realtime
22

24. 23

25. EVENTSOURCE INTERFACE CLIENT REQUEST
HTTP Long Poll with Server Sent Events
GET /connect HTTP/1.1
Accept: text/event-stream
24

26. EVENTSOURCE INTERFACE SERVER RESPONSE
HTTP Long Poll with Server Sent Events
HTTP/1.1 200 OK
- RESPONSE HEADERS -
Content-Type: text/event-stream
25

27. EVENTSOURCE INTERFACE SERVER RESPONSE
HTTP Long Poll with Server Sent Events
HTTP/1.1 200 OK
- RESPONSE HEADERS -
Content-Type: text/event-stream
- RESPONSE BODY -
data: {“like” object}
26

28. EVENTSOURCE INTERFACE SERVER RESPONSE
HTTP Long Poll with Server Sent Events
HTTP/1.1 200 OK
- RESPONSE HEADERS -
Content-Type: text/event-stream
- RESPONSE BODY -
data: {“like” object}
..
data: {“comment” object}
27

29. EVENTSOURCE INTERFACE SERVER RESPONSE
HTTP Long Poll with Server Sent Events
HTTP/1.1 200 OK
- RESPONSE HEADERS -
Content-Type: text/event-stream
- RESPONSE BODY -
data: {“like” object}
..
data: {“comment” object}
..
data: {“like” object}
28

30. WEB
EventSource Client Libraries
var evtSource =
new EventSource("https://www.linkedin.com/realtime/connect");
evtSource.onmessage = function(e) {
var likeObject = JSON.parse(e.data);
}
29

31. ANDROID & IOS SUPPORT
EventSource Client Libraries
Android: https://github.com/tylerjroach/eventsource-android
iOS Swift: https://github.com/inaka/EventSource
30

32. Next Challenge?
Real-time
Delivery
31

33. Next Challenge: Multiple Connections
Real-time
Delivery
32

34. Challenge 2: Connection Management
33

35. PLAY & AKKA
Connection Management
Akka is a toolkit for building highly concurrent,
distributed, and resilient message-driven applications
34

36. © Disney, All Rights Reserved, Pirates of the Caribbean
35

37. AN AKKA ACTOR
Connection Management
State
Behavior
36

38. AN AKKA ACTOR
Connection Management
State
Behavior
ABC
Mailbox
One per Actor
37

39. AN AKKA ACTOR
Connection Management
State
Behavior
ABC
Mailbox
One per Actor
38

40. AN AKKA ACTOR
Connection Management
State
Behavior
ABC
Mailbox
One per Actor
Millions of Actors
39

41. AN AKKA ACTOR
Connection Management
State
Behavior
ABC
Mailbox
One per Actor
No. of threads proportional to
no. of cores
Millions of Actors
40

42. AN AKKA ACTOR
Connection Management
Connection
Publish Event
ABC
Events to be published
Millions of Connections
41

43. INITIALIZE AN AKKA ACTOR
Server: Accepting a EventSource connection
// Client A connects to the server and is assigned connectionIdA
public Result listen() {
} 42

44. INITIALIZE AN AKKA ACTOR
Server: Accepting a EventSource connection
// Client A connects to the server and is assigned connectionIdA
public Result listen() {
return ok(EventSource.whenConnected(eventSource -> {
String connectionId = UUID.randomUUID().toString();
}));
} 43

45. INITIALIZE AN AKKA ACTOR
Server: Accepting a EventSource connection
// Client A connects to the server and is assigned connectionIdA
public Result listen() {
return ok(EventSource.whenConnected(eventSource -> {
String connectionId = UUID.randomUUID().toString();
// construct an Akka Actor to manage connection
_actorSystem.actorOf(
ClientConnectionActor.props(connectionId, eventSource),
connectionId
);
}));
} 44

46. Publish Events Using Akka Actors
Akka
Actor
Real-time
Delivery
System
Connection ID
cid1
45

47. EVENT PUBLISH FROM PUBLISHER
Publish Events Using Akka Actors
Actor 1
Real-time
Delivery
System
Actor 2
Actor 3
Actor 4
Actor 5
Akka
Supervisor
Actor
cid1
cid2
cid3
cid4
cid5
46

48. EVENT PUBLISH FROM SUPERVISOR TO CHILD ACTORS
Publish Events Using Akka Actors
Actor 1
Real-time
Delivery
System
Actor 2
Actor 3
Actor 4
Actor 5
Akka
Supervisor
Actor
cid1
cid2
cid3
cid4
cid5
47

49. EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS
Publish Events Using Akka Actors
Actor 1
Real-time
Delivery
System
Actor 2
Actor 3
Actor 4
Actor 5
Akka
Supervisor
Actor
cid1
cid2
cid3
cid4
cid5
48

50. EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS
Publishing on EventSource
eventSource.send(<like object>);
49

51. EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS
Publish Events Using Akka Actors
Actor 1
Real-time
Delivery
System
Actor 2
Actor 3
Actor 4
Actor 5
Akka
Supervisor
Actor
cid1
cid2
cid3
cid4
cid5
50

52. EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS
Publish Events Using Akka Actors
Actor 1
Real-time
Delivery
System
Actor 2
Actor 3
Actor 4
Actor 5
Akka
Supervisor
Actor
cid1
cid2
cid3
cid4
cid5
51

53. EVENTSOURCE INTERFACE
Received Events on the Clients
HTTP/1.1 200 OK
- RESPONSE HEADERS -
Content-Type: text/event-stream
- RESPONSE BODY -
data: {“User A liked the video…” object}
52

54. Next Challenge?
Real-time
Delivery
53

55. Next Challenge: Multiple Live Videos
Real-time
Delivery
54

56. Challenge 3:Multiple Live Videos
55

57. START WATCHING LIVE VIDEO
Multiple Live Videos
cid3
cid5
Live Video 1
Live Video 2
Real-time
Delivery
System
56

58. HTTP SUBSCRIPTION REQUEST
Viewers subscribe to Live Videos
Topic ConnectionId
In-Memory
Subscriptions
Table
cid3
cid5
Live Video 1
Live Video 2
/subscribe
57

59. IN-MEMORY SUBSCRIPTION ENTRY
Viewers subscribe to Live Videos
Topic ConnectionId
In-Memory
Subscriptions
Table
cid5
Live Video 2
/subscribe
/subscribe
cid3Live Video 1
58

60. IN-MEMORY SUBSCRIPTION ENTRY
Viewers subscribe to Live Videos
Topic ConnectionId
In-Memory
Subscriptions
Table
cid3
cid5
Live Video 1
Live Video 2
/subscribe
/subscribe
59

61. Publish using Subscription
Actor 1
Actor 2
Actor 3
Actor 4
Actor 5
Supervisor
cid1
cid2
cid3
cid4
cid5
Live Video 1 cid3
Live Video 2 cid5
Live Video 2 cid1
Live Video 1 cid4
Live Video 2 cid2
60

62. PUBLISHER PUBLISH TO SUPERVISOR ACTOR
Publish using Subscription
Actor 1
Actor 2
Actor 3
Actor 4
Actor 5
Supervisor
cid1
cid2
cid3
cid4
cid5
Live Video 1 cid3
Live Video 2 cid5
Live Video 2 cid1
Live Video 1 cid4
Live Video 2 cid2
61

63. SUBSCRIBER LOOKUP FROM IN MEMORY TABLE
Publish using Subscription
Actor 1
Actor 2
Actor 3
Actor 4
Actor 5
Supervisor
cid1
cid2
cid3
cid4
cid5
Live Video 1 cid3
Live Video 2 cid5
Live Video 2 cid1
Live Video 1 cid4
Live Video 2 cid2
62

64. PUBLISH TO CLIENT CONNECTIONS
Publish using Subscription
Actor 1
Actor 2
Actor 3
Actor 4
Actor 5
Supervisor
cid1
cid2
cid3
cid4
cid5
Live Video 1 cid3
Live Video 2 cid5
Live Video 2 cid1
Live Video 1 cid4
Live Video 2 cid2
63

65. PUBLISHER PUBLISH TO SUPERVISOR ACTOR
Publish using Subscription
Actor 1
Actor 2
Actor 3
Actor 4
Actor 5
Supervisor
cid1
cid2
cid3
cid4
cid5
Live Video 1 cid3
Live Video 2 cid5
Live Video 2 cid1
Live Video 1 cid4
Live Video 2 cid2
64

66. SUBSCRIBER LOOKUP FROM IN MEMORY TABLE
Publish using Subscription
Actor 1
Actor 2
Actor 3
Actor 4
Actor 5
Supervisor
cid1
cid2
cid3
cid4
cid5
Live Video 1 cid3
Live Video 2 cid5
Live Video 2 cid1
Live Video 1 cid4
Live Video 2 cid2
65

67. PUBLISH TO CLIENT CONNECTIONS
Publish using Subscription
Actor 1
Actor 2
Actor 3
Actor 4
Actor 5
Supervisor
cid1
cid2
cid3
cid4
cid5
Live Video 1 cid3
Live Video 2 cid5
Live Video 2 cid1
Live Video 1 cid4
Live Video 2 cid2
66

68. Next Challenge?
Real-time
Delivery
67

69. Next Challenge: 10K Concurrent Viewers
Real-time
Delivery
68

70. Challenge 4:10K Concurrent Viewers
69

71. 70

72. 10K Concurrent Viewers
Real-time
Delivery
71

73. S C A L I N G B A C K E N D S Y S T E M S
When in doubt, add a machine!
72

74. 10K Concurrent Viewers
Real-time
Delivery
73

75. 10K Concurrent Viewers
Frontend
Frontend
74

76. 10K Concurrent Viewers
Frontend
Frontend
75

77. DISPATCHES BETWEEN FRONTEND NODES
Real-time Dispatcher
Frontend
Frontend
Real-time Dispatcher
76

78. DISPATCHES BETWEEN FRONTEND NODES
Real-time Dispatcher
Frontend
Frontend
Real-time Dispatcher
77

79. DISPATCHES BETWEEN FRONTEND NODES
Real-time Dispatcher
Frontend
Frontend
Real-time Dispatcher
78

80. Frontend Node Subscriptions
node1
node2
Live Video 1
Live Video 2
Frontend
Frontend
Real-time Dispatcher
79
Live Video 1

81. HTTP SUBSCRIPTION REQUEST
Frontend Node Subscriptions
node1
node2
Live Video 1
Live Video 2
Frontend
Frontend
Topic
Frontend
Host
Real-time Dispatcher
Subscriptions
Table
/subscribe
80
Live Video 1

82. SUBSCRIPTIONS TABLE ENTRY
Frontend Node Subscriptions
node1
node2
Live Video 1
Live Video 2
Frontend
Frontend
Topic
Frontend
Host
Real-time Dispatcher
Subscriptions
Table
/subscribe
/subscribe
81
Live Video 1

83. SUBSCRIPTIONS TABLE ENTRY
Frontend Node Subscriptions
node1
node2
Live Video 1
Live Video 2
Frontend
Frontend
Topic
Frontend
Host
Real-time Dispatcher
Subscriptions
Table
/subscribe
/subscribe
82
node2
Live Video 1

84. SUBSCRIPTIONS TABLE ENTRY
Frontend Node Subscriptions
node1
node2
Live Video 1
Live Video 2
Frontend
Frontend
Topic
Frontend
Host
Real-time Dispatcher
Subscriptions
Table
/subscribe
/subscribe
83
node2
Live Video 1

85. Publish to Frontend Nodes
node1
Real-time Dispatcher
Frontend
node2 Frontend
node3 Frontend
node4 Frontend
node5 Frontend
Live Video 1 node1
Live Video 2 node5
Live Video 2 node3
Live Video 1 node2
Live Video 2 node2
84

86. PUBLISHER PUBLISH TO DISPATCHER
Publish to Frontend Nodes
node1
Real-time Dispatcher
Frontend
node2 Frontend
node3 Frontend
node4 Frontend
node5 Frontend
Dispatcher
85
Live Video 1 node1
Live Video 2 node5
Live Video 2 node3
Live Video 1 node2
Live Video 2 node2

87. SUBSCRIBER LOOKUP
Publish to Frontend Nodes
node1
Real-time Dispatcher
Frontend
node2 Frontend
node3 Frontend
node4 Frontend
node5 Frontend
Dispatcher
86
Live Video 1 node1
Live Video 2 node5
Live Video 2 node3
Live Video 1 node2
Live Video 2 node2

88. PUBLISH TO SUBSCRIBED FRONTEND NODES
Publish to Frontend Nodes
node1
Real-time Dispatcher
Frontend
node2 Frontend
node3 Frontend
node4 Frontend
node5 Frontend
Dispatcher
87
Live Video 1 node1
Live Video 2 node5
Live Video 2 node3
Live Video 1 node2
Live Video 2 node2

89. Frontend Node to Client Publish
Frontend
88

90. Frontend Node to Client Publish
Frontend
89

91. What’s the next challenge?
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Live Video
1
node3
Live Video
2
node5
Live Video
2
node1
Live Video
1
node4
Live Video
2
node2
90

92. Challenge 5:100 Likes/second
91

93. 100 Likes/second
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Live Video
1
node3
Live Video
2
node5
Live Video
2
node1
Live Video
1
node4
Live Video
2
node2
92

94. S C A L I N G B A C K E N D S Y S T E M S
When in doubt, add a machine!
93

95. 100 Likes/second
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Live Video
1
node3
Live Video
2
node5
Live Video
2
node1
Live Video
1
node4
Live Video
2
node2
94

96. 100 Likes/second
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Live Video
1
node7
Live Video
2
node5
Live Video
2
node1
Live Video
1
node4
Live Video
2
node2
Dispatcher
Frontend
Frontend
95

97. 100 Likes/second
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Live Video
1
node7
Live Video
2
node5
Live Video
2
node1
Live Video
1
node4
Live Video
2
node2
Dispatcher
Frontend
Frontend
96

98. 100 Likes/second
Frontend
Frontend
Frontend
Frontend
Frontend
Live Video
1
node7
Live Video
2
node5
Live Video
2
node1
Live Video
1
node4
Live Video
2
node2
Dispatcher
Frontend
Frontend
97

99. COUCHBASE/REDIS ETC.
Key-Value Store for Subscriptions
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Dispatcher
Frontend
Frontend
Live Video 1 node7
Live Video 2 node5
Live Video 2 node1
Live Video 1 node4
Live Video 2 node2
Key-Value Store
Dispatcher
98

100. Publish to Dispatcher Nodes
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Dispatcher
Frontend
Frontend
Key-Value Store
Dispatcher
Live Video 1 node7
Live Video 2 node5
Live Video 2 node1
Live Video 1 node4
Live Video 2 node2
99

101. Subscriber lookup from KV Store
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Dispatcher
Frontend
Frontend
Key-Value Store
Dispatcher
Live Video 1 node7
Live Video 2 node5
Live Video 2 node1
Live Video 1 node4
Live Video 2 node2
node1
node2
node3
node4
node5
node6
node7
100

102. Publish to Frontend Nodes
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Dispatcher
Frontend
Frontend
Key-Value Store
Dispatcher
Live Video 1 node7
Live Video 2 node5
Live Video 2 node1
Live Video 1 node4
Live Video 2 node2
node1
node2
node3
node4
node5
node6
node7
101

103. Publish to Frontend Nodes
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Dispatcher
Frontend
Frontend
Key-Value Store
Dispatcher
node1
node2
node3
node4
node5
node6
node7
Live Video 1 node7
Live Video 2 node5
Live Video 2 node1
Live Video 1 node4
Live Video 2 node2
102

104. Publish to Frontend Nodes
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Dispatcher
Frontend
Frontend
Key-Value Store
Dispatcher
node1
node2
node3
node4
node5
node6
node7
Live Video 1 node7
Live Video 2 node5
Live Video 2 node1
Live Video 1 node4
Live Video 2 node2
103

105. Challenge 6: 100 Likes/s, 10K Viewers
Distribution of 1M Likes/s
104

106. Mobile
LINKEDIN LIVE
Likes
105

107. The Subscription Flow
Frontend Dispatcher
Key-Value Store
1
106

108. START WATCHING LIVE VIDEO
Viewers subscribing to Live Video Likes
cid3
cid5
Live Video 1
Live Video 2
Frontend
107

109. HTTP SUBSCRIPTION REQUEST
Viewers subscribing to Live Video Likes
Topic ConnectionId
cid3
cid5
Live Video 1
Live Video 2
/subscribe
Frontend
In-Memory
Subscriptions Table
108

110. FRONTEND IN-MEMORY SUBSCRIPTION ENTRY
Viewers subscribing to Live Video Likes
Topic ConnectionId
cid5
Live Video 2
/subscribe
/subscribe
cid3Live Video 1
Frontend
In-Memory
Subscriptions Table
109

111. FRONTEND IN-MEMORY SUBSCRIPTION ENTRY
Viewers subscribing to Live Video Likes
Topic ConnectionId
cid3
cid5
Live Video 1
Live Video 2
/subscribe
/subscribe
Frontend
In-Memory
Subscriptions Table
110

112. FRONTEND IN-MEMORY SUBSCRIPTION ENTRY
Viewers subscribing to Live Video Likes
Topic ConnectionId
cid3
cid5
Live Video 1
Live Video 2
/subscribe
/subscribe
Frontend
In-Memory
Subscriptions Table
111

113. The Subscription Flow
Frontend Dispatcher
Key-Value Store
1 2 3
112

114. FRONTEND RECEIVES SUBSCRIPTION REQUEST
Frontend Node Subscriptions in Dispatcher
node1
node2
Live Video 1
Live Video 2
Frontend
Frontend
Key-Value Store
Dispatcher
Dispatcher
113

115. FRONTEND HTTP SUBSCRIPTION REQUEST TO DISPATCHER
Frontend Node Subscriptions in Dispatcher
node1
node2
Live Video 1
Live Video 2
Frontend
Frontend
Topic
Frontend
Host
/subscribe
Key-Value Store
Dispatcher
Dispatcher
114

116. DISPATCHER SUBSCRIPTION ENTRY IN KV STORE
Frontend Node Subscriptions in Dispatcher
node1
node2
Live Video 1
Live Video 2
Frontend
Frontend
Topic
Frontend
Host
/subscribe
/subscribe
Dispatcher
Dispatcher
Key-Value Store
115

117. DISPATCHER SUBSCRIPTION ENTRY IN KV STORE
Frontend Node Subscriptions in Dispatcher
node1
node2
Live Video 1
Live Video 2
Frontend
Frontend
Topic
Frontend
Host
/subscribe
/subscribe
Dispatcher
Dispatcher
Key-Value Store
116

118. The Publish Flow
Frontend Dispatcher
Key-Value Store
1
117

119. Viewers liking Live Videos
Likes Backend (Publisher)
118

120. Likes Published to Backend
Likes Backend (Publisher)
119

121. The Publish Flow
Frontend Dispatcher
Key-Value Store
1
2
345
120

122. The Publish Flow
Frontend Dispatcher
Key-Value Store
2
345
121

123. BACKEND PUBLISH TO DISPATCHER NODES
Dispatcher Publish to Frontend Nodes
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Dispatcher
Frontend
Frontend
Key-Value Store
Dispatcher
Live Video 1 node3
Live Video 2 node5
Live Video 2 node1
Live Video 1 node4
Live Video 2 node2
122

124. SUBSCRIBER LOOKUP FROM KV STORE
Dispatcher Publish to Frontend Nodes
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Dispatcher
Frontend
Frontend
Key-Value Store
Dispatcher
Live Video 1 node7
Live Video 2 node5
Live Video 2 node1
Live Video 1 node4
Live Video 2 node2
node1
node2
node3
node4
node5
node6
node7
123

125. SUBSCRIBER LOOKUP FROM KV STORE
Dispatcher Publish to Frontend Nodes
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Dispatcher
Frontend
Frontend
Key-Value Store
Dispatcher
Live Video 1 node7
Live Video 2 node5
Live Video 2 node1
Live Video 1 node4
Live Video 2 node2
node1
node2
node3
node4
node5
node6
node7
124

126. PUBLISH TO FRONTEND NODES
Dispatcher Publish to Frontend Nodes
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
Dispatcher
Frontend
Frontend
Key-Value Store
Dispatcher
node1
node2
node3
node4
node5
node6
node7
Live Video 1 node7
Live Video 2 node5
Live Video 2 node1
Live Video 1 node4
Live Video 2 node2
125

127. The Publish Flow
Frontend Dispatcher
Key-Value Store
5
126

128. DISPATCHER PUBLISH TO FRONTEND NODE SUPERVISOR ACTOR
Frontend Publish to Mobile Clients
Actor 1
Actor 2
Actor 3
Actor 4
Actor 5
Supervisor
cid1
cid2
cid3
cid4
cid5
Live Video 1 cid3
Live Video 2 cid5
Live Video 2 cid1
Live Video 1 cid4
Live Video 2 cid2
Dispatcher
Frontend In-Memory
Subscriptions
Table
127

129. SUBSCRIBED CLIENT CONNECTION LOOKUP FROM IN-MEMORY TABLE
Frontend Publish to Mobile Clients
Actor 1
Frontend
Actor 2
Actor 3
Actor 4
Actor 5
Supervisor
cid1
cid2
cid3
cid4
cid5
Live Video 1 cid3
Live Video 2 cid5
Live Video 2 cid1
Live Video 1 cid4
Live Video 2 cid2
Dispatcher
In-Memory
Subscriptions
Table
128

130. PUBLISH TO MOBILE CLIENTS
Frontend Publish to Mobile Clients
Actor 1
Frontend
Actor 2
Actor 3
Actor 4
Actor 5
Supervisor
cid1
cid2
cid3
cid4
cid5
Live Video 1 cid3
Live Video 2 cid5
Live Video 2 cid1
Live Video 1 cid4
Live Video 2 cid2
Dispatcher
In-Memory
Subscriptions
Table
129

131. Bonus Challenge:
Another data center
130

132. 131

133. D i s t r i b u t e d S y s t e m s
Start small and add simple layers to your
architecture
132

134. Frontend DispatcherDC-1
133

135. Frontend Dispatcher
Frontend DispatcherDC-2
DC-1
134

136. Frontend Dispatcher
Frontend Dispatcher
Frontend Dispatcher
DC-3
DC-2
DC-1
135

137. Frontend Dispatcher
Frontend Dispatcher
Frontend Dispatcher
DC-3
DC-2
DC-1
136

138. Frontend Dispatcher
Frontend Dispatcher
Frontend Dispatcher
DC-3
DC-2
DC-1
137

139. Frontend Dispatcher
Frontend Dispatcher
Frontend Dispatcher
DC-3
DC-2
DC-1
138

140. Frontend Dispatcher
Frontend Dispatcher
Frontend Dispatcher
DC-3
DC-2
DC-1
CrossDataCenterPublish
139

141. Frontend Dispatcher
Frontend Dispatcher
Frontend Dispatcher
DC-3
DC-2
DC-1
140

142. Frontend Dispatcher
Frontend Dispatcher
Frontend Dispatcher
DC-3
DC-2
DC-1
141

143. Performance & Scale
142

144. How many persistent connections can a single
machine hold?
Frontend
Server
?
143

145. > 100,000
144

146. ROYAL WEDDING
Second Largest Stream
>18M
viewers
© time.com
145

147. > 100,000
We can hold those 18M connections with just
180 machines
146

148. LINKEDIN ENGINEERING BLOG
tiny.cc/
linkedinscaling
>100K
147

149. LinkedIn Realtime Performance
Persistent Connections/machine
5K/s
Dispatcher Publish QPS/machine
75ms
E2E Publish Latency
>100K
148

150. Dispatcher Performance
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
How many events per second can be published to a
single dispatcher machine?
?
events/second
149

151. Dispatcher Performance
Number of frontend nodes to publish to is limited
per event
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
?
events/second
150

152. Dispatcher Performance
Frontend
Frontend
Frontend
Frontend
Frontend
Dispatcher
?
events/second
151

153. 5K
We can publish 50K likes per second with just 10
machines
152

154. LinkedIn Realtime Performance
>100K
Persistent Connections/machine Dispatcher Publish QPS/machine
75ms
E2E Publish Latency
5K
153

155. E2E Publish Latency
Frontend Dispatcher
t1
Key-Value Store
154

156. E2E Publish Latency
Frontend Dispatcher
t1
Key-Value Store
t2
t2-t1 = 75ms p90
155

157. LinkedIn Realtime Performance
>100K
Persistent Connections/machine Dispatcher Publish QPS/machine
75ms
E2E Publish Latency
5K
156

158. LINKEDIN ENGINEERING BLOG
tiny.cc/
linkedinlatency
Measurement System
E2E Publish Latency
157

159. Why does this system scale?
Horizontally Scalable System
Frontend Dispatcher
Frontend
Frontend
Frontend
Dispatcher
Dispatcher
Dispatcher
Distributed K/V Storage System
158

160. Active Status/Presence
Online/Offline indicators
LINKEDIN REALTIME TECHNOLOGY
159

161. LINKEDIN ENGINEERING BLOG
tiny.cc/
linkedinpresence
Active Status/Presence
160

162. Key takeaways
161

163. R E A LT I M E I N T E R A C T I O N S
Enable dynamic instant experiences on your apps
162

164. E V E N T S O U R C E / S S E
Built-in support in most server frameworks
Readily available client libraries
HTTP/1.1 200 OK
- RESPONSE HEADERS -
Content-Type: text/event-stream
163

165. P l a y F r a m e w o r k a n d A k k a A c t o r s
Powerful frameworks to manage millions of
connections
State
BehaviorMailbox
164

166. D i s t r i b u t e d S y s t e m s
Start small and add simple layers to your
architecture
165

167. S C A L I N G B A C K E N D S Y S T E M S
When in doubt, add a machine!
166

168. R E A LT I M E I N T E R A C T I O N P L A T F O R M
Can be built on any server/storage technology
Horizontally scalable
Frontend Dispatcher
Key-
167

169. R E A LT I M E I N T E R A C T I O N P L A T F O R M
You can do it for your app!
168

170. R E A LT I M E E N G I N E E R , L I N K E D I N
@agupta03
t i n y . c c / q c o n 2 0 2 0
169
AMA at 1.40pm at Guild, Mezzanine

171. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
linkedin-play-akka-distributed-systems/