Serverless brings with it a groundswell of interest in event-driven architectures. Many advocate FaaS as a natural fit for stream processing and event-driven design, but is that just marketing speak? In this talk, I will dispel the myths behind serverless and show how it can form part of a data flow, event-driven streaming architecture that could power ‘eBay’. I dive into how the data flow model works with streams and stream processors interacting to provide a network of related data and essentially Kafka becomes a deconstructed database. I then elaborate on FaaS, the challenges, and uses and how it is becoming an essential part of being a cloud-native story that fits with Streaming architectures. This is all tied together by walking through data flow functionality like item-placement, item-bidding, item-search, auction analytics and where FaaS fits into these!

1. Serverless and Streaming

2. - Building the auction platform
- Event driven architectures on Kafka
- FaaS for streaming
- Putting it in action

4. ●
●
●
●
●
users
the world
selling
buying
the sketch

5. ●
●
●
●
●

6. Auction items
Items being placed
1.
Bids against item ‘duran duran’
stream-table
join against ‘duran duran’
2.
Item ‘duran duran’
3.
CDC Stream
{ item:duran res: 100 bid: 125 buyer: michael }
{ item:duran res: 100 bid: 120 buyer: nick }
{ item:duran res: 100 bid: 121 buyer: damian }
{ item:duran res: 100 bid: 115 buyer: andy }
{ item:duran res: 100 bid: 110 buyer: andy }
KSQL: SELECT * from bidding-stream
where item=’duran duran’
4.

7. {...}
{...}
{...}
FaaSKafka cluster
stream
processing
Kafka Streams
KSQL
Producer/Consumer

10. SOMETHING
HAPPENED!

12. BEING EVENT-FIRST CHANGES HOW YOU
THINK ABOUT WHAT YOU ARE BUILDING
...more importantly...

13. 1. Any (data) application can be a data flow
2. A series of events and reacting to those events
3. Kafka then becomes a database
● User signup
● Item bid
● Item sold
● Notify user
Neil bid: 100
Michael bid: 200
Ben bid: 200
>> SOLD!
store all events
replay all events
transactional
etc

14. Databases Databases
Customer
Data Updates
Unified 360
Merged Customer
Profiles
“Who bought what”
Events

15. Shoulders of Streaming Giants
subscribe(), poll(), send(),
flush(), beginTransaction(), …
KStream, KTable, filter(),
map(), flatMap(), join(),
aggregate(), transform(), …
CREATE STREAM, CREATE TABLE,
SELECT, JOIN, GROUP BY, SUM, …
KSQL UDFs
Ease of Use
Flexibility

16. Stream
processor
STREAM
/user-reg
FILTER
SELECT users > 18
PROJECT
SELECT user.name
JOIN
SELECT u.name, a.country
from user-reg u JOIN
address a WHERE u.id = a.id
GROUP BY (TABLE)
SELECT u.country,
count(u.name) FROM user-reg u
GROUP BY u.country
WINDOW (TABLE)
SELECT u.country, count(u.name)
FROM user-reg u WINDOW TUMBLING
(SIZE 1 MIN) group by u.country
STREAM
/address

17. 5.
/bidding-history
{ item:duran res: 100 bid: 125 buyer: michael }
{ item:duran res: 100 bid: 120 buyer: nick }
{ item:duran res: 100 bid: 121 buyer: damian }
{ item:duran res: 100 bid: 115 buyer: andy }
{ item:duran res: 100 bid: 110 buyer: andy }
KSQL: SELECT * from
bidding-stream where
item=’duran duran’
6.
Auction items
3.
KSQL Table
Table
Bidding Status
stream-table
join
4.
Table
Kafka Streams - KTable
Kafka Topic
/bids (stream)
2.
Kafka Topic
/items (stream)
1.

18. SELECT u.country, count(u.name) FROM user-reg u
WINDOW TUMBLING (SIZE 1 MIN) GROUP BY u.country

19. KStream<String, String> userBids = builder.stream("stream-user-bids");
final KTable<String, Long> bidCount = userBids
.flatMapValues(value ->
Arrays.asList(pattern.split(value.toLowerCase()))
)
.groupBy((user, count) -> user)
.count();
bidCounts.toStream().to("streams-bid-count-output",
Produced.with(stringSerde, longSerde));

20. Model events not commands,
-> streams of events,
-> series of streams to model the domain
{
item-id: 389*
user: 100
type: bid
cat: bikes/mtb
region: dc-east
}
/bikes/mtb by item-id*
/bikes/ by dc-east‘-’item-id*
-
-
-
-
-
-
-

21. {
user: 100
type: bid
item: 389
cat: bikes/mtb
region: dc-east
}
/bikes/mtb by item-id
key#
partition
topic
key space
Low level analytics
counter
Producer/Consumer
Item Summary
Kafka Streams
User Activity
KSQL

22. /stream-1
/stream-2
join
Stream
processor

23. 1. Ops:
2. Stream-based:
3. Data-based:
→

24. the log, event sourcing, source of truth,
CQRS, event collaboration, replayability,
at-least once, exactly one, evolutionary
architectures, data-virtualization and
more

25. {...}

26. ●
●
●
●
● {...}

27. 1.
stateless
2. Non-stream non-time
3. Analytics
4. Edge
5. incoming
outgoing
{...}

28. ●
●
●
●
●
{...}

29. {...}
{...}
{...}
the log the log
Event sourcing:
How did we get here?

30. {...}
{...}
{...}
{...}
bad stream

31. - Enrich users on signup (address validation)
- Geo-enrich items on placement (city, state, lat-lon cell identification)
- Notify users on item sold or reserve not met
- Perform analytics on auction when item ‘completes’
- Notify user of items-of-interest from their history when browsing
- Ad-placement analytics (watched items, interested items, users-purchased
- Monte-carlo auction simulations to guide users and calc item trending scores

34. ● Search
● Bid
● Item-Complete
● Marketplace analytics

35. Auction items
Items being placed
1.
Bids against item ‘duran duran’
stream-table
join against ‘duran duran’
2.
Item ‘duran duran’
3.
/bidding-stream
{ item:duran res: 100 bid: 125 buyer: michael }
{ item:duran res: 100 bid: 120 buyer: nick }
{ item:duran res: 100 bid: 121 buyer: damian }
{ item:duran res: 100 bid: 115 buyer: andy }
{ item:duran res: 100 bid: 110 buyer: andy }
KSQL: SELECT * from bidding-stream
where item=’duran duran’
4.

36. 4. KStream: Interactive query
3. Table: Materialized view
topic: /auction/records/80sSearch: Identify set of topics
1.
2. Locate KTables

37. 1. Stream processor runs ‘future’ on
local-state of item-bids
auction items
3. Rejoined
removes item
4. Stream triggers FaaS
complete item processor
bid-history
Stream table join
5. FaaS Notify all bidders
bid-notifications
2.
Rewrite ‘completed’ item status
to retrigger join
item status

38. ● How is the item trending? Banding on condition (new, as-new, used)
● What’s the usual bid-offer spread?
● Frequency of sale?
● Percentiles?
{...}
{...}
{...}
{ bid:100; sold:1000;}
/bid-history
{
stream-lib.TDigest(values[])
}
{
stream-lib.TDigest(values[])
}
{
stream-lib.TDigest(values[])
}
Calculating percentiles using the ‘unit-of-work’ pattern
{
digest.merge(digest)
}
{...}

39. /auction/items
{...}
{ item:100;}
Stream
processor
1. Items
{ bid:100;}
{...} Stream
processor
/auction/bids
2. Bidding
{...}
notify bidders & seller
{ item:100; offer:101}
/auction/bids/history
{...}
analytics for bidders
3. Processing

41. ● Event-First
● Streams →
→
● App Infra
● FaaS

42. Event-first forces you to think about behaviour of the system
Event sourcing captures that behaviour

44. ● Stream processing:
● Streaming platform:
● Closer affinity:
● FaaS

45. Serverless and stream processing