This document contains the agenda for a Pulsar Summit keynote on event-driven applications. The keynote will feature talks from Sijie Guo, Co-Founder and CEO of StreamNative, and Matteo Merli, CTO of StreamNative. Guo will discuss the growth of the Pulsar community and platform. Merli will cover the evolution of event-driven applications and the five fundamentals of modern event-driven architecture: data abstraction, API, primitives, processing semantics, and tools. The keynote aims to explain how Pulsar solves challenges in building complex event-driven applications.
Event-Driven Applications Done Right - Pulsar Summit SF 2022
1. Pulsar Summit
San Francisco
Hotel Nikko
August 18 2022
Keynote
Event-Driven
Applications
Done Right
Sijie Guo
Co-Founder & CEO • StreamNative
Matteo Merli
CTO • StreamNative
2. Sijie Guo
Co-Founder & CEO
StreamNative @sijieg @sijie
● Co-Founder & CEO, StreamNative
● PMC Member of Apache Pulsar
● PMC Chair of Apache BookKeeper
● Ex Co-Founder, Streamlio
● Ex Twitter, Yahoo!
3.
4. Bar Chart Race - Pull Requests
HUGE GROWTH IN THE COMMUNITY
5. Top-10 PRs in messaging and streaming
HUGE GROWTH IN THE COMMUNITY
6. Pulsar ranked as a Top 5 ASF project
HUGE GROWTH IN THE COMMUNITY
> 560+
Contributors
> 10,000+
Commits
> 7,000+
Slack Members
> 1,000+
Organizations
using Pulsar
7. Pulsar has already been deployed by thousands
of companies across the globe
HUGE GROWTH IN THE COMMUNITY
16. So if one service fails …
The Evolution of Microservices
x
17. …or even just runs slowly
The Evolution of Microservices
18. The fall-out could be much larger
The Evolution of Microservices
x
x
x
x
19. Others end up feeling that pain
The Evolution of Microservices
x
x
x
x
x
20. At company scale, the majority of processes
are asynchronous to one another
The Evolution of Microservices
Billing Inventory Fulfillment Fraud
21. So it makes sense to DECOUPLE services from
one another
The Evolution of Microservices
Billing Inventory Fulfillment Fraud
DECOUPLE
22. It evolves towards event-driven architecture
The Evolution of Microservices
Billing Inventory Fulfillment Fraud
MESSAGING & STREAMING PLATFORM
23. Messaging & Streaming Technologies Old and New
The Evolution of Microservices
MESSAGING & STREAMING PLATFORM
24. 1. Data abstraction: what is the right data abstraction?
2. API: what is the right API to provide? Is it just Streams?
3. Primitives: what are primitives to provide?
4. Processing Semantics: How to meet business needs?
5. Tools: what are the tools to offer?
25. Matteo Merli
CTO
StreamNative @merlimat @merlimat
● CTO, StreamNative
● Co-creator and PMC Chair of Apache Pulsar
● Ex Co-Founder, Streamlio
● Ex Splunk, Yahoo!
31. ● “Asynchronously broadcast messages to a set of consumers”
● Decouples producers from consumers
● Subscribers state is managed by the system
PUBLISH/SUBSCRIBE, IN THE LARGE
Topics
32. ● Infinite Retention
● Source of truth
● Can be read from any position
● Events can be stored:
○ By row
○ Columnar format → Lakehouse Integration
Replayable Streams
Topics
34. ● Unbound sequence of events
● Ordered
● Replayable
● Partitioned for scalability
Stream
Stream, Queue, and Table
35. ● Queue is the fundamental “decoupling” mechanism
● The “Queue” is the unit to distribute work across a set of
workers
● Multiple routing strategies:
○ Round-robin
○ Per-key ordering
● Individual message tracking
● Delivery delay
Queue
Stream, Queue, and Table
36. ● View the data in the topic as a “table”
● Access last value for each message key
● Can be used for:
○ Sharing state between processes
○ Distributed cache with local reads
● Integrated with transactions
Table
Stream, Queue, and Table
38. Shared vs Exclusive
Shared Producers Shared Producers Exclusive Producers
Shared Subscribers Exclusive Subscribers Exclusive Subscribers
● Exclusive access for producers:
○ Allows to manage producers epoch
○ Allows to implement leader election
● For consumers the choice is:
○ Shared subscribers → Distribute work
○ Exclusive subscribers → Receive all messages in order
40. Semantics
AT LEAST ONCE
Message pulled once or
more times; processed
each time
Duplicates are possible
Receipt guaranteed
No missing data
41. Semantics
AT MOST ONCE
Message pulled once
? May or may not be
received
No duplicates
Possible missing data
?
AT LEAST ONCE
Message pulled once or
more times; processed
each time
Duplicates are possible
Receipt guaranteed
No missing data
42. Semantics
AT MOST ONCE
Message pulled once
? May or may not be
received
No duplicates
Possible missing data
?
AT LEAST ONCE
Message pulled once or
more times; processed
each time
Duplicates are possible
EXACTLY ONCE
Message pulled once or
more times; processed
ONLY once
Receipt guaranteed
No duplicates
No missing data
Receipt guaranteed
No missing data
43. Semantics
AT MOST ONCE
Message pulled once
? May or may not be
received
No duplicates
Possible missing data
?
AT LEAST ONCE
Message pulled once or
more times; processed
each time
EXACTLY ONCE
Message pulled once or
more times; processed
ONLY once
Receipt guaranteed
No duplicates
No missing data
PIP-30: Pulsar Transactions
Receipt guaranteed
No missing data
Duplicates are possible
45. ● A simple, high-performance, language-agnostic TCP protocol
● The protocol is versioned and maintains backward compatibility
with older versions
● Layers of abstraction build upon simple client operations
Client
Three Layers of Abstraction
46. Client - REST
Three Layers of Abstraction
{REST}
admin requests
pulsar-admin, pulsarctl, API
47. Client - REST
Three Layers of Abstraction
{REST}
admin requests
data requests
PIP-64: Introduce REST endpoints
for producing, consuming, and
reading messages
48. Functions - Serverless Event Processing
Three Layers of Abstraction
input topic 1
input topic 2
input topic N
…
output topic 1
output topic 2
49. Functions - Serverless Event Processing
Three Layers of Abstraction
input topic 1
input topic 2
input topic N
…
output topic 1
output topic 2
50. Three Layers of Abstraction
PULSAR CLIENT
PULSAR
FUNCTIONS
Less complexity,
Less flexibility
More flexibility,
More complexity
51. Three Layers of Abstraction
PULSAR CLIENT
PULSAR
FUNCTIONS
?
Less complexity,
Less flexibility
More flexibility,
More complexity
52. Three Layers of Abstraction
PULSAR CLIENT
PULSAR
FUNCTIONS
SQL
Less complexity,
Less flexibility
More flexibility,
More complexity
53. Introducing (pf)SQL - Pulsar Functions made easy
Three Layers of Abstraction
● Designed for Pulsar Functions
● Filtering / Transformation /
Routing
● Transformations for Pulsar IO
connectors
54. SQL - Pulsar Functions made easy
Three Layers of Abstraction
Neng Lu
Platform Engineering Lead
StreamNative
Simplify Pulsar Functions
Development with SQL
3:20 PM - 3:50 PM
56. Concluding
● Complex event-driven applications require a
wide-range of abstractions, API and semantics for
each specific problem
● Pulsar is the system that solves all these problems
in the most natural and comprehensive way
57. Pulsar Summit
San Francisco
Hotel Nikko
August 18 2022
Keynote
Thank you!
Pulsar Summit
San Francisco
Hotel Nikko
August 18 2022
Sijie Guo
Co-Founder & CEO • StreamNative
Matteo Merli
CTO • StreamNative