The Next Generation of Streaming https://www.meetup.com/new-york-city-apache-pulsar-meetup/events/291048765/ https://www.meetup.com/risingwave-community/events/291294056/ The Next Generation of Streaming David Kjerrumgaard, Developer Advocate Tim Spann Building a cloud-native streaming application requires a team, from the start of events to the final analytics. I'll walk you through using Apache Pulsar as your central messaging hub to distribute events and real-time messages from and to everywhere it needs to be. All instantly. The next step is to stream into continuous SQL. Agenda 9:00 -- 9:20 The Next Generation of Streaming by Tim Spann, Developer Advocate at StreamNative 9:20 -- 9:40 RisingWave: Cloud-Native Streaming SQL over Pulsar by Rayees Pasha, Head of Product at RisingWave Labs 23-feb-2023 12:15pm EST https://github.com/tspannhw/Flow-SGP30-MLX90640 Apache NiFi, Apache Pulsar, Apache Kafka, MQTT, RabbitMQ, AMQP, Source, Sink, Stream Processing

1. The Next Generation
of Streaming
Tim Spann
Developer Advocate

2. Tim Spann
Developer Advocate
● FLiP(N) Stack = Flink, Pulsar and NiFi Stack
● Streaming Systems/ Data Architect
● Experience:
○ 15+ years of experience with batch and streaming
technologies including Pulsar, Kafka, Flink, Spark, NiFi,
Spring, Java, Big Data, Cloud, MXNet, Hadoop,
Datalakes, IoT and more.

3. David Kjerrumgaard
● Apache Pulsar Committer | Author of Pulsar In Action
● Former Principal Software Engineer on Splunk’s
messaging team that is responsible for Splunk’s
internal Pulsar-as-a-Service platform.
● Former Director of Solution Architecture at Streamlio.
Your photo
goes here,
smile :)

5. • Guaranteed delivery
• Data buffering
- Backpressure
- Pressure release
• Prioritized queuing
• Flow specific QoS
- Latency vs. throughput
- Loss tolerance
• Data provenance
• Supports push and pull
models
• Hundreds of processors
• Visual command and
control
• Over a sixty sources
• Flow templates
• Pluggable/multi-role
security
• Designed for extension
• Clustering
• Version Control
DataFlows for Data Ingest, Movement and Routing

6. • Moving Binary, Unstructured, Image
and Tabular Data
• Enrichment
• Universal Visual Processor
• Simple Event Processor
• Routing
• Feeding data to Central Messaging
• Support for modern protocols
• Kafka Protocol Source/Sink
• Pulsar Protocol Source/Sink
The Power of Apache NiFi

7. Architecture
https://nifi.apache.org/docs/nifi-docs/html/overview.html

10. Unified
Messaging
Platform
Guaranteed
Message
Delivery
Resiliency Infinite
Scalability

11. Streaming
Consumer
Consumer
Consumer
Subscription
Shared
Failover
Consumer
Consumer
Subscription
In case of failure in
Consumer B-0
Consumer
Consumer
Subscription
Exclusive
X
Consumer
Consumer
Key-Shared
Subscription
Pulsar
Topic/Partition
Messaging
11

12. Kafka
On Pulsar
(KoP)
Kafka Protocol (Binary Data over TCP)

13. Schema Registry for Schema Support
Schema Registry
schema-1 (value=Avro/Protobuf/JSON) schema-2 (value=Avro/Protobuf/JSON) schema-3
(value=Avro/Protobuf/JSON)
Schema
Data
ID
Local Cache
for Schemas
+
Schema
Data
ID +
Local Cache
for Schemas
Send schema-1
(value=Avro/Protobuf/JSON) data
serialized per schema ID
Send (register)
schema (if not in
local cache)
Read schema-1
(value=Avro/Protobuf/JSON) data
deserialized per schema ID
Get schema by ID (if
not in local cache)
Producers Consumers

14. Data Offloaders
(Tiered Storage)
Client Libraries
Apache Pulsar ecosystem
hub.streamnative.io
Connectors
(Sources & Sinks)
Protocol Handlers
Pulsar Functions
(Lightweight Stream
Processing)
Processing Engines
… and more!
… and more!

15. Apache NiFi - Apache Pulsar Connector
15
https://github.com/streamnative/pulsar-nifi-bundle

16. Example

17. https://github.com/tspannhw/Flow-SGP30-MLX90640
Example

18. FLiP Stack Weekly
This week in Apache Kafka, Apache Flink, Apache
Pulsar, Apache NiFi, Apache Spark, Java and Open
Source friends.
https://bit.ly/32dAJft

19. 19
Apache
Pulsar
in Action
Please enjoy David’s complete
book which is the ultimate
guide to Pulsar.

20. @PaaSDev
https://www.linkedin.com/in/timothyspann
https://github.com/tspannhw
20
Tim Spann
Developer Advocate

21. Notices
Apache Pulsar™
Apache®, Apache Pulsar™, Pulsar™, Apache Flink®, Flink®, Apache Spark®, Spark®, Apache
NiFi®, NiFi® and the logo are either registered trademarks or trademarks of the Apache
Software Foundation in the United States and/or other countries. No endorsement by The
Apache Software Foundation is implied by the use of these marks.
Copyright © 2021-2022 The Apache Software Foundation. All Rights Reserved. Apache,
Apache Pulsar and the Apache feather logo are trademarks of The Apache Software
Foundation.