Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Pulsar Virtual Summit Europe 2021
Interactive Analytics on
Pulsar with Pulsar SQL
Axel Sirota
AI and Coud Consultant
@Axel...
Who am I?
QR to my Pluralsight
courses
QR to my O’Reilly
trainings
–Microsoft Certified Trainer
–Author, Instructor and Ed...
Pulsar Virtual Summit Europe 2021
Catalogue
• A Simple Scenario
• Inspecting and Debugging Topics with Pulsar SQL
• Intera...
Pulsar Virtual Summit Europe 2021
Catalogue
• A Simple Scenario
Pulsar Virtual Summit Europe 2021
Ann
a,28
,$50
Application Instance Pulsar Deployment
File Source
Pulsar Function
Ingress...
Pulsar Virtual Summit Europe 2021
1. You check the status on the Pulsar Function and there
are some exceptions
2. And you ...
Pulsar Virtual Summit Europe 2021
Catalogue
• Inspecting and Debugging Topics with Pulsar SQL
Pulsar Virtual Summit Europe 2021
Pulsar SQL enhances the Pulsar Presto connector to query
topics interactively
One can ch...
Pulsar Virtual Summit Europe 2021
BookKeeper
Pulsar Broker Presto
Bookie 1 Bookie 2 Bookie 3
Presto
Connector
content page
Configuration file
Specify where are the zookeepers
and brokers
connector.name=pulsar
pulsar.broker-service-u...
content page
Two commands and magic
Start the worker inside the Presto
cluster
->./bin/pulsar sql-worker start
Running in ...
content page
Two commands and magic
->./bin/pulsar sql
presto>
Start the console
So simple, yet so powerful!
Pulsar Virtual Summit Europe 2021
The Full Architecture
Pulsar Virtual Summit Europe 2021
1. Validate schemas in a readable SQL format
2. Easily debug bad messages that make Puls...
Pulsar Virtual Summit Europe 2021
Catalogue
• Interactive Analytics
Pulsar Virtual Summit Europe 2021
Equivalence
Pulsar Presto
Namespaces Schemas
Topics Tables
Fields Columns
Unserialized
m...
Pulsar Virtual Summit Europe 2021
presto> show columns from pulsar."public/default"."voo";
Column | Type | Extra | Comment...
Pulsar Virtual Summit Europe 2021
2021-09-13, 12 2021-09-14, 9 2021-09-15, 15
metrics topic without Schema in public/pulsa...
Pulsar Virtual Summit Europe 2021
2021-09-13, 12 2021-09-14, 9 2021-09-15, 15
metrics topic with Schema in public/pulsar-s...
Pulsar Virtual Summit Europe 2021
2021-09-13, 12 2021-09-14, 9 2021-09-15, 15
metrics topic with Schema in public/pulsar-s...
Pulsar Virtual Summit Europe 2021
2021-09-13, 12 2021-09-14, 9 2021-09-15, 15
metrics topic with Schema in public/pulsar-s...
Pulsar Virtual Summit Europe 2021
1. Interactively debug topics without open subscriptions
2. Audit who send each message,...
Pulsar Virtual Summit Europe 2021
Thanks!!
Questions?
Axel Sirota
AI and Coud Consultant
@AxelSirota
Upcoming SlideShare
Loading in …5
×

of

Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 1 Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 2 Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 3 Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 4 Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 5 Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 6 Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 7 Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 8 Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 9 Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 10 Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 11 Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 12 Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 13 Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 14 Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 15 Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 16 Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 17 Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 18 Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 19 Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 20 Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 21 Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 22 Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021 Slide 23
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

0 Likes

Share

Download to read offline

Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021

Download to read offline

Suppose you want to know analytics on your Pulsar topics, or you want to debug those hard corner cases that fail to be sent, or even you want to monitor your Pulsar deployment: how do you do it?
A tool exists to do this and more: Pulsar SQL. Since the 2.2.0 release, Pulsar SQL provides an abstraction layer to run any SQL query we may want against Pulsar effortlessly and without affecting performance. There is nothing like it on the pub-sub ecosystem.
In this short session, we will revisit what Pulsar SQL is, how to make the best out of it, how to deploy it, and how to use it!

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europe 2021

  1. 1. Pulsar Virtual Summit Europe 2021 Interactive Analytics on Pulsar with Pulsar SQL Axel Sirota AI and Coud Consultant @AxelSirota
  2. 2. Who am I? QR to my Pluralsight courses QR to my O’Reilly trainings –Microsoft Certified Trainer –Author, Instructor and Editor at Pluralsight, O’Reilly Media, and Develop Intelligence –AI and Cloud Consultant
  3. 3. Pulsar Virtual Summit Europe 2021 Catalogue • A Simple Scenario • Inspecting and Debugging Topics with Pulsar SQL • Interactive Analytics
  4. 4. Pulsar Virtual Summit Europe 2021 Catalogue • A Simple Scenario
  5. 5. Pulsar Virtual Summit Europe 2021 Ann a,28 ,$50 Application Instance Pulsar Deployment File Source Pulsar Function Ingress topic Processed topic
  6. 6. Pulsar Virtual Summit Europe 2021 1. You check the status on the Pulsar Function and there are some exceptions 2. And you haven’t set a log topic for each Pulsar function (at least it happened to us) 3. You don’t want downtime to debug locally Some issues appear… What can you do?
  7. 7. Pulsar Virtual Summit Europe 2021 Catalogue • Inspecting and Debugging Topics with Pulsar SQL
  8. 8. Pulsar Virtual Summit Europe 2021 Pulsar SQL enhances the Pulsar Presto connector to query topics interactively One can check every message that passed through the topic easily and in a safe manner It is lightweight, simple, enables high concurrent access, and you can reuse existing Presto clusters Introducing… Pulsar SQL
  9. 9. Pulsar Virtual Summit Europe 2021 BookKeeper Pulsar Broker Presto Bookie 1 Bookie 2 Bookie 3 Presto Connector
  10. 10. content page Configuration file Specify where are the zookeepers and brokers connector.name=pulsar pulsar.broker-service-url=https://my-pulsar- deployment.com pulsar.zookeeper-uri=https://my-pulsar- deployment.com:2181 Put in conf/presto/catalog/pulsar.proper ties
  11. 11. content page Two commands and magic Start the worker inside the Presto cluster ->./bin/pulsar sql-worker start Running in 6896
  12. 12. content page Two commands and magic ->./bin/pulsar sql presto> Start the console So simple, yet so powerful!
  13. 13. Pulsar Virtual Summit Europe 2021 The Full Architecture
  14. 14. Pulsar Virtual Summit Europe 2021 1. Validate schemas in a readable SQL format 2. Easily debug bad messages that make Pulsar Functions fail unexpectedly 3. Leverage SQL tools and queries for analytics But… why should we use it? What can you do?
  15. 15. Pulsar Virtual Summit Europe 2021 Catalogue • Interactive Analytics
  16. 16. Pulsar Virtual Summit Europe 2021 Equivalence Pulsar Presto Namespaces Schemas Topics Tables Fields Columns Unserialized message __value__
  17. 17. Pulsar Virtual Summit Europe 2021 presto> show columns from pulsar."public/default"."voo"; Column | Type | Extra | Comment -------------------+-----------+-------+----------------------------------------------------------------------------- __value__ | varchar | | The value of the message with primitive type schema __partition__ | integer | | The partition number which the message belongs to __event_time__ | timestamp | | Application defined timestamp in milliseconds of when the event occurred __publish_time__ | timestamp | | The timestamp in milliseconds of when event as published __message_id__ | varchar | | The message ID of the message used to generate this row __sequence_id__ | bigint | | The sequence ID of the message used to generate this row __producer_name__ | varchar | | The name of the producer that publish the message used to generate this row __key__ | varchar | | The partition key for the topic __properties__ | varchar | | User defined properties (9 rows)
  18. 18. Pulsar Virtual Summit Europe 2021 2021-09-13, 12 2021-09-14, 9 2021-09-15, 15 metrics topic without Schema in public/pulsar-summit SELECT * from “public/pulsar-summit”.metrics __value__ 2021-09-13,12 2021-09-14,9 2021-09-15,15
  19. 19. Pulsar Virtual Summit Europe 2021 2021-09-13, 12 2021-09-14, 9 2021-09-15, 15 metrics topic with Schema in public/pulsar-summit (Date, Metric) Date Metric 2021-09-13 12 2021-09-14 9 2021-09-15 15 SELECT * from “public/pulsar-summit”.metrics
  20. 20. Pulsar Virtual Summit Europe 2021 2021-09-13, 12 2021-09-14, 9 2021-09-15, 15 metrics topic with Schema in public/pulsar-summit (Date, Metric) SELECT count(1) from “public/pulsar-summit”.metrics where Metric > 10 Count 3 2021-10-15, 120
  21. 21. Pulsar Virtual Summit Europe 2021 2021-09-13, 12 2021-09-14, 9 2021-09-15, 15 metrics topic with Schema in public/pulsar-summit (Date, Metric) Select as month(Date) as month, SUM(Metric) as agg_metric from “public/pulsar-summit”.metrics group by 1, order by 2 DESC Month agg_metric 10 120 9 36 2021-10-15, 120
  22. 22. Pulsar Virtual Summit Europe 2021 1. Interactively debug topics without open subscriptions 2. Audit who send each message, when, where, what did it send, and how much it took 3. Do analytics on the messages flowing through Pulsar If you need to… Then Pulsar SQL is what you look for! And all of this without affecting production performance
  23. 23. Pulsar Virtual Summit Europe 2021 Thanks!! Questions? Axel Sirota AI and Coud Consultant @AxelSirota

Suppose you want to know analytics on your Pulsar topics, or you want to debug those hard corner cases that fail to be sent, or even you want to monitor your Pulsar deployment: how do you do it? A tool exists to do this and more: Pulsar SQL. Since the 2.2.0 release, Pulsar SQL provides an abstraction layer to run any SQL query we may want against Pulsar effortlessly and without affecting performance. There is nothing like it on the pub-sub ecosystem. In this short session, we will revisit what Pulsar SQL is, how to make the best out of it, how to deploy it, and how to use it!

Views

Total views

195

On Slideshare

0

From embeds

0

Number of embeds

1

Actions

Downloads

0

Shares

0

Comments

0

Likes

0

×