Tracking Apache Pulsar Messages with Apache SkyWalking - Pulsar Virtual Summit Europe 2021

•

0 gefällt mir•197 views

Apache Pulsar is a cloud-native, distributed messaging and streaming platform. Apache SkyWalking is a popular application performance monitoring tool for distributed systems, specially designed for microservices, cloud-native, and container-based (Docker, K8s) architectures. When Apache SkyWalking meets Apache Pulsar, what will happen next? As we all know, message tracing is a good way that helps engineers troubleshoot problems related to messages publishing and receiving. In this talk, Sheng Wu and Penghui Li will walk through the features of Apache SkyWalking and Apache Pulsar, and run one step-by-step demo showing how to track Apache Pulsar messages by Apache SkyWalking.

Technologie

Pulsar Virtual Summit Europe 2021
Tracking Apache Pulsar Messages
with Apache SkyWalking

Pulsar Virtual Summit Europe 2021
Tracing

Pulsar Virtual Summit Europe 2021
Logging

Pulsar Virtual Summit Europe 2021
Show How SkyWalking works with Pulsar

Pulsar Virtual Summit Europe 2021
- Publish messages
- Publish receipt
- Consume messages
- Consume message with listener mode
- Acknowledge messages (not supported yet)
- Negative acknowledgement (not supported yet)
Tracing Pulsar Messages

Pulsar Virtual Summit Europe 2021
Tracing Pulsar Messages

Pulsar Virtual Summit Europe 2021
- 2.2.x
- 2.3.x
- 2.4.x
- 2.5.x
- 2.6.x
- 2.7.x
Supported Pulsar version

$Pulsar Virtual Summit Europe 2021 Enable the skywalking agent -javaagent:/{skywalking_home}/agent/skywalking-agent.jar -DSW_AGENT_NAME=<your-application-name> More details to see https://skywalking.apache.org/docs/skywalking-java/latest/en/setup/service -agent/java-agent/readme/#setup-java-agent$

Pulsar Virtual Summit Europe 2021
Introduce the demo
https://github.com/streamnative/skywalking-pulsar-demo
A simple web application for generating data to a topic and simulate 3
consumers to receive data from the topic with 3 different subscriptions
- The application start a web servlet with port 8088
- Exposing a REST endpoint for generating data (/demo/generate-data)
- The producer inside the application publishes a message once the
endpoint get called
- 3 consumers continues to consume data from the topic

Pulsar Virtual Summit Europe 2021
Introduce the demo
https://github.com/streamnative/skywalking-pulsar-demo

Pulsar Virtual Summit Europe 2021
Log integration
https://github.com/streamnative/skywalking-pulsar-demo

Pulsar Virtual Summit Europe 2021
Demo time ...
https://github.com/streamnative/skywalking-pulsar-demo
- Trace integration
- Log integration

Pulsar Virtual Summit Europe 2021
More in the future?
- Use Pulsar transport all SkyWalking data to server side
- Metrics of Pulsar server. Prometheus instrumented metrics could be
shipped into SkyWalking.
- Server logs collecting and analyze.
- Dashboards for Pulsar server clusters.

Pulsar Virtual Summit Europe 2021
Thank You

Empfohlen

Building a Streaming Microservice Architecture: with Apache Spark Structured ...Databricks

Kafka Streams: What it is, and how to use it?confluent

Show Me Kafka Tools That Will Increase My Productivity! (Stephane Maarek, Dat...confluent

Introduction to Kafka connectKnoldus Inc.

An Introduction to Confluent Cloud: Apache Kafka as a Serviceconfluent

From Zero to Hero with Kafka Connectconfluent

Common issues with Apache Kafka® Producerconfluent

Grafana Loki: like Prometheus, but for LogsMarco Pracucci

Empfohlen

Building a Streaming Microservice Architecture: with Apache Spark Structured ...Databricks

Kafka Streams: What it is, and how to use it?confluent

Show Me Kafka Tools That Will Increase My Productivity! (Stephane Maarek, Dat...confluent

Introduction to Kafka connectKnoldus Inc.

An Introduction to Confluent Cloud: Apache Kafka as a Serviceconfluent

From Zero to Hero with Kafka Connectconfluent

Common issues with Apache Kafka® Producerconfluent

Grafana Loki: like Prometheus, but for LogsMarco Pracucci

Amazon EKS - Elastic Container Service for KubernetesAmazon Web Services

Kubernetes and PrometheusWeaveworks

Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안SANG WON PARK

Streaming all over the world Real life use cases with Kafka Streamsconfluent

Prometheuswyukawa

TerraformDiego Pacheco

Introduction to Apache KafkaJeff Holoman

Apache Flink internalsKostas Tzoumas

Kafka presentationMohammed Fazuluddin

Deploying Confluent Platform for Productionconfluent

Monitoring Apache Kafkaconfluent

MicroservicesMeysam Javadi

Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...confluent

Introduction to Kafka StreamsGuozhang Wang

Gatekeeper: API gatewayChengHui Weng

Monitoring with prometheusKasper Nissen

Infrastructure & System Monitoring using PrometheusMarco Pas

Service discovery with Eureka and Spring CloudMarcelo Serpa

A Deep Dive into Kafka Controllerconfluent

Deep Dive into Apache Kafkaconfluent

Fast Streaming into Clickhouse with Apache PulsarTimothy Spann

[Big Data Spain] Apache Spark Streaming + Kafka 0.10: an Integration StoryJoan Viladrosa Riera

Weitere ähnliche Inhalte

Was ist angesagt?

Amazon EKS - Elastic Container Service for KubernetesAmazon Web Services

Kubernetes and PrometheusWeaveworks

Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안SANG WON PARK

Streaming all over the world Real life use cases with Kafka Streamsconfluent

Prometheuswyukawa

TerraformDiego Pacheco

Introduction to Apache KafkaJeff Holoman

Apache Flink internalsKostas Tzoumas

Kafka presentationMohammed Fazuluddin

Deploying Confluent Platform for Productionconfluent

Monitoring Apache Kafkaconfluent

MicroservicesMeysam Javadi

Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...confluent

Introduction to Kafka StreamsGuozhang Wang

Gatekeeper: API gatewayChengHui Weng

Monitoring with prometheusKasper Nissen

Infrastructure & System Monitoring using PrometheusMarco Pas

Service discovery with Eureka and Spring CloudMarcelo Serpa

A Deep Dive into Kafka Controllerconfluent

Deep Dive into Apache Kafkaconfluent

Was ist angesagt? (20)

Amazon EKS - Elastic Container Service for Kubernetes

Kubernetes and Prometheus

Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안

Streaming all over the world Real life use cases with Kafka Streams

Prometheus

Terraform

Introduction to Apache Kafka

Apache Flink internals

Kafka presentation

Deploying Confluent Platform for Production

Monitoring Apache Kafka

Microservices

Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...

Introduction to Kafka Streams

Gatekeeper: API gateway

Monitoring with prometheus

Infrastructure & System Monitoring using Prometheus

Service discovery with Eureka and Spring Cloud

A Deep Dive into Kafka Controller

Deep Dive into Apache Kafka

Ähnlich wie Tracking Apache Pulsar Messages with Apache SkyWalking - Pulsar Virtual Summit Europe 2021

Fast Streaming into Clickhouse with Apache PulsarTimothy Spann

[Big Data Spain] Apache Spark Streaming + Kafka 0.10: an Integration StoryJoan Viladrosa Riera

Spark streaming + kafka 0.10Joan Viladrosa Riera

Using the FLiPN Stack for Edge AI (Flink, NiFi, Pulsar) - Pulsar Virtual Summ...StreamNative

Using the FLiPN stack for edge ai (flink, nifi, pulsar)Timothy Spann

The Physical World meets the WebMaximiliano Firtman

Kafka Connect & Streams - the ecosystem around KafkaGuido Schmutz

Spark Streaming InfoDoug Chang

Istio PlaygroundQAware GmbH

Cloud Foundry Meetup Stuttgart 2017 - Spring Cloud DevelopmentAndreas Falk

Encode x NEAR: Technical Overview of NEAR 1KlaraOrban

Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europ...StreamNative

Jax WS JAX RS and Java Web Apps with WSO2 PlatformWSO2

Kafka Connect & Kafka Streams/KSQL - the ecosystem around KafkaGuido Schmutz

(ARC402) Deployment Automation: From Developers' Keyboards to End Users' Scre...Amazon Web Services

Openstack Quantumopenstackindia

Kubernetes monitoring using prometheus stackJuraj Hantak

Running Microservices on AWS Elastic BeanstalkAmazon Web Services

Get visibility into your Containers/Kubernetes using Azure MonitorBizTalk360

Containerizing your Security Operations CenterJimmy Mesta

Ähnlich wie Tracking Apache Pulsar Messages with Apache SkyWalking - Pulsar Virtual Summit Europe 2021 (20)

Fast Streaming into Clickhouse with Apache Pulsar

[Big Data Spain] Apache Spark Streaming + Kafka 0.10: an Integration Story

Spark streaming + kafka 0.10

Using the FLiPN Stack for Edge AI (Flink, NiFi, Pulsar) - Pulsar Virtual Summ...

Using the FLiPN stack for edge ai (flink, nifi, pulsar)

The Physical World meets the Web

Kafka Connect & Streams - the ecosystem around Kafka

Spark Streaming Info

Istio Playground

Cloud Foundry Meetup Stuttgart 2017 - Spring Cloud Development

Encode x NEAR: Technical Overview of NEAR 1

Interactive Analytics on Pulsar with Pulsar SQL - Pulsar Virtual Summit Europ...

Jax WS JAX RS and Java Web Apps with WSO2 Platform

Kafka Connect & Kafka Streams/KSQL - the ecosystem around Kafka

(ARC402) Deployment Automation: From Developers' Keyboards to End Users' Scre...

Openstack Quantum

Kubernetes monitoring using prometheus stack

Running Microservices on AWS Elastic Beanstalk

Get visibility into your Containers/Kubernetes using Azure Monitor

Containerizing your Security Operations Center

Mehr von StreamNative

Is Using KoP (Kafka-on-Pulsar) a Good Idea? - Pulsar Summit SF 2022StreamNative

Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...StreamNative

Blue-green deploys with Pulsar & Envoy in an event-driven microservice ecosys...StreamNative

Distributed Database Design Decisions to Support High Performance Event Strea...StreamNative

Simplify Pulsar Functions Development with SQL - Pulsar Summit SF 2022StreamNative

Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022StreamNative

Validating Apache Pulsar’s Behavior under Failure Conditions - Pulsar Summit ...StreamNative

Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...StreamNative

Message Redelivery: An Unexpected Journey - Pulsar Summit SF 2022StreamNative

Unlocking the Power of Lakehouse Architectures with Apache Pulsar and Apache ...StreamNative

Understanding Broker Load Balancing - Pulsar Summit SF 2022StreamNative

Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...StreamNative

Pulsar's Journey in Yahoo!: On-prem, Cloud and Hybrid - Pulsar Summit SF 2022StreamNative

Event-Driven Applications Done Right - Pulsar Summit SF 2022StreamNative

Pulsar @ Scale. 200M RPM and 1K instances - Pulsar Summit SF 2022StreamNative

Data Democracy: Journey to User-Facing Analytics - Pulsar Summit SF 2022StreamNative

Beam + Pulsar: Powerful Stream Processing at Scale - Pulsar Summit SF 2022StreamNative

Welcome and Opening Remarks - Pulsar Summit SF 2022StreamNative

Log System As Backbone – How We Built the World’s Most Advanced Vector Databa...StreamNative

MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...StreamNative

Mehr von StreamNative (20)

Is Using KoP (Kafka-on-Pulsar) a Good Idea? - Pulsar Summit SF 2022

Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...

Blue-green deploys with Pulsar & Envoy in an event-driven microservice ecosys...

Distributed Database Design Decisions to Support High Performance Event Strea...

Simplify Pulsar Functions Development with SQL - Pulsar Summit SF 2022

Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022

Validating Apache Pulsar’s Behavior under Failure Conditions - Pulsar Summit ...

Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...

Message Redelivery: An Unexpected Journey - Pulsar Summit SF 2022

Unlocking the Power of Lakehouse Architectures with Apache Pulsar and Apache ...

Understanding Broker Load Balancing - Pulsar Summit SF 2022

Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...

Pulsar's Journey in Yahoo!: On-prem, Cloud and Hybrid - Pulsar Summit SF 2022

Event-Driven Applications Done Right - Pulsar Summit SF 2022

Pulsar @ Scale. 200M RPM and 1K instances - Pulsar Summit SF 2022

Data Democracy: Journey to User-Facing Analytics - Pulsar Summit SF 2022

Beam + Pulsar: Powerful Stream Processing at Scale - Pulsar Summit SF 2022

Welcome and Opening Remarks - Pulsar Summit SF 2022

Log System As Backbone – How We Built the World’s Most Advanced Vector Databa...

MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...

Kürzlich hochgeladen

Zeshan Sattar- Assessing the skill requirements and industry expectations for...itnewsafrica

Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Mark Goldstein

Design pattern talk by Kaya Weers - 2024 (v2)Kaya Weers

Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentPim van der Noll

The State of Passkeys with FIDO Alliance.pptxLoriGlavin3

Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxLoriGlavin3

A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3

Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integrationmarketing932765

Time Series Foundation Models - current state and future directionsNathaniel Shimoni

Scale your database traffic with Read & Write split using MySQL RouterMydbops

Glenn Lazarus- Why Your Observability Strategy Needs Security Observabilityitnewsafrica

The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech

[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra

Data governance with Unity Catalog PresentationKnoldus Inc.

Generative AI - Gitex v1Generative AI - Gitex v1.pptxfnnc6jmgwh

How to write a Business Continuity PlanDatabarracks

MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesManik S Magar

Generative Artificial Intelligence: How generative AI works.pdfIngrid Airi González

Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada

UiPath Community: Communication Mining from Zero to HeroUiPathCommunity

Kürzlich hochgeladen (20)

Zeshan Sattar- Assessing the skill requirements and industry expectations for...

Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24

Design pattern talk by Kaya Weers - 2024 (v2)

Emixa Mendix Meetup 11 April 2024 about Mendix Native development

The State of Passkeys with FIDO Alliance.pptx

Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx

A Deep Dive on Passkeys: FIDO Paris Seminar.pptx

Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration

Time Series Foundation Models - current state and future directions

Scale your database traffic with Read & Write split using MySQL Router

Glenn Lazarus- Why Your Observability Strategy Needs Security Observability

The Ultimate Guide to Choosing WordPress Pros and Cons

[Webinar] SpiraTest - Setting New Standards in Quality Assurance

Data governance with Unity Catalog Presentation

Generative AI - Gitex v1Generative AI - Gitex v1.pptx

How to write a Business Continuity Plan

MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes

Generative Artificial Intelligence: How generative AI works.pdf

Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024

UiPath Community: Communication Mining from Zero to Hero

Tracking Apache Pulsar Messages with Apache SkyWalking - Pulsar Virtual Summit Europe 2021

1. Pulsar Virtual Summit Europe 2021 Tracking Apache Pulsar Messages with Apache SkyWalking

2. Pulsar Virtual Summit Europe 2021

3. Pulsar Virtual Summit Europe 2021

4. Pulsar Virtual Summit Europe 2021 Tracing

5. Pulsar Virtual Summit Europe 2021

6. Pulsar Virtual Summit Europe 2021

7. Pulsar Virtual Summit Europe 2021

8. Pulsar Virtual Summit Europe 2021

9. Pulsar Virtual Summit Europe 2021 Logging

10. Pulsar Virtual Summit Europe 2021

11. Pulsar Virtual Summit Europe 2021

12. Pulsar Virtual Summit Europe 2021

13. Pulsar Virtual Summit Europe 2021

14. Pulsar Virtual Summit Europe 2021

15. Pulsar Virtual Summit Europe 2021

16. Pulsar Virtual Summit Europe 2021 Show How SkyWalking works with Pulsar

17. Pulsar Virtual Summit Europe 2021 - Publish messages - Publish receipt - Consume messages - Consume message with listener mode - Acknowledge messages (not supported yet) - Negative acknowledgement (not supported yet) Tracing Pulsar Messages

18. Pulsar Virtual Summit Europe 2021 Tracing Pulsar Messages

19. Pulsar Virtual Summit Europe 2021 - 2.2.x - 2.3.x - 2.4.x - 2.5.x - 2.6.x - 2.7.x Supported Pulsar version

20. Pulsar Virtual Summit Europe 2021 Enable the skywalking agent -javaagent:/{skywalking_home}/agent/skywalking-agent.jar -DSW_AGENT_NAME=<your-application-name> More details to see https://skywalking.apache.org/docs/skywalking-java/latest/en/setup/service -agent/java-agent/readme/#setup-java-agent

21. Pulsar Virtual Summit Europe 2021 Introduce the demo https://github.com/streamnative/skywalking-pulsar-demo A simple web application for generating data to a topic and simulate 3 consumers to receive data from the topic with 3 different subscriptions - The application start a web servlet with port 8088 - Exposing a REST endpoint for generating data (/demo/generate-data) - The producer inside the application publishes a message once the endpoint get called - 3 consumers continues to consume data from the topic

22. Pulsar Virtual Summit Europe 2021 Introduce the demo https://github.com/streamnative/skywalking-pulsar-demo

23. Pulsar Virtual Summit Europe 2021 Introduce the demo https://github.com/streamnative/skywalking-pulsar-demo

24. Pulsar Virtual Summit Europe 2021 Introduce the demo https://github.com/streamnative/skywalking-pulsar-demo

25. Pulsar Virtual Summit Europe 2021 Log integration https://github.com/streamnative/skywalking-pulsar-demo

26. Pulsar Virtual Summit Europe 2021 Demo time ... https://github.com/streamnative/skywalking-pulsar-demo - Trace integration - Log integration

27. Pulsar Virtual Summit Europe 2021 More in the future? - Use Pulsar transport all SkyWalking data to server side - Metrics of Pulsar server. Prometheus instrumented metrics could be shipped into SkyWalking. - Server logs collecting and analyze. - Dashboards for Pulsar server clusters.

28. Pulsar Virtual Summit Europe 2021 Thank You