Pulsar Watermarking - Pulsar Virtual Summit Europe 2021
•Als PPTX, PDF herunterladen•
0 gefällt mir•301 views
This session is a technical deep-dive into the Apache Pulsar community's plan to support event-time watermarking in a Pulsar topic. The motivation is to simplify and improve the correctness of stream processing applications.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Pulsar Watermarking - Pulsar Virtual Summit Europe 2021
Ähnlich wie Pulsar Watermarking - Pulsar Virtual Summit Europe 2021 (20)
Mehr von StreamNative
Mehr von StreamNative (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Pulsar Watermarking - Pulsar Virtual Summit Europe 2021
- 1. Pulsar Virtual Summit Europe 2021 Pulsar Watermarking Eron Wright Cloud Engineering Lead, StreamNative
- 2. Pulsar Virtual Summit Europe 2021 Overview
- 3. Pulsar Virtual Summit Europe 2021 What are Watermarks? A watermark is an assertion that all events up to a certain timestamp have arrived.
- 4. Pulsar Virtual Summit Europe 2021 Stream Processing with Watermarks In stream processing applications, watermarks regulate the event-time clock. Watermarks allow the processor to make progress in its time-based calculations.
- 5. Pulsar Virtual Summit Europe 2021 Generating Watermarks Strategy: Use a heuristic based on observation of event timestamps. Flaw: Highly dependent on order-of-observation.
- 6. Pulsar Virtual Summit Europe 2021 Generating Watermarks Pulsar offers a per-key ordering guarantee. No guarantees about event-time ordering. Complications: subscription modes, message ACKs, partitioning, transactions.
- 7. Pulsar Virtual Summit Europe 2021 Introducing Pulsar Watermarks (PIP- 102)
- 8. Pulsar Virtual Summit Europe 2021 What are Pulsar Watermarks? A new type of message, generated by producers, aggregated by brokers, and delivered by subscriptions. An assertion by a producer about the progression of event time.
- 9. Pulsar Virtual Summit Europe 2021 Producer API Send watermarks in-band with messages, using broadcast semantics for partitioned topics. package org.apache.pulsar.client.api; public interface Producer<T> extends Closeable { /** * Create a new watermark builder. */ WatermarkBuilder newWatermark(); WatermarkBuilder newWatermark(Transaction txn); TypedMessageBuilder<T> newMessage(); } public interface WatermarkBuilder { /** * Set the event time for a given watermark. */ WatermarkBuilder eventTime(long timestamp); WatermarkBuilder markIdle(); WatermarkId send(); CompletableFuture<WatermarkId> sendAsync(); }
- 10. Pulsar Virtual Summit Europe 2021 Consumer API Receive watermarks in- band with messages. package org.apache.pulsar.client.api; public interface Consumer<T> extends Closeable { /** * @return The latest watermark. */ Watermark getLastWatermark(); Message<T> receive(); CompletableFuture<Message<T>> receiveAsync(); } public interface MessageListener<T> { /** * This method is called when a watermark is received. */ default void receivedWatermark( Consumer<T> consumer, Watermark watermark) {} }
- 11. Pulsar Virtual Summit Europe 2021 Consumer API Supports all subscription modes. Watermarks advance as messages are acknowledged.
- 12. Pulsar Virtual Summit Europe 2021 Brokering Broker aggregates watermarks, taking the minimum across all producers to a topic. Materializes the current watermark for each subscription as messages are acknowledged.
- 13. Pulsar Virtual Summit Europe 2021 Flink Integration Pulsar source automatically provides source-based watermarks. Pulsar sink emits watermarks from Flink pipeline to output topic. FlinkPulsarSource<Example> source = new FlinkPulsarSource<>("public/default/example", ...); source.assignTimestampsAndWatermarks( PulsarWatermarkStrategy .forPulsarWatermarks() .withTimestampAssigner(...)); CREATE TABLE example ( `eventTime` TIMESTAMP(3) METADATA, WATERMARK FOR `eventTime` AS SOURCE_WATERMARK() `key` STRING ) WITH ( 'connector' = 'pulsar', 'topic' = 'persistent://public/default/example' )
- 14. Pulsar Virtual Summit Europe 2021 Recap Proposal: ● Enhance Pulsar to propagate watermarks from producer to consumer. Benefits: ● Improved correctness of Pulsar-based stream processing applications. ● Easier to build multi-stage applications and event-driven microservices.
- 15. Pulsar Virtual Summit Europe 2021 Project Status
- 16. Pulsar Virtual Summit Europe 2021 Proposal PIP-102 is the proposal and is in discussion. - https://github.com/apache/pulsar/issues/12267 A prototype was made at the 2021 Apache Pulsar Hackathon. - https://github.com/EronWright/pulsar-hackathon-2021-projects-watermarking Help wanted! Active discussions include: - Producer participant lifecycle - Automatic watermark generation on broker side Main Contact: - Eron Wright (ewright@streamnative.io)
- 17. Pulsar Virtual Summit Europe 2021 Resources - Streaming 101: The world beyond batch - O'Reilly - Streaming 102: The world beyond batch - O'Reilly
- 18. Pulsar Virtual Summit Europe 2021 Thank-You!
Hinweis der Redaktion
- This slide introduces the abstract concept of a watermark. Explain: Each stream consists of a unbounded series of events and watermarks. Each event has an event timestamp (simplified for illustrative purposes). A watermark with time t indicates there should be no subsequent events with a timestamp older than t. A watermark is similar to an end-to-file (EOF) in that it punctuates a stream, but is in the event domain (specifically a point in time) rather than in the processing domain. A processing domain analogue to EOF in Pulsar would be reaching the end of a terminated topic.
- This slide reveals how stream processors use watermarks to make progress, and the importance of watermark correctness. Explain: Events and watermarks flow through the stream processing pipeline, e.g. from source, to transformer, to sink. We call these operators. A watermark may trigger a time-based computation or cause a state change, such as finalizing a time-based aggregation. The correctness of the watermark is essential for producing correct program output. Some processors support special-case handling of late records. Watermarks may be pessimistic at the expense of end-to-end latency.
- This slide focuses on the challenge of generating correct watermarks. Explain heuristics-based watermarking. Infer the progression of event time by observing event timestamps. A typical strategy is to assume a bounded out-of-orderness, where the watermark lags behind the maximum observed timestamp by a fixed amount of time (a “grace period”). These strategies are highly sensitive ot the order of observation, and may break in dynamic/complex/unified scenarios. Also: generator state may be ephemeral. Explain the concept of regimes and that of unified batch/streaming. Imagine a stream processor that is acting as a consumer of a topic. Real-time: order is largely determined by the producer. Historical: abundance of data, order is determined by system guarantees and by system internals. Ths may cause a heuristics-based generator to fail spectacularly.
- This slide is about the specific challenge of generating watermarks as a consumer of a Pulsar topic. This is the last slide before we dive into the proposal. Explain the specific challenges faced by Pulsar consumers using a diagram showing how events are multiplexed during consumption. Imagine how producing to, and consuming from, a partitioned topic involves fan-out and fan-in. The complexity rises in a distributed, partitioned, multi-party scenario. Only the broker has a complete view of the topic; no one consumer observes all events. This complicates heuristics-based watermark generation at the consumer. The Pulsar client encapsulates various system internals such as scheduling, retries, acknowledgements, fan-in. Transactions tend to disturb the order-of-observation.
- We introduced the concept of watermarks and discussed the challenge of generating correct watermarks. Now we introduce a proposal for an enhancement to Pulsar to address this challenge.
- This slide introduces the overall concept of brokered watermarks. The next slides dive into the details for producer, broker, and consumer. PIP-102 is a community proposal to add first-class support to Pulsar for brokered watermarks. Recall: watermarks are an assertion about the progression of event time with respect to the events in a particular stream. Explain: Place the responsibility (of making assertions) on the producer(s) to a topic. Each producer is responsible only for watermarking its own events. Aggregate the watermarks across producers. Deliver aggregated watermarks to topic consumers via a subscription. Subscriptions are like cursors over a topic, and materialize watermarks as the cursor advances.
- This slide focuses on the producer experience. Remarks: PIP-102 introduces a “watermark” message to the Producer API. For aggregation purposes, producers are recognized by their producer name. Note that Pulsar messages already have an event timestamp. (see TypedMessageBuilder::eventTime). All routing modes are supported, using broadcast semantics for watermarks sent to partitioned topics. Results in a WatermarkId containing a MessageId (position information) per partition. A producer may idle itself to cease contributing watermarks. Is otherwise invariant to producer disconnections. Fully supports transactions.
- This slide focuses on the consumer experience. Continues on next slide with a diagram about subscription modes. Explain: Watermarks are an opt-in feature on the Consumer API. Watermarks are delivered via the MessageListener interface and the latest is available via Consumer::getLastWatermark(). When opted in, slight behavior change on Consumer::receiveAsync() - completes with null when a watermark arrives. Alternatively use shorter timeouts on Consumer::receive().
- Explain: All subscription modes are supported. Some modes have the effect of re-delivering a message to another consumer, e.g. when a message goes unacknowledged or a consumer becomes disconnected. It would be wrong to prematurely advance the watermark for such messages. Hence, watermarks track with acknowledged messages. Internally this is known as the mark-delete point within the subscription cursor. Consider pausing to reiterate how painful watermark generation is in the status-quo. e.g. consumers of key-shared subscriptions lack information, and consumers of failover subscriptions are over-eager.
- This slide focuses on how watermarks are brokered. Explain: Watermarks are stored as special ‘marker’ messages in the managed ledger (shown in orange). When a subscription is created or seeked, the broker materializes the latest watermark at the subscription’s current position. Is implemented with a secondary cursor that is associated with the subscription. For transactional messaging, watermarks are read-committed (as are ordinary messages). Periodic snapshots improve efficiency and allow for ledger operations. Watermarks from outstanding transactions are held in snapshot state.
- This slide talks about how Pulsar watermarking works in Apache Flink with the Pulsar connector. Source: Emits watermarks into the Flink pipeline Provides a ‘source-based’ watermarking strategy Use ordinary timestamp extractor or use eventTime message metadata. Sink: Receives watermarks from Flink pipeline (see FLIP-167) Produces watermarks into the topic Future: better support for idle pipelines
- Explain correctness: This approach improves the determinism of watermarks and thus of program output. Program should produce same output in batch and streaming modes. This is the dream of ‘unified batch and stream processing’. Explain multi-stage: Multi-stage applications use Pulsar topics to move information from stage to stage. Now able to preserve watermarks with high fidelity.