Pulsar for Kafka People_Jesse anderson

Pulsar For Kafka People
1 / 25Copyright © 2020 Smoking Hand LLC. All rights Reserved. Version: 82982da9

Introducing Pulsar
Architectures
API and Programming Differences
Use Cases

Apache Pulsar is a distributed
event streaming system
Apache Pulsar
• It uses a distributed log to
durably store messages
• Pulsar was originally created
at Yahoo
• Open sourced in 2016
• Graduated to a top-level
Apache project in 2018

Kafka is a distributed publish
subscribe system
Apache Kafka
• It uses a commit log to track
changes
• Kafka was originally created
at LinkedIn
• Open sourced in 2011
• Graduated to a top-level
Apache project in 2012

Introducing Pulsar
Architectures
Use Cases

Basic Kafka Architecture
Publisher SubscriberKafka
Publisher sends data and
doesn't know about the
subscribers or their status.
Subscriber recieves data from
publisher and never directly
interacts with it.
All interactions go through
Kafka and it handles all
communication.

Basic Pulsar Architecture
Producer ConsumerPulsar
Producers do not directly
interact with the BookKeeper
cluster.
Consumers do not directly
interact with the BookKeeper
cluster.
All Brokers in the Pulsar
cluster are stateless and can
be scaled independently.
BookKeeper All Bookies in the BookKeeper
cluster are stateful and can
be scaled independently.

Kafka Partitions
Producer 0
Data is divided into partitions.
Partitions are both logical
and physical divisions.
Producer 1
Topic
Partition 0 Partition 1 Partition 2
All data is sent and received
on topics. Topics group like
data together.

Pulsar Ledgers
The individual messages that
are produced are stored as
records in the ledger.
Stream
Ledger 1 Ledger 2 Ledger 3
Record 1 Record 2 Record 3 Record 4
Each topic has it's own stream
and all data for a topic
is stored in it.
As more data is added to a
topic, new ledgers are
allocated to store the data.

Kafka Consumers
Producer 0
Consumer recieves data from
all topic partitions and connects
to brokers 0, 1, and 2.
Producer 1
Broker 0 Broker 1 Broker 2
Topic
Consumer (P012)

Pulsar Subscriptions
Producer 0
In failover, all partitions are
consumed by one consumer and
will fail over to hot spare on fail.
Producer 1
Broker 0 Broker 1 Broker 2
Topic
Failover Sub. (P012)
Shared Sub. (P012)
In shared, messages are
sent in a round robin way to
all consumers.
Shared Sub. (P012)
Key Shared (P012)
Key Shared (P012)
Failover Sub. (P)
In key shared, messages with
the same key are consistently
routed to the same consumer.

Introducing Pulsar
Architectures
Use Cases

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import static org.apache.kafka.clients.producer.ProducerConfig.*;
Properties props = new Properties();
// Configure brokers to connect to
props.put(BOOTSTRAP_SERVERS_CONFIG, "broker1:9092");
// Create a producer with the key as a string and value as a string
KafkaProducer<String, String> producer = new KafkaProducer<>(props,
new StringSerializer(), new StringSerializer());
// Create ProducerRecord and send it
String key = "mykey";
String value = "myvalue";
ProducerRecord<String, String> record = new
ProducerRecord<>("hello_topic", key, value);
producer.send(record);
producer.close();
Kafka Producer API

PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar://broker1:6650")
.build();
// Create a producer that will send values as strings
// Default is byte[]
Producer<String> producer = client
.newProducer(Schema.STRING)
.topic("hellotopic")
.create();
// Create a new message, send it, and block until it is
// acknowledged
producer.newMessage()
.key("mykey")
.value("myvalue")
.send();
// Create a new message, send it, and don't block until it is
// acknowledged
producer.newMessage()
.key("mykey2")
.value("myvalue2")
.sendAsync();
// Close producer and client
producer.close();
client.close();
Pulsar Producer API

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import static org.apache.kafka.clients.consumer.ConsumerConfig.*;
@SuppressWarnings("unused")
public class HelloConsumer {
KafkaConsumer<String, String> consumer;
public void createConsumer() {
String topic = "hello_topic";
Properties props = new Properties();
// Configure initial location bootstrap servers
props.put(BOOTSTRAP_SERVERS_CONFIG, "broker1:9092");
// Configure consumer group
props.put(GROUP_ID_CONFIG, "group1");
// Create the consumer with the key as a string and value as a string
consumer = new KafkaConsumer<>(props, new StringDeserializer(),
new StringDeserializer());
Kafka Consumer API (1/2)

consumer.subscribe(Arrays.asList(topic));
while (true) {
// Poll for ConsumerRecords for a certain amount of time
ConsumerRecords<String, String> records = consumer.poll(
Duration.ofMillis(100));
// Process the ConsumerRecords, if any, that came back
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
String value = record.value();
// Do something with message
}
}
}
public void close() {
consumer.close();
}
public static void main(String[] args) {
HelloConsumer consumer = new HelloConsumer();
consumer.createConsumer();
consumer.close();
}
Kafka Consumer API (2/2)

client = PulsarClient.builder()
.serviceUrl("pulsar://broker1:6650")
.build();
String myTopic = "hellotopic";
String mySubscriptionName = "my-subscription";
// Create a consumer that will receive values as strings
// Default is byte[]
consumer = client.newConsumer(Schema.STRING)
.topic(myTopic)
.subscriptionName(mySubscriptionName)
.subscribe();
while (true) {
// Block and wait until a single message is available
Message<String> message = consumer.receive();
try {
// Do something with the message
System.out.println("Key is "" + message.getKey()
+ "" value is "" + message.getValue()
+ """);
// Acknowledge the message so that it can be
// deleted by the message broker
consumer.acknowledgeCumulative(message);
} catch (Exception e) {
// Message failed to process, redeliver later
consumer.negativeAcknowledge(message);
}
}
Pulsar Consumer API

Both projects have an ecosystem
associated with them
Ecosystem
Projects
• Kafka Streams -> Pulsar
Functions
• KSQLDB (prop) -> Pulsar SQL
• Kafka Connect -> Pulsar IO
• Kafka API compatibility for
Pulsar

Introducing Pulsar
Architectures
Use Cases

Kafka++
All Kafka use cases plus
more

Work Queues
http://tiny.bdi.io/workqueues

Geo-Replication
Built-in geo-replication

Unified
Do both MQ-style and
Pub/Sub-style with the
same cluster

Lots of Topics
Supports millions of topics

Thank You
bigdatainstitute.io

Pulsar for Kafka People_Jesse anderson

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Pulsar for Kafka People_Jesse anderson

Ähnlich wie Pulsar for Kafka People_Jesse anderson (20)

Mehr von StreamNative

Mehr von StreamNative (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Pulsar for Kafka People_Jesse anderson