SlideShare a Scribd company logo

Effectively-once semantics in Apache Pulsar

•
•
Matteo Merli
Matteo Merli

“Exactly-once” is a controversial term in the messaging landscape. In this presentation we offer a detailed look at effectively-once delivery semantics in Apache Pulsar and how this is achieved without sacrificing performance.

Related slideshows

Clean architectures with fast api pycones
Clean architectures with fast api pyconesClean architectures with fast api pycones
Clean architectures with fast api pycones
 
Kafka replication apachecon_2013
Kafka replication apachecon_2013Kafka replication apachecon_2013
Kafka replication apachecon_2013
 
Apache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals ExplainedApache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals Explained
 
1 of 37
Download to read offline
Matteo Merli Guaranteed “effectively-once” messaging semantic
What is Apache Pulsar? • Distributed pub/sub messaging • Backed by a scalable log store — Apache BookKeeper • Streaming & Queuing • Low latency • Multi-tenant • Geo-Replication 2
Architecture view • Separate layers between brokers bookies • Broker and bookies can be added independently • Traffic can be shifted very quickly across brokers • New bookies will ramp up on traffic quickly 3 Pulsar Broker 1 Pulsar Broker 2 Pulsar Broker 3 Bookie 1 Bookie 2 Bookie 3 Bookie 4 Bookie 5 Apache BookKeeper Apache Pulsar Producer Consumer
Messaging model 4
Messaging semantics At most once At least once Exactly once 5
“Exactly once” • There is no agreement in industry on what it really means • Any vendor has claimed exactly once at some point • Many caveats… “only if there are no crashes…” • No formal definition of exactly once — unlike “consensus” or “atomic broadcast” 6
“Effectively once” • Identify and discard duplicated messages with 100% accuracy • In presence of any kind of failures • Messages can be received and processed more than once • …but effects on the resulting state will be observed only once 7
What can fail? 8
What can fail? 9
What can fail? 10
What can fail? 11
What can fail? — Geo-Replication 12
Breaking the problem 1. Store the message once — ”producer idempotency” 2. Allow applications to “process data only-once” 13
Idempotent producer • Pulsar broker detects and discards messages that are being retransmitted • It works when a broker crashes and topic is reassigned • It works when a producer application crashes 14
Identifying producers • Use “sequence ids” to detect retransmissions • Each producer on a topic has it own sequence of messages • Use “producer-name” to identify producers 15
Detecting duplicates 16
Detecting duplicates 17
Detecting duplicates 18
Detecting duplicates 19
Sequence Id snapshot 20
Sequence Id snapshot 21
Sequence Id snapshot • Snapshots are taken every N entries to limit recovery time • Snapshot & cursor updates are atomic • Cursor updates are stored in BookKeeper — durable & replicated • On recovery • Load the snapshot from the cursor • Replay the entries from the cursor position 22
What if application producer crashes? • Pulsar needs to identify the new producer as being the same “logical” producer as before • In practice, this is only useful if you have a “replayable” source (eg: file, stream, …) 23
Resuming a producer session ProducerConfiguration conf = new ProducerConfiguration(); conf.setProducerName("my-producer-name"); conf.setSendTimeout(0, TimeUnit.SECONDS); Producer producer = client.createProducer(MY_TOPIC, conf); // Get last committed sequence id before crash long lastSequenceId = producer.getLastSequenceId(); 24
Using sequence Ids // Fictitious record reader class RecordReader source = new RecordReader("/my/file/path"); long fileOffset = producer.getLastSequenceId(); source.seekToOffset(fileOffset); while (source.hasNext()) { long currentOffset = source.currentOffset(); Message msg = MessageBuilder.create() .setSequenceId(currentOffset) .setContent(source.next()).build(); producer.send(msg); } 25
Consuming messages only once • Pulsar Consumer API is very convenient • Managed subscription — tracking individual messages Consumer consumer = client.subscribe(MY_TOPIC, MY_SUBSCRIPTION_NAME); while (true) { Message msg = consumer.receive(); // Process the message... consumer.acknowledge(msg); } 26
Effectively-once with Consumer • Consumer is very simple but doesn’t allow a large degree of control • Processing and acknowledge are not atomic • To achieve “effectively once” we need to rely on an external system to deduplicate the processing results. Eg: • RDBMS — Keep the message id as a column with a “unique” index • Critical write to update the state — compareAndSet() or similar 27
Pulsar Reader • Reader is a low level API to receive data from a Pulsar topic • There is no managed subscription • Application always specifies the message id where it wants to start reading from 28
Reader example MessageId lastMessageId = recoverLastMessageIdFromDB(); Reader reader = client.createReader(MY_TOPIC, lastMessageId, new ReaderConfiguration()); while (true) { Message msg = reader.readNext(); byte[] msgId = msg.getMessageId().toByteArray(); // Process the message and store msgId atomically } 29
Example — Pulsar Functions 30
Pulsar Functions • A function gets messages from 1 or more topics • An instance of the function is invoked to process the event • The output of the function is published on 1 or more topics • Super simple to use — No SDK required — Python example: def process(input): return input + '!' 31
Pulsar Functions 32
Effectively once with functions • Use the message id from source topic as sequence id for sink topic • Works with “Consumer” API • When consuming from multiple topics or partitions, creates 1 producer per each source topic/partition, to ensure monotonic sequence ids 33
Performance • Pulsar approach guarantees deduplication in all failure scenarios • Overhead is minimal: 2 in memory hashmap updates • No reduction in throughput — No increased latency • Controllable increase in recovery time 34
Performance — Benchmark OpenMessaging Benchmark 1 Topic / 1 Partition 1 Partition / 1 Consumer 1Kb msg 35
Difference with Kafka approach 36 Kafka Pulsar Producer Idempotency Best-effort (in memory only) Guaranteed after crash Transactions 2 phase commit No transactions Dedup across producer sessions No Yes Dedup with geo- replication No Yes Throughput Lower (1 in-flight message/batch for ordering) Equal
Curious to Learn More? • Apache Pulsar — https://pulsar.incubator.apache.org • Follow Us — @apache_pulsar • Streamlio blog — https://streaml.io/blog 37

More Related Content

Effectively-once semantics in Apache Pulsar

  • 2. What is Apache Pulsar? • Distributed pub/sub messaging • Backed by a scalable log store — Apache BookKeeper • Streaming & Queuing • Low latency • Multi-tenant • Geo-Replication 2
  • 3. Architecture view • Separate layers between brokers bookies • Broker and bookies can be added independently • Traffic can be shifted very quickly across brokers • New bookies will ramp up on traffic quickly 3 Pulsar Broker 1 Pulsar Broker 2 Pulsar Broker 3 Bookie 1 Bookie 2 Bookie 3 Bookie 4 Bookie 5 Apache BookKeeper Apache Pulsar Producer Consumer
  • 5. Messaging semantics At most once At least once Exactly once 5
  • 6. “Exactly once” • There is no agreement in industry on what it really means • Any vendor has claimed exactly once at some point • Many caveats… “only if there are no crashes…” • No formal definition of exactly once — unlike “consensus” or “atomic broadcast” 6
  • 7. “Effectively once” • Identify and discard duplicated messages with 100% accuracy • In presence of any kind of failures • Messages can be received and processed more than once • …but effects on the resulting state will be observed only once 7
  • 12. What can fail? — Geo-Replication 12
  • 13. Breaking the problem 1. Store the message once — ”producer idempotency” 2. Allow applications to “process data only-once” 13
  • 14. Idempotent producer • Pulsar broker detects and discards messages that are being retransmitted • It works when a broker crashes and topic is reassigned • It works when a producer application crashes 14
  • 15. Identifying producers • Use “sequence ids” to detect retransmissions • Each producer on a topic has it own sequence of messages • Use “producer-name” to identify producers 15
  • 22. Sequence Id snapshot • Snapshots are taken every N entries to limit recovery time • Snapshot & cursor updates are atomic • Cursor updates are stored in BookKeeper — durable & replicated • On recovery • Load the snapshot from the cursor • Replay the entries from the cursor position 22
  • 23. What if application producer crashes? • Pulsar needs to identify the new producer as being the same “logical” producer as before • In practice, this is only useful if you have a “replayable” source (eg: file, stream, …) 23
  • 24. Resuming a producer session ProducerConfiguration conf = new ProducerConfiguration(); conf.setProducerName("my-producer-name"); conf.setSendTimeout(0, TimeUnit.SECONDS); Producer producer = client.createProducer(MY_TOPIC, conf); // Get last committed sequence id before crash long lastSequenceId = producer.getLastSequenceId(); 24
  • 25. Using sequence Ids // Fictitious record reader class RecordReader source = new RecordReader("/my/file/path"); long fileOffset = producer.getLastSequenceId(); source.seekToOffset(fileOffset); while (source.hasNext()) { long currentOffset = source.currentOffset(); Message msg = MessageBuilder.create() .setSequenceId(currentOffset) .setContent(source.next()).build(); producer.send(msg); } 25
  • 26. Consuming messages only once • Pulsar Consumer API is very convenient • Managed subscription — tracking individual messages Consumer consumer = client.subscribe(MY_TOPIC, MY_SUBSCRIPTION_NAME); while (true) { Message msg = consumer.receive(); // Process the message... consumer.acknowledge(msg); } 26
  • 27. Effectively-once with Consumer • Consumer is very simple but doesn’t allow a large degree of control • Processing and acknowledge are not atomic • To achieve “effectively once” we need to rely on an external system to deduplicate the processing results. Eg: • RDBMS — Keep the message id as a column with a “unique” index • Critical write to update the state — compareAndSet() or similar 27
  • 28. Pulsar Reader • Reader is a low level API to receive data from a Pulsar topic • There is no managed subscription • Application always specifies the message id where it wants to start reading from 28
  • 29. Reader example MessageId lastMessageId = recoverLastMessageIdFromDB(); Reader reader = client.createReader(MY_TOPIC, lastMessageId, new ReaderConfiguration()); while (true) { Message msg = reader.readNext(); byte[] msgId = msg.getMessageId().toByteArray(); // Process the message and store msgId atomically } 29
  • 30. Example — Pulsar Functions 30
  • 31. Pulsar Functions • A function gets messages from 1 or more topics • An instance of the function is invoked to process the event • The output of the function is published on 1 or more topics • Super simple to use — No SDK required — Python example: def process(input): return input + '!' 31
  • 33. Effectively once with functions • Use the message id from source topic as sequence id for sink topic • Works with “Consumer” API • When consuming from multiple topics or partitions, creates 1 producer per each source topic/partition, to ensure monotonic sequence ids 33
  • 34. Performance • Pulsar approach guarantees deduplication in all failure scenarios • Overhead is minimal: 2 in memory hashmap updates • No reduction in throughput — No increased latency • Controllable increase in recovery time 34
  • 35. Performance — Benchmark OpenMessaging Benchmark 1 Topic / 1 Partition 1 Partition / 1 Consumer 1Kb msg 35
  • 36. Difference with Kafka approach 36 Kafka Pulsar Producer Idempotency Best-effort (in memory only) Guaranteed after crash Transactions 2 phase commit No transactions Dedup across producer sessions No Yes Dedup with geo- replication No Yes Throughput Lower (1 in-flight message/batch for ordering) Equal
  • 37. Curious to Learn More? • Apache Pulsar — https://pulsar.incubator.apache.org • Follow Us — @apache_pulsar • Streamlio blog — https://streaml.io/blog 37