Transaction preview of Apache Pulsar
•
3 likes•1,069 views
This document discusses transaction support in Apache Pulsar. It provides an overview of Pulsar and its existing semantics of at-most-once, at-least-once, and exactly-once delivery. It then introduces the concept of transactions to enable exactly-once semantics across multiple partitions and subscriptions. Key aspects discussed include the transaction coordinator, transaction buffer, and transactional subscription state. The transaction execution flow is explained step-by-step with an example of a cash transfer function between Alice and Bob using transactions in Pulsar.
Report
Share
![CoordinatorBroker-0 Broker-1
InputTopic OutputTopic-1 OutputTopic-2
Cursor
Transaction Log
Data Log
Txn Buffer
Data Log
Txn Buffer
Pulsar Client
Input
Consumer
Producer 1 Producer 2
Txn
New Txn
2.0 Add Produced Topics
To Txn
Tx1
Tx1: add [T1, T2] Tx1: M1 Tx1: M2
2.1 Produced Messages
To Topics with Txn](https://cdn.statically.io/img/image.slidesharecdn.com/transaction-preview-191202070031/85/Transaction-preview-of-Apache-Pulsar-64-320.jpg)
![CoordinatorBroker-0 Broker-1
InputTopic OutputTopic-1 OutputTopic-2
Cursor
Transaction Log
Data Log
Txn Buffer
Data Log
Txn Buffer
Pulsar Client
Input
Consumer
Producer 1 Producer 2
Txn
New Txn
3.0 Add Acked Subscriptions To Txn
Tx1
Tx1: add [T1, T2] Tx1: M1 Tx1: M2
3.0 Ack messages with Txn
Tx1: ACK (M0)
Tx1: add [S0]](https://cdn.statically.io/img/image.slidesharecdn.com/transaction-preview-191202070031/85/Transaction-preview-of-Apache-Pulsar-66-320.jpg)
![CoordinatorBroker-0 Broker-1
InputTopic OutputTopic-1 OutputTopic-2
Cursor
Transaction Log
Data Log
Txn Buffer
Data Log
Txn Buffer
Pulsar Client
Input
Consumer
Producer 1 Producer 2
Txn
New Txn
4.0 Commit Txn
Tx1
Tx1: add [T1, T2] Tx1: M1 Tx1: M2
Tx1: ACK (M0)
Tx1: add [S0]
4.0 Committing Txn
Tx1: Committing](https://cdn.statically.io/img/image.slidesharecdn.com/transaction-preview-191202070031/85/Transaction-preview-of-Apache-Pulsar-68-320.jpg)
![CoordinatorBroker-0 Broker-1
InputTopic OutputTopic-1 OutputTopic-2
Cursor
Transaction Log
Data Log
Txn Buffer
Data Log
Txn Buffer
Pulsar Client
Input
Consumer
Producer 1 Producer 2
Txn
New Txn
Tx1
Tx1: add [T1, T2] Tx1: M1 Tx1: M2
Tx1: ACK (M0)
Tx1: add [S0]
4.1.0 Commit Txn
On Topics
4.1.1 Commit Txn
On Subscriptions
Tx1 (c) Tx1 (c)
Tx1: Committing
Tx1: Committed
Tx1: Committed](https://cdn.statically.io/img/image.slidesharecdn.com/transaction-preview-191202070031/85/Transaction-preview-of-Apache-Pulsar-69-320.jpg)
![CoordinatorBroker-0 Broker-1
InputTopic OutputTopic-1 OutputTopic-2
Cursor
Transaction Log
Data Log
Txn Buffer
Data Log
Txn Buffer
Pulsar Client
Input
Consumer
Producer 1 Producer 2
Txn
New Txn
Tx1
Tx1: add [T1, T2] Tx1: M1 Tx1: M2
Tx1: ACK (M0)
Tx1: add [S0]
Tx1: Committing
Tx1 (c) Tx1 (c)
Tx1: Committed
Tx1: Committed
4.2 Committed Txn](https://cdn.statically.io/img/image.slidesharecdn.com/transaction-preview-191202070031/85/Transaction-preview-of-Apache-Pulsar-70-320.jpg)
Transaction preview of Apache Pulsar
- 1. Penghui Li Apache Pulsar PMC Member Transaction Support in Pulsar Yong Zhang Apache Pulsar Contributor
- 2. What is Apache Pulsar?
- 4. “Flexible Pub/Sub messaging backed by durable log/stream storage”
- 6. 2012: Pulsar idea started at Yahoo! 5 years on production, 100+ applications, 10+ data centers 2016/09 Yahoo open sourced Pulsar 2017/06 Yahoo donated Pulsar to ASF 2018/09 Pulsar graduated as a Top-Level project 2018/09 InfoWorld Best Open Source Project
- 9. • At-most once • At-least once • Exactly once Messaging Semantics
- 10. • At-most once • At-least once • Exactly once Messaging Semantics Before 1.20.0-incubating
- 11. • At-most once • At-least once • Exactly once Messaging Semantics PIP-6: Guaranteed Message Deduplication
- 21. Pulsar’s Existing Semantics Log BrokerProducer m1 m2 ack(m2) What do we do now?
- 25. • Broker can fail • The request from Producer to Broker can fail • Producer or Consumer can fail Why the duplicates are introduced?
- 27. • Producer: Idempotent Producer • Broker: Guaranteed Message Deduplication (PIP-6) • Consumer: Reader + Checkpoints (Flink / Spark) Message Deduplication
- 28. • Producer Name - Identify who is producing the messages • Sequence ID - Identify the message • Producer Name + Sequence ID: The unique identifier for a message Idempotent Producer
- 29. • Broker maintains a map between Producer Name and Last- Produced-Sequence-ID • Broker accepts messages if the sequence id of a new message is larger than its last produced sequence id • Broker treats messages whose sequence id are smaller • Broker keeps the map in a de-duplication cursor (stored in bookkeeper) Guaranteed Message Deduplication
- 33. Exactly Once Log BrokerProducer 1,m1 2,m2 ack(2, m2) What do we do now?
- 38. • `bin/pulsar-admin set-deduplication -e tenant/namespace` • Set producer name when creating a Producer • Specify increasing sequence id when producing messages Enable Exactly Once
- 39. • It only works when producing messages to one partition • It only works for producing one message • There is no atomicity when producing multiple messages to one partition or many partitions • Consumers are required to store the MessageId along with its state and seek back to the MessageId when restoring the state Limitations
- 41. PulsarCash
- 43. • Transfer Topic : record the transfer requests • Cash Transfer Function: perform the cash transfer action • BalanceUpdate Topic: record the balance-update requests PulsarCash, powered by Apache Pulsar
- 44. PulsarCash Cash Transfer Function Balance user:alice, debit($10) balance update balance update user:bob, credit($10) (100,0,0): transfer($10, alice -> bob) Transfer Topic
- 45. Ack Transfer Cash Transfer function Balance user:alice, debit($10) balance update balance update user:bob, credit($10) (100,0,0): transfer($10, alice -> bob) Ack: (100,0,0)
- 46. Reprocessed Transfer! Cash Transfer function Balance user:alice, debit($10) balance update balance update user:bob, credit($10) (100,0,0): transfer($10, alice -> bob) Ack: (100,0,0)
- 47. Lost Money! Cash Transfer function Balance user:alice, debit($10) balance update balance update user:bob, credit($10) (100,0,0): transfer($10, alice -> bob) Ack: (100,0,0)
- 49. • Atomic writes across multiple partitions • Atomic acknowledges across multiple subscriptions • All the actions made within one transaction either all succeed or all fail • Consumers are *ONLY* allowed to read committed messages Transaction Semantics
- 50. Message<String> message = inputConsumer.receive(); CompletableFuture<MessageId> sendFuture1 = producer1.newMessage().value(“output-message-1”).sendAsync(); CompletableFuture<MessageId> sendFuture2 = producer2.newMessage().value(“output-message-2”).sendAsync(); inputConsumer.acknowledgeAsync(message.getMessageId()); Without Transaction API
- 51. Broker-0 Broker-1 InputTopic OutputTopic-1 OutputTopic-2 Cursor Data Log Data Log Pulsar Client Input Consumer Producer 1 Producer 2 0) Receive Message 1) Produce Messages 2) Ack Messages
- 52. Message<String> message = inputConsumer.receive(); Transaction txn = client.newTransaction().withTransactionTimeout(…).build().get(); CompletableFuture<MessageId> sendFuture1 = producer1.newMessage(txn).value(“output-message-1”).sendAsync(); CompletableFuture<MessageId> sendFuture2 = producer2.newMessage(txn).value(“output-message-2”).sendAsync(); inputConsumer.acknowledgeAsync(message.getMessageId(), txn); txn.commit().get(); MessageId msgId1 = sendFuture1.get(); MessageId msgId2 = sendFuture2.get(); Transaction API
- 53. CoordinatorBroker-0 Broker-1 InputTopic OutputTopic-1 OutputTopic-2 Cursor Transaction Log Data Log Txn Buffer Data Log Txn Buffer Pulsar Client Input Consumer Producer 1 Producer 2
- 54. CoordinatorBroker-0 Broker-1 InputTopic OutputTopic-1 OutputTopic-2 Cursor Transaction Log Data Log Txn Buffer Data Log Txn Buffer Pulsar Client Input Consumer Producer 1 Producer 2
- 55. • TC: transaction manager, coordinating committing and aborting transactions • In-Memory + Transaction Log • Transaction Log is powered by a partitioned Pulsar topic • `pulsar/system/__transaction_coordinator_log` • Locating a TC is locating a partition of the transaction log topic Transaction Coordinator (TC)
- 56. CoordinatorBroker-0 Broker-1 InputTopic OutputTopic-1 OutputTopic-2 Cursor Transaction Log Data Log Txn Buffer Data Log Txn Buffer Pulsar Client Input Consumer Producer 1 Producer 2
- 57. • TB: store and index transaction data per topic partition • TB is implemented using another ML (managed-ledger) as TB log • Messages are appended to into TB log • Transaction Index is maintained in memory and snapshotted to ledgers • Transaction Index can be replayed from TB log Transaction Buffer (TB)
- 58. CoordinatorBroker-0 Broker-1 InputTopic OutputTopic-1 OutputTopic-2 Cursor Transaction Log Data Log Txn Buffer Data Log Txn Buffer Pulsar Client Input Consumer Producer 1 Producer 2
- 59. • Introduce ACK_PENDING state • Add response for acknowledgement, aka ack-on-ack • Ack state is updated to cursor ledger • Ack state can be replayed from cursor ledger Transactional Subscription State
- 61. Message<String> message = inputConsumer.receive(); Transaction txn = client.newTransaction().withTransactionTimeout(…).build().get(); CompletableFuture<MessageId> sendFuture1 = producer1.newMessage(txn).value(“output-message-1”).sendAsync(); CompletableFuture<MessageId> sendFuture2 = producer2.newMessage(txn).value(“output-message-2”).sendAsync(); inputConsumer.acknowledgeAsync(message.getMessageId(), txn); txn.commit().get(); MessageId msgId1 = sendFuture1.get(); MessageId msgId2 = sendFuture2.get(); Transaction API - New Transaction
- 62. CoordinatorBroker-0 Broker-1 InputTopic OutputTopic-1 OutputTopic-2 Cursor Transaction Log Data Log Txn Buffer Data Log Txn Buffer Pulsar Client Input Consumer Producer 1 Producer 2 Txn New Txn 1. New Txn Tx1
- 63. Message<String> message = inputConsumer.receive(); Transaction txn = client.newTransaction().withTransactionTimeout(…).build().get(); CompletableFuture<MessageId> sendFuture1 = producer1.newMessage(txn).value(“output-message-1”).sendAsync(); CompletableFuture<MessageId> sendFuture2 = producer2.newMessage(txn).value(“output-message-2”).sendAsync(); inputConsumer.acknowledgeAsync(message.getMessageId(), txn); txn.commit().get(); MessageId msgId1 = sendFuture1.get(); MessageId msgId2 = sendFuture2.get(); Transaction API - Produce Messages
- 64. CoordinatorBroker-0 Broker-1 InputTopic OutputTopic-1 OutputTopic-2 Cursor Transaction Log Data Log Txn Buffer Data Log Txn Buffer Pulsar Client Input Consumer Producer 1 Producer 2 Txn New Txn 2.0 Add Produced Topics To Txn Tx1 Tx1: add [T1, T2] Tx1: M1 Tx1: M2 2.1 Produced Messages To Topics with Txn
- 65. Message<String> message = inputConsumer.receive(); Transaction txn = client.newTransaction().withTransactionTimeout(…).build().get(); CompletableFuture<MessageId> sendFuture1 = producer1.newMessage(txn).value(“output-message-1”).sendAsync(); CompletableFuture<MessageId> sendFuture2 = producer2.newMessage(txn).value(“output-message-2”).sendAsync(); inputConsumer.acknowledgeAsync(message.getMessageId(), txn); txn.commit().get(); MessageId msgId1 = sendFuture1.get(); MessageId msgId2 = sendFuture2.get(); Transaction API - Acknowledges
- 66. CoordinatorBroker-0 Broker-1 InputTopic OutputTopic-1 OutputTopic-2 Cursor Transaction Log Data Log Txn Buffer Data Log Txn Buffer Pulsar Client Input Consumer Producer 1 Producer 2 Txn New Txn 3.0 Add Acked Subscriptions To Txn Tx1 Tx1: add [T1, T2] Tx1: M1 Tx1: M2 3.0 Ack messages with Txn Tx1: ACK (M0) Tx1: add [S0]
- 67. Message<String> message = inputConsumer.receive(); Transaction txn = client.newTransaction().withTransactionTimeout(…).build().get(); CompletableFuture<MessageId> sendFuture1 = producer1.newMessage(txn).value(“output-message-1”).sendAsync(); CompletableFuture<MessageId> sendFuture2 = producer2.newMessage(txn).value(“output-message-2”).sendAsync(); inputConsumer.acknowledgeAsync(message.getMessageId(), txn); txn.commit().get(); MessageId msgId1 = sendFuture1.get(); MessageId msgId2 = sendFuture2.get(); Transaction API - Commit
- 68. CoordinatorBroker-0 Broker-1 InputTopic OutputTopic-1 OutputTopic-2 Cursor Transaction Log Data Log Txn Buffer Data Log Txn Buffer Pulsar Client Input Consumer Producer 1 Producer 2 Txn New Txn 4.0 Commit Txn Tx1 Tx1: add [T1, T2] Tx1: M1 Tx1: M2 Tx1: ACK (M0) Tx1: add [S0] 4.0 Committing Txn Tx1: Committing
- 69. CoordinatorBroker-0 Broker-1 InputTopic OutputTopic-1 OutputTopic-2 Cursor Transaction Log Data Log Txn Buffer Data Log Txn Buffer Pulsar Client Input Consumer Producer 1 Producer 2 Txn New Txn Tx1 Tx1: add [T1, T2] Tx1: M1 Tx1: M2 Tx1: ACK (M0) Tx1: add [S0] 4.1.0 Commit Txn On Topics 4.1.1 Commit Txn On Subscriptions Tx1 (c) Tx1 (c) Tx1: Committing Tx1: Committed Tx1: Committed
- 70. CoordinatorBroker-0 Broker-1 InputTopic OutputTopic-1 OutputTopic-2 Cursor Transaction Log Data Log Txn Buffer Data Log Txn Buffer Pulsar Client Input Consumer Producer 1 Producer 2 Txn New Txn Tx1 Tx1: add [T1, T2] Tx1: M1 Tx1: M2 Tx1: ACK (M0) Tx1: add [S0] Tx1: Committing Tx1 (c) Tx1 (c) Tx1: Committed Tx1: Committed 4.2 Committed Txn
- 71. inputConsumer.receiveAsync().thenCompose(message -> { return client.newTransaction().withTransactionTimeout(…).build().thenCompose(txn -> { producer1.newMessage(txn).value(“output-message-1”).sendAsync(); producer2.newMessage(txn).value(“output-message-2”).sendAsync(); inputConsumer.acknowledgeAsync(message.getMessageId(), txn); return txn.commit(); }); }) Transaction API - Async Example
- 72. PulsarCash Cash Transfer function Balance user:alice, debit($10) balance update balance update user:bob, credit($10) (100,0,0): transfer($10, alice -> bob)Ack: (100,0,0)
- 73. PulsarCash Cash Transfer function Balance user:alice, debit($10) balance update balance update user:bob, credit($10) (100,0,0): transfer($10, alice -> bob)Ack: (100,0,0) Transaction
- 74. Make Event Streaming easy, simple, and reliable for everyone Pulsar Transaction
- 75. Available to use in Pulsar 2.6.0 When is it available?
- 76. • Transaction support in other languages (e.g. C++, Go) • Transaction in Pulsar Functions & Pulsar IO • Transaction in Kafka-on-Pulsar (KOP) • Transaction for Flink / Spark job • Transaction for State storage in Pulsar Functions • … Roadmap
- 77. • Ivan Kelly • Matteo Merli • Jia Zhai • Penghui Li • Marvin Cai • Yong Zhang • … and many other Pulsar users & contributors Credits
- 78. Wechat Subscription: ApachePulsar Mailing Lists dev@pulsar.apache.org, users@pulsar.apache.org Slack https://apache-pulsar.slack.com (#china) register: https://apache-pulsar.herokuapp.com/ https://github.com/apache/pulsar https://github.com/apache/bookkeeper
- 79. Thanks! Penghui Li Yong Zhang