2024-08-01

Broker

• [feat][broker] PIP-264: Add OpenTelemetry consumer metrics (#22693)
• [feat][broker] PIP-321 Introduce allowed-cluster at the namespace level (#22378) (#22961)
• [fix] Make operations on individualDeletedMessages in lock scope (#22966)
• [fix] Remove blocking calls from BookieRackAffinityMapping (#22846)
• [fix][admin] Fix half deletion when attempt to topic with a incorrect API (#23002)
• [fix][broker] Asynchronously return brokerRegistry.lookupAsync when checking if broker is active(ExtensibleLoadManagerImpl only) (#22899)
• [fix][broker] Can't connecte to non-persist topic when enable broker client tls (#22991)
• [fix][broker] Check the markDeletePosition and calculate the backlog (#22947)
• [fix][broker] Ensure that PulsarService is ready for serving incoming requests (#22977)
• [fix][broker] EntryFilters fix NoClassDefFoundError due to closed classloader (#22767)
• [fix][broker] Fix MessageDeduplication replay timeout cause topic loading stuck (#23004)
• [fix][broker] Fix NPE after publishing a tombstone to the service unit channel (#22859)
• [fix][broker] Fix ProducerBusy issue due to incorrect userCreatedProducerCount on non-persistent topic (#22685)
• [fix][broker] Fix broker OOM when upload a large package. (#22989)
• [fix][broker] Fix cursor should use latest ledger config (#22644)
• [fix][broker] Fix geo-replication admin client url (#22584)
• [fix][broker] Fix stuck when enable topic level replication and build remote admin fails (#23028)
• [fix][broker] Fix topic status for oldestBacklogMessageAgeSeconds continuously increases even when there is no backlog. (#22907)
• [fix][broker] Fix updatePartitionedTopic when replication at ns level and topic policy is set (#22971)
• [fix][broker] Handle BucketDelayedDeliveryTracker recover failed (#22735)
• [fix][broker] Immediately tombstone Deleted and Free state bundles (#22743)
• [fix][broker] Messages lost on the remote cluster when using topic level replication (#22890)
• [fix][broker] PulsarStandalone started with error if --stream-storage-port is not 4181 (#22993)
• [fix][broker] Remove blocking calls from Subscription.getStats (#23088)
• [fix][broker] Replication stuck when partitions count between two clusters is not the same (#22983)
• [fix][broker] Support advertised listeners when gracefully transferring topics (ExtensibleLoadManagerImpl only) (#22862)
• [fix][broker] The topic might reference a closed ledger (#22860)
• [fix][broker] Update init and shutdown time and other minor logic (ExtensibleLoadManagerImpl only) (#22930)
• [fix][broker] broker log a full thread dump when a deadlock is detected in healthcheck every time (#22916)
• [fix][broker] disable loadBalancerDirectMemoryResourceWeight by default (#22821)
• [fix][broker] fix deadlock when disable topic level Geo-Replication (#22738)
• [fix][broker] fix replicated subscriptions for transactional messages (#22452)
• [fix][broker] fix topic partitions was expanded even if disabled topic level replication (#22769)
• [fix][broker] maintain last active info in memory only. (#22794)
• [fix][broker] rename to changeMaxReadPositionCount (#22656)
• [fix][broker] response not-found error if topic does not exist when calling getPartitionedTopicMetadata (#22838)
• [fix][broker] usedLocallySinceLastReport should always be reset (#22672)
• [fix][broker]Fix lookupService.getTopicsUnderNamespace can not work with a quote pattern (#23014)
• [fix][broker][branch-3.3] Do not try to clean owned bundles from inactive source brokers (ExtensibleLoadManagerImpl only) (#23064) (#23085)
• [fix][misc] Disable JFR based telemetry collection since it's not used (#22869)
• [fix][misc] Topic name from persistence name should decode local name (#22879)
• [fix][ml] Add entry fail due to race condition about add entry failed/timeout and switch ledger (#22221)
• [fix][ml] Fix race conditions in RangeCache (#22789)
• [improve][ml] RangeCache refactoring: test race conditions and prevent endless loops (#22814)
• [improve][ml] RangeCache refactoring follow-up: use StampedLock instead of synchronized (#22818)
• [fix][ml]: subscription props could be lost in case of missing ledger during recovery (#22637)
• [improve] Refactored BK ClientFactory to return futures (#22853)
• [improve] Validate range of argument before long -> int conversion (#22830)
• [improve][broker] Add additionalSystemCursorNames ignore list for TTL check (#22614)
• [improve][broker] Add subscription prefix for internal reader (#23044)
• [improve][broker] Check max producers/consumers limitation first before other ops to save resources (#23074)
• [improve][broker] Close protocol handlers before unloading namespace bundles (#22728)
• [improve][broker] Do not call cursor.isCursorDataFullyPersistable if disabled dispatcherPauseOnAckStatePersistentEnabled (#22729)
• [improve][broker] GetPartitionMetadata fail also can produce messages (#23050)
• [improve][broker] Improve CPU resources usege of TopicName Cache (#23052)
• [improve][broker] Improve exception for topic does not have schema to check (#22974)
• [improve][broker] PIP-355: Enhancing Broker-Level Metrics for Pulsar (#22779)
• [improve][broker] PIP-356 Support Geo-Replication starts at earliest position (#22856)
• [improve][broker] Reduce number of OpenTelemetry consumer attributes (#22837)
• [improve][broker] Reduce the CPU pressure from the transaction buffer in rolling restarts (#23062)
• [improve][broker] Remove ClassLoaderSwitcher to avoid objects allocations and consistent the codestyle (#22796)
• [improve][broker] Reuse topic OpenTelemetry attributes (#22876)
• [improve][broker] Trigger offload on topic load (#22652)
• [improve][broker] Use RoaringBitmap in tracking individual acks to reduce memory usage (#23006)
• [improve][broker] avoid creating new objects when intercepting (#22790)
• [improve][broker] high CPU usage caused by list topics under namespace (#23049)
• [improve][broker] make system topic distribute evenly. (#22953)
• [improve][broker]Ensure namespace deletion doesn't fail (#22627)
• [improve][broker][branch-3.0] PIP-364: Introduce a new load balance algorithm AvgShedder (#23053)

Client

• [fix][client] Fix negative acknowledgement by messageId (#23060)
• [fix][client] Fix orphan consumer when reconnection and closing are concurrency executing (#22958)
• [fix][client] Fix pattern consumer create crash if a part of partitions of a topic have been deleted ([#22854](#22...

2024-08-01

Broker

• [fix][broker] Asynchronously return brokerRegistry.lookupAsync when checking if broker is active(ExtensibleLoadManagerImpl only) (#22899)
• [fix][broker] Check the broker is available for the SLA monitor bundle when the ExtensibleLoadManager is enabled (#22485)
• [fix][broker] Check the markDeletePosition and calculate the backlog (#22947)
• [fix][broker] Ensure that PulsarService is ready for serving incoming requests (#22977)
• [fix][broker] EntryFilters fix NoClassDefFoundError due to closed classloader (#22767)
• [fix][broker] Fix MessageDeduplication replay timeout cause topic loading stuck (#23004)
• [fix][broker] Fix NPE after publishing a tombstone to the service unit channel (#22859)
• [fix][broker] Fix Replicated Topic unload bug when ExtensibleLoadManager is enabled (#22496)
• [fix][broker] Fix broker OOM when upload a large package. (#22989)
• [fix][broker] Fix cursor should use latest ledger config (#22644)
• [fix][broker] Fix geo-replication admin client url (#22584)
• [fix][broker] Fix stuck when enable topic level replication and build remote admin fails (#23028)
• [fix][broker] Fix updatePartitionedTopic when replication at ns level and topic policy is set (#22971)
• [fix][broker] Immediately tombstone Deleted and Free state bundles (#22743)
• [fix][broker] Make ExtensibleLoadManagerImpl.getOwnedServiceUnits async (#22727)
• [fix][broker] Messages lost on the remote cluster when using topic level replication (#22890)
• [fix][broker] PulsarStandalone started with error if --stream-storage-port is not 4181 (#22993)
• [fix][broker] Support advertised listeners when gracefully transferring topics (ExtensibleLoadManagerImpl only) (#22862)
• [fix][broker] Support lookup options for extensible load manager (#22487)
• [fix][broker] The topic might reference a closed ledger (#22860)
• [fix][broker] Update init and shutdown time and other minor logic (ExtensibleLoadManagerImpl only) (#22930)
• [fix][broker] broker log a full thread dump when a deadlock is detected in healthcheck every time (#22916)
• [fix][broker] fix deadlock when disable topic level Geo-Replication (#22738)
• [fix][broker] fix topic partitions was expanded even if disabled topic level replication (#22769)
• [fix][broker] maintain last active info in memory only. (#22794)
• [fix][broker]Fix lookupService.getTopicsUnderNamespace can not work with a quote pattern (#23014)
• [fix][broker][branch-3.2] Do not try to clean owned bundles from inactive source brokers (ExtensibleLoadManagerImpl only) (#23064) (#23084)
• [cleanup][ml] ManagedCursor clean up. (#22246)
• [fix] Make operations on individualDeletedMessages in lock scope (#22966)
• [fix] Remove blocking calls from BookieRackAffinityMapping (#22846)
• [fix][misc] Topic name from persistence name should decode local name (#22879)
• [fix][ml] Add entry fail due to race condition about add entry failed/timeout and switch ledger (#22221)
• [fix][ml] Fix race conditions in RangeCache (#22789)
• [improve][ml] RangeCache refactoring: test race conditions and prevent endless loops (#22814)
• [improve][ml] RangeCache refactoring follow-up: use StampedLock instead of synchronized (#22818)
• [fix][ml]: subscription props could be lost in case of missing ledger during recovery (#22637)
• [improve] Refactored BK ClientFactory to return futures (#22853)
• [improve][broker] Clear thread local BrokerEntryMetadata instance before reuse (#22752)
• [improve][broker] Close protocol handlers before unloading namespace bundles (#22728)
• [improve][broker] GetPartitionMetadata fail also can produce messages (#23050)
• [improve][broker] Improve exception for topic does not have schema to check (#22974)
• [improve][broker] Include runtime dependencies in server distribution (#22001)
• [improve][broker] Optimize PersistentTopic.getLastDispatchablePosition (#22707)
• [improve][broker] Remove ClassLoaderSwitcher to avoid objects allocations and consistent the codestyle (#22796)
• [improve][broker] Trigger offload on topic load (#22652)
• [improve][broker] Use RoaringBitmap in tracking individual acks to reduce memory usage (#23006)
• [improve][broker] avoid creating new objects when intercepting (#22790)
• [improve][broker] make system topic distribute evenly. (#22953)
• [improve][broker]Ensure namespace deletion doesn't fail (#22627)
• [improve][broker][branch-3.0] PIP-364: Introduce a new load balance algorithm AvgShedder (#23053)

Client

• [fix][client] Fix negative acknowledgement by messageId (#23060)
• [fix][client] Fix orphan consumer when reconnection and closing are concurrency executing (#22958)
• [fix][client] Fix pattern consumer create crash if a part of partitions of a topic have been deleted (#22854)
• [fix][client] Fix resource leak in Pulsar Client since HttpLookupService doesn't get closed (#22858)
• [improve][client] improve the class GetTopicsResult (#22766)

Pulsar IO and Pulsar Functions

• [fix][fn] Enable optimized Netty direct byte buffer support for Pulsar Function runtimes (#22910)
• [fix][fn] Support compression type and crypto config for all producers in Functions and Connectors (#22950)
• [improve][fn] Make producer cache bounded and expiring in Functions/Connectors (#22945)

Others

• [fix][cli] Fix Pulsar standalone "--wipe-data" (#22885)
• [fix][cli] Fix Pulsar standalone shutdown - bkCluster wasn't closed (#22868)
• [fix][cli] Fix expiration of tokens created with "pulsar tokens create" (#22815)
• [fix][cli] Fix the pulsar-daemon parameter passthrough syntax (#22905)
• [fix][meta] Check if metadata store is closed in RocksdbMetadataStore (#22852)
• [fix][offload] Break the fillbuffer loop when met EOF (#22722)

Library updates

• [fix][sec] Upgrade Bouncycastle libraries to address CVEs (#22826)
• [improve] Upgrade IPAddress to 5.5.0 (#22886)
• [improve] Upgrade Jetcd to 0.7.7 and VertX to 4.5.8 (#22835)
• [cleanup][misc] Remove classifier from netty-transport-native-unix-common dependency (#22951)
• [fix][misc] Remove RoaringBitmap dependency from pulsar-common (#23008)
• [fix][misc] Rename netty native libraries in pulsar-client-admin-shaded (#22954)
• [improve][misc] Replace rename-netty-native-libs.sh script with renaming with maven-shade-plugin (#22957)
• [improve][misc] Upgrade to Netty 4.1.111.Final and switch to use grpc-netty-shaded (#22892)
• [improve][misc][branch-3.2] Upgrade to Bookkeeper 4.16.6 (#22963)

Tests...

2024-08-01

Broker

• [feat][broker] Implementation of PIP-323: Complete Backlog Quota Telemetry (#21816) (#22740)
• [feat][broker][branch-3.0] PIP-321 Introduce allowed-cluster at the namespace level (#22378) (#22960)
• [fix] Make operations on individualDeletedMessages in lock scope (#22966)
• [fix] Remove blocking calls from BookieRackAffinityMapping (#22846)
• [improve] Refactored BK ClientFactory to return futures (#22853)
• [fix][broker] Asynchronously return brokerRegistry.lookupAsync when checking if broker is active(ExtensibleLoadManagerImpl only) (#22899)
• [fix][broker] Can't connecte to non-persist topic when enable broker client tls (#22991)
• [fix][broker] Check the broker is available for the SLA monitor bundle when the ExtensibleLoadManager is enabled (#22485)
• [fix][broker] Check the markDeletePosition and calculate the backlog (#22947)
• [fix][broker] Ensure that PulsarService is ready for serving incoming requests (#22977)
• [fix][broker] EntryFilters fix NoClassDefFoundError due to closed classloader (#22767)
• [fix][broker] Fix MessageDeduplication replay timeout cause topic loading stuck (#23004)
• [fix][broker] Fix NPE after publishing a tombstone to the service unit channel (#22859)
• [fix][broker] Fix Replicated Topic unload bug when ExtensibleLoadManager is enabled (#22496)
• [fix][broker] Fix broker OOM when upload a large package. (#22989)
• [fix][broker] Fix cursor should use latest ledger config (#22644)
• [fix][broker] Fix geo-replication admin client url (#22584)
• [fix][broker] Fix stuck when enable topic level replication and build remote admin fails (#23028)
• [fix][broker] Fix topic status for oldestBacklogMessageAgeSeconds continuously increases even when there is no backlog. (#22907)
• [fix][broker] Fix updatePartitionedTopic when replication at ns level and topic policy is set (#22971)
• [fix][broker] Fix wrong logic of method TopicName.getPartition(int index) (#19841)
• [fix][broker] Handle BucketDelayedDeliveryTracker recover failed (#22735)
• [fix][broker] Immediately tombstone Deleted and Free state bundles (#22743)
• [fix][broker] Make ExtensibleLoadManagerImpl.getOwnedServiceUnits async (#22727)
• [fix][broker] Messages lost on the remote cluster when using topic level replication (#22890)
• [fix][broker] PulsarStandalone started with error if --stream-storage-port is not 4181 (#22993)
• [fix][broker] Remove blocking calls from Subscription.getStats (#23088)
• [fix][broker] Replication stuck when partitions count between two clusters is not the same (#22983)
• [fix][broker] Support lookup options for extensible load manager (#22487)
• [fix][broker] Update init and shutdown time and other minor logic (ExtensibleLoadManagerImpl only) (#22930)
• [fix][broker] broker log a full thread dump when a deadlock is detected in healthcheck every time (#22916)
• [fix][broker] fix deadlock when disable topic level Geo-Replication (#22738)
• [fix][broker] fix topic partitions was expanded even if disabled topic level replication (#22769)
• [fix][broker] response not-found error if topic does not exist when calling getPartitionedTopicMetadata (#22838)
• [fix][broker]Fix lookupService.getTopicsUnderNamespace can not work with a quote pattern (#23014)
• [fix][broker][branch-3.0] Do not try to clean owned bundles from inactive source brokers (ExtensibleLoadManagerImpl only) (#23064) (#23077)
• [fix][broker][branch-3.0] The topic might reference a closed ledger (#22860) (#22900)
• [improve][broker] Check max producers/consumers limitation first before other ops to save resources (#23074)
• [improve][broker] Clear thread local BrokerEntryMetadata instance before reuse (#22752)
• [improve][broker] Follow up #4196 use PulsarByteBufAllocator handle OOM (#20837)
• [improve][broker] GetPartitionMetadata fail also can produce messages (#23050)
• [improve][broker] Improve CPU resources usege of TopicName Cache (#23052)
• [improve][broker] Improve efficiency of checking message deletion (#20490)
• [improve][broker] Improve exception for topic does not have schema to check (#22974)
• [improve][broker] Include runtime dependencies in server distribution (#22001)
• [improve][broker] Optimize PersistentTopic.getLastDispatchablePosition (#22707)
• [improve][broker] PIP-356 Support Geo-Replication starts at earliest position (#22856)
• [improve][broker] Remove ClassLoaderSwitcher to avoid objects allocations and consistent the codestyle (#22796)
• [improve][broker] Use RoaringBitmap in tracking individual acks to reduce memory usage (#23006)
• [improve][broker] [break change] Do not create partitioned DLQ/Retry topic automatically (#22705)
• [improve][broker] avoid creating new objects when intercepting (#22790)
• [improve][broker] checkTopicExists supports checking partitioned topic without index (#21701)
• [improve][broker] do not grant permission for each partition to reduce unnecessary zk metadata (#18222)
• [improve][broker] high CPU usage caused by list topics under namespace (#23049)
• [improve][broker] make system topic distribute evenly. (#22953)
• [improve][broker]Ensure namespace deletion doesn't fail (#22627)
• [improve][broker][branch-3.0] PIP-364: Introduce a new load balance algorithm AvgShedder (#23053)
• [cleanup][ml] ManagedCursor clean up. (#22246)
• [fix][ml] Add entry fail due to race condition about add entry failed/timeout and switch ledger (#22221)
• [fix][ml]: subscription props could be lost in case of missing ledger during recovery (#22637)
• [fix][ml] Fix race conditions in RangeCache (#22789)
• [improve][ml] RangeCache refactoring: test race conditions and prevent endless loops (#22814)
• [improve][ml] RangeCache refactoring follow-up: use StampedLock instead of synchronized (#22818)
• [fix][admin] Clearly define REST API on Open API (#22783)
• [fix][admin] Clearly define REST API on Open API for Namesaces@v2 (#22775)
• [fix][admin] Clearly define REST API on Open API for Topics (#22782)
• [fix][admin] Fix can't delete tenant for v1 (#22550)
• [fix][admin] Fix half deletion when attempt to topic with a incorrect API (#23002)
• [fix][admin][part-1]Clearly define REST API on Open API (#22774)
• [improve][admin] Check if the topic existed before the permission operations (#22742)

Client

• [fix][client] Fix negative acknowledgement by messageId (#23060)
• [fix][client] Fix o...

PIP

• PIP-315: Configurable max delay limit for delayed delivery (#21798)
• PIP-321 Introduce allowed-cluster at the namespace level (#21648)
• PIP-324: Alpine Docker images (#21716)
• PIP-325: Add command to abort transaction (#21731)
• PIP-326: Create a BOM to ease dependency management (#21747)
• PIP-329: Strategy for maintaining the latest tag to Pulsar docker images (#21872)
• PIP-330: getMessagesById gets all messages (#21873)
• PIP 342: Support OpenTelemetry metrics in Pulsar client (#22178)
• PIP-343: Use picocli instead of jcommander (#22181)
• PIP-344 Correct the behavior of the public API pulsarClient.getPartitionsForTopic(topicName) (#22182)
• PIP-335: Oxia metadata plugin (#22009)
• PIP-339: Introducing the --log-topic Option for Pulsar Sinks and Sources (#22071)

Broker

• [admin][broker] Fix force delete subscription not working (#22423)
• [cleanup][admin] Remove unused methods in PersistentTopicsBase (#22424)
• [cleanup][broker] Remove unused NamespaceBundleFactory parameter when creating OwnershipCache (#22482)
• [cleanup][broker]∂ fix doc for TransactionBuffer#isTxnAborted (#21956)
• [cleanup][broker] fix return value of TransactionTimeoutTrackerImpl#addTransaction (#22020)
• [cleanup][broker] remove useless code comment (#22459)
• [cleanup][meta] Remove com.beust.jcommander.internal import (#22294)
• [cleanup][ml] ManagedCursor clean up. (#22246)
• [cleanup][test] remove useless TestAuthorizationProvider2 (#22595)
• [feat][admin] Enable Gzip Compression by Default in Admin Client (#22464)
• [feat][admin] PIP-330: getMessagesById gets all messages (#21918)
• [feat][broker] Implementation of PIP-323: Complete Backlog Quota Telemetry (#21816)
• [feat][broker] PIP-264: Add Java runtime metrics (#22616)
• [feat][broker] PIP-264: Add topic messaging metrics (#22467)
• [feat][misc] Add Pulsar BOM (Bill of Materials) (#21871)
• [feat][misc] PIP-264: Implement topic lookup metrics using OpenTelemetry (#22058)
• [feat][misc] PIP-320: Add OpenTelemetry scaffolding (#22010)
• [fix] Fix Reader can be stuck from transaction aborted messages. (#22610)
• [fix] Fixed implicit conversions from long -> int (#22055)
• [fix] Include swagger annotations in shaded client lib (#22570)
• [fix] Restored method as deprecated in AbstractMetadataStore (#21950)
• [fix] Test was leaving client instance to null (#22631)
• [fix][admin] Clearly define REST API on Open API (#22783)
• [fix][admin] Clearly define REST API on Open API for Namesaces@v2 (#22775)
• [fix][admin] Clearly define REST API on Open API for Topics (#22782)
• [fix][admin] Fix can't delete tenant for v1 (#22550)
• [fix][admin] Fix deprecated check (#22653)
• [fix][admin] Fix namespace admin api exception response (#22587)
• [fix][admin][part-1]Clearly define REST API on Open API (#22774)
• [fix][broker] Add the missed opentelemetry-sdk-testing dependency to tests of pulsar-broker-auth-sasl (#22665)
• [fix][broker] Avoid being stuck when closing the broker with extensible load manager (#22573)
• [fix][broker] Avoid consumers receiving acknowledged messages from compacted topic after reconnection (#21187)
• [fix][broker] Avoid execute prepareInitPoliciesCacheAsync if namespace is deleted (#22268)
• [fix][broker] Avoid expired unclosed ledgers when checking expired messages by ledger closure time (#22335)
• [fix][broker] Check cursor state before adding it to the waitingCursors (#22191)
• [fix][broker] Check the broker is available for the SLA monitor bundle when the ExtensibleLoadManager is enabled (#22485)
• [fix][broker] Close dispatchers stuck due to mismatch between dispatcher.consumerList and dispatcher.consumerSet (#22270)
• [fix][broker] Continue closing even when executor is shut down (#22599)
• [fix][broker] Create new ledger after the current ledger is closed (#22034)
• [fix][broker] Delete compacted ledger when topic is deleted (#21745)
• [fix][broker] Disable system topic message deduplication (#22582)
• [fix][broker] Do not migrate internal topics during Blue-Green Migration when ExtensibleLoadBalancer is used (#22478)
• [fix][broker] Enabling batch causes negative unackedMessages due to ack and delivery concurrency (#22090)
• [fix][broker] Expire messages according to ledger close time to avoid client clock skew (#21940)
• [fix][broker] Fix BufferOverflowException and EOFException bugs in /metrics gzip compression (#22576)
• [fix][broker] Fix NPE causing dispatching to stop when using Key_Shared mode and allowOutOfOrderDelivery=true (#22533)
• [fix][broker] Fix OpReadEntry.skipCondition NPE issue (#22367)
• [fix][broker] Fix PersistentSubscription duplicate implementation interface Subscription (#22359)
• [fix][broker] Fix Replicated Topic unload bug when ExtensibleLoadManager is enabled (#22496)
• [fix][broker] Fix ResourceGroup report local usage (#22340)
• [fix][broker] Fix ResourceGroups loading (#21781)
• [fix][broker] Fix String wrong format (#21829)
• [fix][broker] Fix a deadlock in SystemTopicBasedTopicPoliciesService during NamespaceEventsSystemTopicFactory init (#22528)
• [fix][broker] Fix break change: could not subscribe partitioned topic with a suffix-matched regexp due to a mistake of PIP-145 (#21885)
• [fix][broker] Fix broken topic policy implementation compatibility with old pulsar version (#22535)
• [fix][broker] Fix broker not starting when both transactions and the Extensible Load Manager are enabled (#22139)
• [fix][broker] Fix can not subscribe partitioned topic with a suffix-matched regexp (#22025)
• [fix][broker] Fix compaction/replication data loss when expire messages (#21865)
• [fix][broker] Fix configurationMetadataSyncEventTopic is marked supporting dynamic setting, but not implemented (#22684)
• [fix][broker] Fix consumer stops receiving messages when with large backlogs processing (#22454)
• [fix][broker] Fix deadlock while skip non-recoverable ledgers. (#21915)
• [fix][broker] Fix getMessageById throws 500 (#21919)
• [fix][broker] Fix hash collision when using a consumer name that ends with a number (#22053)
• [fix][broker] Fix invalid condition in logging exceptions (#22412)
• [fix][broker] Fix issue of field 'topic' is not set when handle GetSchema request (#22377)
• [fix][broker] Fix issue with GetMessageIdByTimestamp can't find match messageId from compacted ledger (#21600)
• [fix][broker] Fix leader broker cannot be determined when the advertised address and advertised listeners are configured (#21894)
• [fix][broker] Fix message drop record in producer stat (#22458)
• [fix][broker] Fix metrics pulsar_topic_load_failed_count is 0 when load non-persistent topic fails and fix the flaky test testBrokerStatsTopicLoadFailed (#22580)
• [fix][broker] Fix nothing changed after removing dynamic configs (#22673)
• [fix][broker] Fix potential NPE when disabling the liveness check (#21840)
• [fix][broker] Fix reader stuck when read from compacted topic with read compact mode disable (#21969)
• [fix][broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set (#21842)
• [fix][broker] Fix schema deletion error when deleting a partitioned topic with many partitions and schema (#21977)
• [fix][broker] Fix thread safety of loadSheddingTask and loadResourceQuotaTask fields (#22660)
• [fix][broker] Fix typo (#21843)
• [fix][broker] Fix typo in CompactedTopicImpl. (#22235)
• [fix][broker] Fix typos in Consumer class (#22532)
• [fix][broker] Fix typos in PersistentTopic class (#22364)
• [fix][broker] Fix typos in UniformLoadShedder class (#21907)
• [fix][broker] Fix write all compacted out entry into compacted topic (#21917)
• [fix][broker] Fix wrong double-checked locking for readOnActiveConsumerTask in dispatcher (#22279)
• [fix][broker] Make ExtensibleLoadManagerImpl.getOwnedServiceUnits async (#22727)
• [fix][broker] One topic can be closed multiple times concurrently (#17524)
• [fix][broker] Optimize /metrics, fix unbounded request queue issue and fix race conditions in metricsBufferResponse mode (#22494)
• [fix][broker] Part-1: Replicator can not created successfully due to an orphan replicator in the previous topic owner (#21946)
• [fix][broker] Part-2: Replicator can not created successfully due to an orphan replicator in the previous topic owner (#21948)
• [fix][broker] Prevent long deduplication cursor backlog so that topic loading wouldn't timeout (#22479)
• [fix][broker] Reader stuck after call hasMessageAvailable when enable replicateSubscriptionState (#22572)
• [fix][broker] Replication stopped due to unload topic failed (#21947)
• [fix][broker] Restore the broker id to match the format used in existing Pulsar releases (#21937)
• [fix][broker] Return getOwnerAsync without waiting on source broker upon Assigning and Releasing and handle role change during role init (#22112)
• [fix][broker] Sanitize values before logging in apply-config-from-env.py script (#22044)
• [fix][broker] Set ServiceUnitStateChannel topic compaction threshold explicitly, improve getOwnerAsync, and fix other bugs (#22064)
• [fix][broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsync for ExtensibleLoadBalancer (#22379)
• [fix][broker] Subscription stuck due to called Admin API analyzeSubscriptionBacklog (#22019)
• [fix][broker] Support OIDC providers with JWK without alg field set in keys (#22421)
• [fix][broker] Support lookup options for extensible load manager (#22487)
• [fix][broker] Support running docker container with gid != 0 (#22081)
• [fix][broker] Unify topic-level policies enable judgment conditions (#19501)
• [fix][broker] Update TransferShedder underloaded broker check to consider max loaded broker's msgThroughputEMA and update IsExtensibleLoadBalancerImpl check (#22321)
• [fix][broker] Update topic partition failed when config maxNumPartitionsPerPartitionedTop...

Broker

• [fix][admin] Fix force delete subscription not working (#22423)
• [fix][broker] Fix Reader can be stuck from transaction aborted messages. (#22610)
• [fix][admin] Fix can't delete tenant for v1 (#22550)
• [fix][admin] Fix namespace admin api exception response (#22587)
• [fix][broker] Avoid being stuck when closing the broker with extensible load manager (#22573)
• [fix][broker] Create new ledger after the current ledger is closed (#22034)
• [fix][broker] Disable system topic message deduplication (#22582)
• [fix][broker] Do not migrate internal topics during Blue-Green Migration when ExtensibleLoadBalancer is used (#22478)
• [fix][broker] Fix BufferOverflowException and EOFException bugs in /metrics gzip compression (#22576)
• [fix][broker] Fix NPE causing dispatching to stop when using Key_Shared mode and allowOutOfOrderDelivery=true (#22533)
• [fix][broker] Fix ProducerBusy issue due to incorrect userCreatedProducerCount on non-persistent topic (#22685)
• [fix][broker] Fix a deadlock in SystemTopicBasedTopicPoliciesService during NamespaceEventsSystemTopicFactory init (#22528)
• [fix][broker] Fix broken topic policy implementation compatibility with old pulsar version (#22535)
• [fix][broker] Fix consumer stops receiving messages when with large backlogs processing (#22454)
• [fix][broker] Fix invalid condition in logging exceptions (#22412)
• [fix][broker] Fix message drop record in producer stat (#22458)
• [fix][broker] Fix metrics pulsar_topic_load_failed_count is 0 when load non-persistent topic fails and fix the flaky test testBrokerStatsTopicLoadFailed (#22580)
• [fix][broker] Fix nothing changed after removing dynamic configs (#22673)
• [fix][broker] Fix typos in Consumer class (#22532)
• [fix][broker] One topic can be closed multiple times concurrently (#17524)
• [fix][broker] Optimize /metrics, fix unbounded request queue issue and fix race conditions in metricsBufferResponse mode (#22494)
• [fix][broker] Part-1: Replicator can not created successfully due to an orphan replicator in the previous topic owner (#21946)
• [fix][broker] Part-2: Replicator can not created successfully due to an orphan replicator in the previous topic owner (#21948)
• [fix][broker] Prevent long deduplication cursor backlog so that topic loading wouldn't timeout (#22479)
• [fix][broker] Prevent long deduplication cursor backlog so that topic loading wouldn't timeout (#22479)
• [fix][broker] Reader stuck after call hasMessageAvailable when enable replicateSubscriptionState (#22572)
• [fix][broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsync for ExtensibleLoadBalancer (#22379) (#22407)
• [fix][broker] Support OIDC providers with JWK without alg field set in keys (#22421)
• [fix][broker] Update TransferShedder underloaded broker check to consider max loaded broker's msgThroughputEMA and update IsExtensibleLoadBalancerImpl check (#22321) (#22418)
• [fix][broker] Update topic partition failed when config maxNumPartitionsPerPartitionedTopic<0 (#22397)
• [fix][broker] avoid offload system topic (#22497)
• [fix][broker] fix replicated subscriptions for transactional messages (#22452)
• [fix][broker] rename to changeMaxReadPositionCount (#22656)
• [fix][broker] usedLocallySinceLastReport should always be reset (#22672)
• [fix][broker][admin] Fix cannot update properties on NonDurable subscription. (#22411)
• [improve][broker] Make the config metricsBufferResponse description more effective (#22490)
• [improve][meta] Retry re-validating ResourceLock with backoff after errors (#22617)
• [improve][admin] Check if the topic existed before the permission operations (#22547)
• [improve][broker] Add additionalSystemCursorNames ignore list for TTL check (#22614)
• [improve][broker] Add topic name to emitted error messages. (#22506)
• [improve][broker] Avoid repeated Read-and-discard when using Key_Shared mode (#22245)
• [improve][broker] Create partitioned topics automatically when enable topic level replication (#22537)
• [improve][broker] Don't log brokerClientAuthenticationParameters and bookkeeperClientAuthenticationParameters by default (#22395)
• [improve][broker] Improve Gzip compression, allow excluding specific paths or disabling it (#22370)
• [improve][broker] Optimize gzip compression for /metrics endpoint by sharing/caching compressed result (#22521)
• [improve][broker] Propagate cause exception in TopicBusyException when applicable (#22596)
• [improve][broker] Repeat the handleMetadataChanges callback when configurationMetadataStore equals localMetadataStore (#22519)
• [improve][broker] Servlet support response compression (#21667)
• [improve][broker] Support X-Forwarded-For and HA Proxy Protocol for resolving original client IP of http/https requests (#22524)
• [improve][broker] backlog quota exceed limit log replaced with debug (#22488)
• [fix][ml] Fix NPE of getValidPositionAfterSkippedEntries when recovering a terminated managed ledger (#22552)
• [fix][ml] Mark delete stuck due to switching cursor ledger fails (#22662)
• [fix][ml] No rollover inactive ledgers when metadata service invalid (#22284)
• [fix][offload] Fix OOM in tiered storage, caused by unbounded offsets cache (#22679)
• [fix][offload] Increase file upload limit from 2048MiB to 4096MiB for GCP/GCS offloading (#22554)
• [fix][storage] ReadonlyManagedLedger initialization does not fill in the properties (#22630)
• [fix][txn] Handle exceptions in the transaction pending ack init (#21274)
• [improve][meta] Log a warning when ZK batch fails with connectionloss (#22566)
• [improve][offload] Apply autoSkipNonRecoverableData configuration to tiered storage (#22531)
• [improve][offload] Replace usage of shaded class in OffsetsCache (#22683)

Client

• [fix][client] Include swagger annotations in shaded client lib (#22570)
• [fix][client] Fix Consumer should return configured batch receive max messages (#22619)
• [fix][client] Fix ReaderBuilder doest not give illegalArgument on connection failure retry (#22639)
• [fix][client] Fix client side memory leak when call MessageImpl.create and fix imprecise client-side metrics: pendingMessagesUpDownCounter, pendingBytesUpDownCounter, latencyHistogram (#22393)

Pulsar IO and Pulsar Functions

• [fix][fn] make sure the classloader for ContextImpl is functionClassLoader in different runtimes (#22501)
• [fix][io] CompressionEnabled didn't work on elasticsearch sink (#22565)
• [fix][io] Fix es index creation (#22654)
• [fix][io] Kafka Source connector maybe stuck (#22511)
• [improve][io]: Add ...

Broker

• [fix][broker] Fix force delete subscription not working (#22423)
• [fix][broker] Avoid being stuck when closing the broker with extensible load manager (#22573)
• [fix][broker] Create new ledger after the current ledger is closed (#22034)
• [fix][broker] Disable system topic message deduplication (#22582)
• [fix][broker] Fix BufferOverflowException and EOFException bugs in /metrics gzip compression (#22576)
• [fix][broker] Fix NPE causing dispatching to stop when using Key_Shared mode and allowOutOfOrderDelivery=true (#22533)
• [fix][broker] Fix ProducerBusy issue due to incorrect userCreatedProducerCount on non-persistent topic (#22685)
• [fix][broker] Fix a deadlock in SystemTopicBasedTopicPoliciesService during NamespaceEventsSystemTopicFactory init (#22528)
• [fix][broker] Fix broken topic policy implementation compatibility with old pulsar version (#22535)
• [fix][broker] Fix configurationMetadataSyncEventTopic is marked supporting dynamic setting, but not implemented (#22684)
• [fix][broker] Fix consumer stops receiving messages when with large backlogs processing (#22454)
• [fix][broker] Fix invalid condition in logging exceptions (#22412)
• [fix][broker] Fix message drop record in producer stat (#22458)
• [fix][broker] Fix metrics pulsar_topic_load_failed_count is 0 when load non-persistent topic fails and fix the flaky test testBrokerStatsTopicLoadFailed (#22580)
• [fix][broker] Fix nothing changed after removing dynamic configs (#22673)
• [fix][broker] Fix typos in Consumer class (#22532)
• [fix][broker] Fix typos lister -> listener (#21068)
• [fix][broker] One topic can be closed multiple times concurrently (#17524)
• [fix][broker] Optimize /metrics, fix unbounded request queue issue and fix race conditions in metricsBufferResponse mode (#22494)
• [fix][broker] Part-1: Replicator can not created successfully due to an orphan replicator in the previous topic owner (#21946)
• [fix][broker] Part-2: Replicator can not created successfully due to an orphan replicator in the previous topic owner (#21948)
• [fix][broker] Prevent long deduplication cursor backlog so that topic loading wouldn't timeout (#22479)
• [fix][broker] Reader stuck after call hasMessageAvailable when enable replicateSubscriptionState (#22572)
• [fix][broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsync for ExtensibleLoadBalancer (#22379) (#22403)
• [fix][broker] Support OIDC providers with JWK without alg field set in keys (#22421)
• [fix][broker] Update TransferShedder underloaded broker check to consider max loaded broker's msgThroughputEMA and update IsExtensibleLoadBalancerImpl check (#22321) (#22416)
• [fix][broker] Update topic partition failed when config maxNumPartitionsPerPartitionedTopic<0 (#22397)
• [fix][broker] avoid offload system topic (#22497)
• [fix][broker] fix replicated subscriptions for transactional messages (#22452)
• [fix][broker] rename to changeMaxReadPositionCount (#22656)
• [fix][broker] usedLocallySinceLastReport should always be reset (#22672)
• [fix][broker][admin] Fix cannot update properties on NonDurable subscription. (#22411)
• [fix][broker] Fix Reader can be stuck from transaction aborted messages. (#22610)
• [improve][broker] Make the config metricsBufferResponse description more effective (#22490)
• [improve][broker] Retry re-validating ResourceLock with backoff after errors (#22617)
• [improve][broker] Add topic_load_failed metric (#19236)
• [improve][broker] Add topic name to emitted error messages. (#22506)
• [improve][broker] Avoid repeated Read-and-discard when using Key_Shared mode (#22245)
• [improve][broker] Create partitioned topics automatically when enable topic level replication (#22537)
• [improve][broker] Don't log brokerClientAuthenticationParameters and bookkeeperClientAuthenticationParameters by default (#22395)
• [improve][broker] Improve Gzip compression, allow excluding specific paths or disabling it (#22370)
• [improve][broker] Optimize gzip compression for /metrics endpoint by sharing/caching compressed result (#22521)
• [improve][broker] Propagate cause exception in TopicBusyException when applicable (#22596)
• [improve][broker] Repeat the handleMetadataChanges callback when configurationMetadataStore equals localMetadataStore (#22519)
• [improve][broker] Servlet support response compression (#21667)
• [improve][broker] Support X-Forwarded-For and HA Proxy Protocol for resolving original client IP of http/https requests (#22524)
• [improve][broker] backlog quota exceed limit log replaced with debug (#22488)
• [fix][ml] Fix NPE of getValidPositionAfterSkippedEntries when recovering a terminated managed ledger (#22552)
• [fix][ml] Mark delete stuck due to switching cursor ledger fails (#22662)
• [fix][ml] No rollover inactive ledgers when metadata service invalid (#22284)
• [fix][offload] Fix OOM in tiered storage, caused by unbounded offsets cache (#22679)
• [fix][offload] Increase file upload limit from 2048MiB to 4096MiB for GCP/GCS offloading (#22554)
• [fix][storage] ReadonlyManagedLedger initialization does not fill in the properties (#22630)
• [fix][admin] Fix namespace admin api exception response (#22587)
• [fix][txn]Handle exceptions in the transaction pending ack init (#21274)
• [improve][log] Print source client addr when enabled haProxyProtocolEnabled (#22686)
• [improve][meta] Log a warning when ZK batch fails with connectionloss (#22566)
• [improve][offload] Apply autoSkipNonRecoverableData configuration to tiered storage (#22531)
• [improve][offload] Replace usage of shaded class in OffsetsCache (#22683)
• [improve][sec] Align some namespace level policy authorisation check (#21640)

Client

• [fix][client] Fix Consumer should return configured batch receive max messages (#22619)
• [fix][client] Fix ReaderBuilder doest not give illegalArgument on connection failure retry (#22639)
• [fix][client] Fix client side memory leak when call MessageImpl.create and fix imprecise client-side metrics: pendingMessagesUpDownCounter, pendingBytesUpDownCounter, latencyHistogram (#22393)

Pulsar IO and Pulsar Functions

• [fix][fn] make sure the classloader for ContextImpl is functionClassLoader in different runtimes (#22501)
• [fix][io] CompressionEnabled didn't work on elasticsearch sink (#22565)
• [fix][io] Fix es index creation (#22654) (#22701)
• [fix][io] Kafka Source connector maybe stuck (#22511)
• [improve][io]: Add validation for JDBC sink not supporting primitive schema (#22376)

...

What's Changed

Broker

• [fix][broker] Avoid execute prepareInitPoliciesCacheAsync if namespace is deleted (#22268)
• [fix][broker] Avoid expired unclosed ledgers when checking expired messages by ledger closure time (#22335)
• [fix][broker] Check cursor state before adding it to the waitingCursors (#22191)
• [fix][broker] Close dispatchers stuck due to mismatch between dispatcher.consumerList and dispatcher.consumerSet (#22270)
• [fix][broker] Fix OpReadEntry.skipCondition NPE issue (#22367)
• [fix][broker] Fix ResourceGroup report local usage (#22340)
• [fix][broker] Fix ResourceGroups loading (#21781)
• [fix][broker] Fix issue of field 'topic' is not set when handle GetSchema request (#22377)
• [fix][broker] Fix wrong double-checked locking for readOnActiveConsumerTask in dispatcher (#22279)
• [fix][broker] fix mismatch between dispatcher.consumerList and dispatcher.consumerSet (#22283)
• [fix][ml]Expose ledger timestamp (#22338)
• [fix][offload] Fix Offload readHandle cannot close multi times. (#22162)
• [fix][txn]Fix TopicTransactionBuffer potential thread safety issue (#22149)
• [improve][admin] Fix the createMissingPartitions doesn't response correctly (#22311)
• [improve][broker] Add createTopicIfDoesNotExist option to RawReader constructor (#22264)
• [improve][broker] Add fine-grain authorization to ns/topic management endpoints (#22305)
• [improve][broker] Add missing configuration keys for caching catch-up reads (#22295)
• [improve][broker] Support create RawReader based on configuration (#22280)

Client

• [fix][client] Consumer lost message ack due to race condition in acknowledge with batch message (#22353)
• [fix][client] Do no retrying for error subscription not found when disabled allowAutoSubscriptionCreation (#22164)
• [fix][client] Fix wrong results of hasMessageAvailable and readNext after seeking by timestamp (#22363)
• [fix][client] GenericProtobufNativeSchema not implement getNativeSchema method. (#22204)
• [fix][client] Unclear error message when creating a consumer with two same topics (#22255)
• [fix][client] fix Reader.hasMessageAvailable might return true after seeking to latest (#22201)
• [fix][client]Fixed getting an incorrect maxMessageSize value when accessing multiple clusters in the same process (#22306)

Others

• [fix][misc] Make ConcurrentBitSet thread safe (#22361)
• [improve][misc] Remove the call to sun InetAddressCachePolicy (#22329)
• [fix][ws] Check the validity of config before start websocket service (#22231)

Library updates

• [fix][sec] Upgrade jose4j to 0.9.4 (#22273)
• [fix][sec] Go Functions security updates (#21844)
• [fix][sec] Upgrade Zookeeper to 3.9.2 to address CVE-2024-23944 (#22275)
• [improve][misc] Include native epoll library for Netty for arm64 (#22319)
• [improve][misc] Upgrade Netty version to 4.1.105.Final (#21923)

Tests & CI

• [fix][ci] Enable CI for branch-3.2 (#22287)
• [fix][ci] Tolerate mount option change failing in CI (#22241)
• [fix][test] Fix flaky ManagedLedgerErrorsTest.recoverAfterZnodeVersionError (#22368)
• [fix][test] Fix flaky RGUsageMTAggrWaitForAllMsgsTest (#22252)
• [fix][test] Fix flaky test BrokerServiceAutoSubscriptionCreationTest (#22190)
• [fix][test] Fix flaky test ManagedLedgerTest.testGetNumberOfEntriesInStorage (#22344)
• [fix][test] Fix thread leak in TopicPoliciesAuthZTest (#22257)
• [fix][test][branch-3.2] Fix broken ManagedLedgerTest.testGetNumberOfEntriesInStorage
• [fix][fn] fix broken function-go test (#22260)
• [fix][test] Fix the tests with same namespace name (#22240)
• [fix][build] Fix version in pulsar-bom/pom.xml and fix src/set-project-version.sh
• [improve][test][branch-3.2] Improve ManagedLedgerTest.testGetNumberOfEntriesInStorage
• [improve][misc] Upgrade checkstyle to 10.14.2 (#22291)

For the complete list, check the full changelog.

What's Changed

Broker

• [fix][broker] Avoid execute prepareInitPoliciesCacheAsync if namespace is deleted (#22268)
• [fix][broker] Avoid expired unclosed ledgers when checking expired messages by ledger closure time (#22335)
• [fix][broker] Check cursor state before adding it to the waitingCursors (#22191)
• [fix][broker] Close dispatchers stuck due to mismatch between dispatcher.consumerList and dispatcher.consumerSet (#22270)
• [fix][broker] Fix OpReadEntry.skipCondition NPE issue (#22367)
• [fix][broker] Fix ResourceGroup report local usage (#22340)
• [fix][broker] Fix ResourceGroups loading (#21781)
• [fix][broker] Fix issue of field 'topic' is not set when handle GetSchema request (#22377)
• [fix][broker] Fix wrong double-checked locking for readOnActiveConsumerTask in dispatcher (#22279)
• [fix][broker] fix mismatch between dispatcher.consumerList and dispatcher.consumerSet (#22283)
• [fix][ml]Expose ledger timestamp (#22338)
• [improve][admin] Fix the createMissingPartitions doesn't response correctly (#22311)
• [improve][broker] Add createTopicIfDoesNotExist option to RawReader constructor (#22264)
• [improve][broker] Add fine-grain authorization to ns/topic management endpoints (#22309)
• [improve][broker] Add missing configuration keys for caching catch-up reads (#22295)
• [improve][broker] Change log level to reduce duplicated logs (#22147)

Client

• [fix][client] Consumer lost message ack due to race condition in acknowledge with batch message (#22353)
• [fix][client] Fix wrong results of hasMessageAvailable and readNext after seeking by timestamp (#22363)
• [fix][client] GenericProtobufNativeSchema not implement getNativeSchema method. (#22204)
• [fix][client] Unclear error message when creating a consumer with two same topics (#22255)
• [fix][client] fix Reader.hasMessageAvailable might return true after seeking to latest (#22201)
• [fix][client]Fixed getting an incorrect maxMessageSize value when accessing multiple clusters in the same process (#22306)
• [improve][client] Add backoff for seek (#20963)

Pulsar IO and Pulsar Functions

• [fix][fn] enable Go function token auth and TLS (#20468)

Others

• [improve][misc] Remove the call to sun InetAddressCachePolicy (#22329)
• [fix][misc] Make ConcurrentBitSet thread safe (#22361)
• [fix][ws] Check the validity of config before start websocket service (#22231)

Library updates

• [fix][sec] Upgrade jose4j to 0.9.4 (#22273)
• [fix][sec] Bump google.golang.org/grpc from 1.38.0 to 1.56.3 in /pulsar-function-go (#21444)
• [fix][sec] Go Functions security updates (#21844)
• [fix][sec] Upgrade Zookeeper to 3.9.2 to address CVE-2024-23944 (#22275)
• [fix][sec] Upgrade prometheus client_golang to v1.12.2 to fix CVE-2022-21698 (#20579)
• [fix][build] Upgrade alluxio version to 2.9.3 to fix CVE-2023-38889 (#21715)
• [improve][misc] Include native epoll library for Netty for arm64 (#22319)
• [improve][misc] Pin Netty version in pulsar-io/alluxio (#21728)
• [improve][misc] Upgrade Netty version to 4.1.105.Final (#21923)

Tests & CI

• [fix][ci] Tolerate mount option change failing in CI (#22241)
• [fix][ci][branch-3.0] Increase Maven's heap size from 1024m to 1500m as it is in master
• [fix][test] Fix flaky ManagedLedgerErrorsTest.recoverAfterZnodeVersionError (#22368)
• [fix][test] Fix flaky RGUsageMTAggrWaitForAllMsgsTest (#22252)
• [fix][test] Fix flaky test BrokerServiceAutoSubscriptionCreationTest (#22190)
• [fix][test] Fix flaky test ManagedLedgerTest.testGetNumberOfEntriesInStorage (#22344)
• [fix][test][branch-3.0] Fix broken ManagedLedgerTest.testGetNumberOfEntriesInStorage
• [fix][test] Fix the tests with same namespace name (#22240)
• [improve][test][branch-3.0] Improve ManagedLedgerTest.testGetNumberOfEntriesInStorage
• [improve][misc] Upgrade checkstyle to 10.14.2 (#22291)

For the complete list, check the full changelog.

What's Changed

Broker

• [improve][broker] Consistently add fine-grain authorization to REST API 22202

• [improve][broker] Add fine-grain authorization to retention admin API 22163

• [improve][broker] Cache the internal writer when sent to system topic. 22099

• [improve][broker] Add an error log to troubleshoot the failure of starting broker registry. 22065

• [improve][broker] Do not retain the data in the system topic 22022

• [improve] [broker] Do not try to open ML when the topic meta does not exist and do not expect to create a new one. #21995 22004

• [improve] [broker] Do not print an Error log when responding to HTTP-404 when calling Admin API and the topic does not exist. 21995

• [improve][broker] Do not close the socket if lookup failed due to LockBusyException 21993

• [improve][broker] Don't rollover empty ledgers based on inactivity 21893

• [improve][broker] Improve NamespaceUnloadStrategy error message 21880

• [improve][broker] Skip loading the NAR packages if not configured 21867

• [improve][broker] defer the ownership checks if the owner is inactive (ExtensibleLoadManager) 21857

• [improve][broker] defer the ownership checks if the owner is inactive (ExtensibleLoadManager) 21811

• [improve][broker] cleanup the empty subscriptionAuthenticationMap in zk when revoke subscription permission 21696

• [improve] [broker] Let the producer request success at the first time if the previous one is inactive 21220

• [improve][broker] Avoid record inactiveproducers when deduplication is disable. 21193

• [improve][broker] Support not retaining null-key message during topic compaction (#21578) 21662

• [improve][broker] Print recoverBucketSnapshot log if cursorProperties are empty 21651

• [improve][broker][PIP-318] Support not retaining null-key message during topic compaction 21578

• [fix] [broker] print non log when delete partitioned topic failed 22153

• [fix][broker] Fix broker not starting when both transactions and the Extensible Load Manager are enabled 22139

• [fix][broker] Return getOwnerAsync without waiting on source broker upon Assigning and Releasing and handle role change during role init 22112

• [fix] [broker] Enabling batch causes negative unackedMessages due to ack and delivery concurrency 22090

• [fix][broker] Support running docker container with gid != 0 22081

• [fix][broker] Set ServiceUnitStateChannel topic compaction threshold explicitly, improve getOwnerAsync, and fix other bugs 22064

• [fix][broker] Fix hash collision when using a consumer name that ends with a number 22053

• [fix][broker] Sanitize values before logging in apply-config-from-env.py script 22044

• [fix] [broker] Fix can not subscribe partitioned topic with a suffix-matched regexp 22025

• [fix] [broker] Subscription stuck due to called Admin API analyzeSubscriptionBacklog 22019

• [fix][broker][branch-3.1] Avoid PublishRateLimiter use an already closed RateLimiter 22011

• [fix] [broker] [branch-3.0] Fast fix infinite HTTP call createSubscriptions caused by wrong topicName 21997

• [fix][broker]Support setting autoSkipNonRecoverableData dynamically in expiryMon… 21991

• [fix] [broker] add timeout for health check read. 21990

• [fix][broker] Fix schema deletion error when deleting a partitioned topic with many partitions and schema 21977

• [fix] [broker] Fix reader stuck when read from compacted topic with read compact mode disable 21969

• [fix] [broker] Replication stopped due to unload topic failed 21947

• [fix] [broker] Expire messages according to ledger close time to avoid client clock skew 21940

• [fix][broker] Restore the broker id to match the format used in existing Pulsar releases 21937

• [fix][broker] Fix getMessageById throws 500 21919

• [fix] [broker] fix write all compacted out entry into compacted topic 21917

• [fix][broker] Fix deadlock while skip non-recoverable ledgers. 21915

• [fix][broker] Fix leader broker cannot be determined when the advertised address and advertised listeners are configured 21894

• [fix] [broker] Fix break change: could not subscribe partitioned topic with a suffix-matched regexp due to a mistake of PIP-145 21885

• [fix][broker] Fix compaction/replication data loss when expire messages 21865

• [fix][broker]Fix NonPersistentDispatcherMultipleConsumers ArrayIndexOutOfBoundsException 21856

• [fix][broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set 21842

• [fix][broker] Fix String wrong format 21829

• [fix] [broker] Update topic policies as much as possible when some ex was thrown 21810

• [fix][broker] Fix TableViewLoadDataStoreImpl NPE 21777

• [fix][broker] Fix the wrong value of BrokerSrevice.maxUnackedMsgsPerDispatcher 21765

• [fix][broker] Fixed the ExtensibleLoadManagerImpl internal system getTopic failure when the leadership changes 21764

• [fix][broker] Delete compacted ledger when topic is deleted 21745

• [fix][broker] Skip topic auto-creation for ExtensibleLoadManager internal topics 21729

• [fix][broker] Fixed ServiceUnitStateChannel monitor to tombstone only inactive bundle states 21721

• [fix][broker] Avoid compaction task stuck when the last message to compact is a marker 21718

• [fix][broker] Fix the issue of topics possibly being deleted. 21704

• [fix][broker] Fix typo in the config key 21690

• [fix] [broker] network package lost if enable haProxyProtocolEnabled 21684

• [fix][broker] Record GeoPersistentReplicator.msgOut before producer#sendAsync 21673

• [fix][broker] Fix memory leak during topic compaction 21647

• [fix][broker] Fix lookupRequestSemaphore leak when topic not found 21646

• [fix][broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set 21633

• [fix][broker] Fixed getting incorrect KeyValue schema version 21632
  (#21620)

• [fix] [broker] Delete topic timeout due to NPE 21595

• [fix][broker] Fix incorrect unack count when using shared subscription on non-persistent topic 21592

• [fix][broker] Duplicate LedgerOffloader creation when namespace/topic… 21591

• [fix][broker] Correct schema deletion for partitioned topic 21574

• [fix][broker] Fix setReplicatedSubscriptionStatus incorrect behavior 21510

• [fix][broker] Do not write replicated snapshot marker when the topic which is not enable replication 21495

• [fix][broker] Fix issue with consumer read uncommitted messages from compacted topic 21465

• [fix][broker] Fix resource_quota_zpath 21461

• [fix][broker] Fix MultiRoles token provider NPE when using anonymous clients 21429

• [fix][broker]...

Broker

• Consistently add fine-grain authorization to REST API #22202
• Add fine-grain authorization to retention admin API #22163
• Print non log when delete partitioned topic failed #22153
• Fix broker not starting when both transactions and the Extensible Load Manager are enabled #22139
• Return getOwnerAsync without waiting on source broker upon Assigning and Releasing and handle role change during role init #22112
• Cache the internal writer when sent to system topic. #22099
• Enabling batch causes negative unackedMessages due to ack and delivery concurrency #22090
• Support running docker container with gid != 0 #22081
• Add an error log to troubleshoot the failure of starting broker registry #22065
• Set ServiceUnitStateChannel topic compaction threshold explicitly, improve getOwnerAsync, and fix other bugs #22064
• Fix hash collision when using a consumer name that ends with a number #22053
• Sanitize values before logging in apply-config-from-env.py script #22044
• Fix can not subscribe partitioned topic with a suffix-matched regexp #22025
• Do not retain the data in the system topic #22022
• Subscription stuck due to called Admin API analyzeSubscriptionBacklog #22019
• Do not try to open ML when the topic meta does not exist and do not expect to create a new one #22004
• Do not print an Error log when responding to HTTP-404 when calling Admin API and the topic does not exist #21995
• Do not close the socket if lookup failed due to LockBusyException #21993
• Support setting autoSkipNonRecoverableData dynamically in PersistentMessageExpiryMonitor #21991
• Add timeout for health check read #21990
• Fix schema deletion error when deleting a partitioned topic with many partitions and schema #21977
• Defer the ownership checks if the owner is inactive (ExtensibleLoadManager) #21811
• Fix compaction/replication data loss when expire messages #21865
• Skip loading the NAR packages if not configured #21867
• Improve NamespaceUnloadStrategy error message #21880
• Don't rollover empty ledgers based on inactivity #21893
• Expire messages according to ledger close time to avoid client clock skew #21940
• Fix reader stuck when read from compacted topic with read compact mode disable #21969

Client

• Fix multi-topics consumer could receive old messages after seek #21945
• Fix ConsumerBuilderImpl#subscribe silent stuck when using pulsar-client:3.0.x with jackson-annotations prior to 2.12.0 #21985

Pulsar IO and Pulsar Functions

• [FN] Add configuration for connector & functions package url sources #22184
• [FN] Add missing "exception" argument to some log.error #22140
• [FN] Prevent putstate uses empty values #22127
• [FN] Optimize Function Worker startup by lazy loading and direct zip/bytecode access #22122
• [FN] Prevent create state table from API calls for non-exists instances #22107
• [FN] Use unified PackageManagement service to download packages #21955

CLI

• [Admin] Expose the offload threshold in seconds to the admin #22101
• [CLI] Fix the bug when set-retention specified size with -T #22150
• [CLI] Error out when --destinationBroker is set when --bundle is empty #21879

Others

• [MISC] Add a check for the input time value #22023
• [Proxy] Add a check for brokerServiceURL that does not support multi uri yet #21972
• [TX] Get previous position by managed ledger #22024
• [ML] Fix retry mechanism of deleting ledgers to invalidate #21869
• [ML] Filter out deleted entries before read entries from ledger #21739

Library updates

• Upgrade Jetty to 9.4.54.v20240208 to address CVE-2024-22201 #22144
• pgrade commons-compress to 1.26.0 #22086
• Upgrade BookKeeper dependency to 4.16.4 #21983

A special thanks to the following contributors who contributed to Pulsar 3.2.1:
mattisonchao, lhotari, poorbarcode, Technoboy-, jiangpengcheng, dragosvictor, freeznet, liangyepianzhou, heesung-sn, zymap, massakam, thetumbled, BewareMyPower, chenhongSZ, Shawyeok, hangc0276, 315157973, coderzc, dao-jun.