Building Pulsar (incl. docker images) from source with Java 9+ (Java 11, Java 15, etc.) causes silent NoSuchMethodErrors at runtime

[lhotari]

Building Pulsar (incl. docker images) from source with Java 11 causes silent NoSuchMethodErrors at runtime

To Reproduce
Steps to reproduce the behavior:

Use Java 11.
Then, compile v2.6.2-candidate-1 docker images locally:

git clone --depth 1 -b v2.6.2-candidate-1 https://github.com/apache/pulsar pulsar-2.6
cd pulsar-2.6
mvn -DskipTests clean install
./docker/build.sh

Start pulsar in docker

docker run -p 6650:6650 pulsar:2.6.2 bin/pulsar standalone --no-functions-worker -nss

Compile & run this Pulsar code with required libraries:

import org.apache.pulsar.client.api.*;

public class PulsarJava11Problem {
    public static void main(String[] args) throws PulsarClientException {
        PulsarClient pulsarClient = PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build();
        Producer<String> producer = pulsarClient.newProducer(Schema.STRING).topic("mytopic").create();
    }
}

The producer never gets created and it times out.

What happens on a server is a java.lang.NoSuchMethodError: java.nio.ByteBuffer.rewind()Ljava/nio/ByteBuffer;

This happens at

pulsar/pulsar-common/src/main/java/org/apache/pulsar/common/schema/LongSchemaVersion.java

Line 44 in 097108a

buffer.rewind();

.

java.lang.NoSuchMethodError: java.nio.ByteBuffer.rewind()Ljava/nio/ByteBuffer;
	at org.apache.pulsar.common.schema.LongSchemaVersion.bytes(LongSchemaVersion.java:44)
	at org.apache.pulsar.common.protocol.Commands.newProducerSuccessCommand(Commands.java:377)
	at org.apache.pulsar.broker.service.PulsarCommandSenderImpl.sendProducerSuccessResponse(PulsarCommandSenderImpl.java:95)
	at org.apache.pulsar.broker.service.ServerCnx.lambda$handleProducer$19(ServerCnx.java:1137)
	at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670)
	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
	at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670)
	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
	at org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.lambda$createSchemaLocator$28(BookkeeperSchemaStorage.java:504)
	at org.apache.bookkeeper.util.ZkUtils$1.processResult(ZkUtils.java:80)
	at org.apache.bookkeeper.zookeeper.ZooKeeperClient$10$1.processResult(ZooKeeperClient.java:737)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:636)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)

This exception never gets printed to the log, this stacktrace was captured using the debugger.

Expected behavior

Compiling with an unsupported Java version should result in an error and the NoSuchMethodError that happens in Pulsar shouldn't get "swallowed".

It should be possible to compile the code with Java 9+ without running into issues when running with Java 8.

There's an explanation of the issue with ByteBuffer at jetty/jetty.project#3244 . A solution is also described:

The solution is to cast the ByteBuffer to Buffer when calling those methods:

((Buffer)byteBuffer).position(0);

[lhotari]

@eolivelli I noticed that you had done some changes to allow compiling with JDK 14+. Have you come across the ByteBuffer incompatibilities?

[wolfstudy]

This problem should not only exist in 2.6.2, but it should also be a master problem, @zymap helped to verify the logic with JDK 1.8, it seems that everything is OK. It looks like we need to support different JDK versions to build Apache Pulsar.

[eolivelli]

@lhotari
my efforts about building with JDK14+ are about using JDK14 for development,
if you build it with JDK>8 you cannot run with JDK8.

We should add maven.compiler.release=8 in our pom.xml, it must be done in a profile activated when you are on jdk9+, it is a quite easy fix.

that's a very common problem:
java.lang.NoSuchMethodError: java.nio.ByteBuffer.rewind()Ljava/nio/ByteBuffer;

[lhotari]

We should add maven.compiler.release=8 in our pom.xml, it must be done in a profile activated when you are on jdk9+, it is a quite easy fix.

@eolivelli I think it would be valuable. It's easy to miss that you must compile Pulsar with Java 8 since the compilation passes without errors and even at runtime the errors are swallowed and not logged (at least in this particular case). Very hard to debug.

[eolivelli]

I guess that if you build with JDK14+ and you run everything on JDK14+ you won't hit problems.
probably you are not going to run Pulsar on JDK8, as newer JVMs are generally (I don't want to start a discussion now) more efficient and provide better performances.

In BK community we are switching to JDK11+ (but still next release will work on JDK8)

[lhotari]

There is a blog post with an easy solution: https://www.morling.dev/blog/bytebuffer-and-the-dreaded-nosuchmethoderror/ .
This property should be defined in pom.xml:

<properties>
<maven.compiler.release>8</maven.compiler.release> </properties>

[eolivelli]

You have to add it in a profile, because if you add only this property the build will break on jdk8 because the 'release' option is not valid with jdk8