PIP-201 : Extensions mechanism for Pulsar Admin CLI tools

[eolivelli]

Motivation

There are many projects that are in the Pulsar ecosystem like Protocol Handlers (Kafka, MQTT, RabbitMQ) and libraries (JMS…) that need additional tools for operating Pulsar following specific conventions adopted by each project and to handle custom domain objects (like JMS queues, Kafka Consumer Groups...).

Some examples:

• Kafka: tools for inspecting internal systems, handle Tenant namespaces, SchemaRegistry, Transaction Manager, Consumers Groups
• JMS: tools to handling Subscriptions and Selectors
• RabbitMQ: tools to handle Pulsar topics and subscription following the convention

This is very important as it is hard to follow the conventions of each project using pulsar-admin and the administrator may inadvertently break the system.

This feature will enhance the UX of the Pulsar Admin CLI tools for the benefit of the whole ecosystem and users.

Goal

As we do for many other components in Pulsar, we need a way to enhance the CLI tools, pulsar-admin and pulsar-shell, with additional commands that are specific to the additional features.

The proposal here is to add an extension mechanism to the pulsar-admin (and so pulsar-shell) tool.
Following the usual conventions for extensions the extension will be bundled in a .nar file that may contain additional third party libraries.

The extension will be able to provide new top level commands

pulsar-admin my-command-group my-command arguments…

The extension will be able to access the PulsarAdmin API provided by the environment.

The extension must not depend directly on the JCommander library but we will provide an API to declare the parameters and the other metadata necessary to document and execute the command.
This is very important because during the lifecycle of Pulsar the project may decide to upgrade JCommander to an incompatible version or to drop the dependency at all.

API Changes

We will introduce a new Maven module pulsar-tools-api that contains the public API that can be used by implementations of the custom commands.

The implementation will be bundled in a .nar file with a descriptor with the following fields:

factoryClass: xxxxx.CommandFactory
name: extension-name
description: Description...

There are the new classes:

/**
   Access to the environment
*/
public interface CommandExecutionContext {
    PulsarAdmin getPulsarAdmin();
    Properties getConfiguration();
}


/**
 * Custom command implementation
 */
public interface CustomCommand {
    String name();
    String description();
    List<ParameterDescriptor> parameters();
    boolean execute(Map<String, Object> parameters, CommandExecutionContext context) throws Exception;
}

/**
 * A group of commands.
 */
public interface CustomCommandGroup {
    String name();
    String description();
    List<CustomCommand> commands(CommandExecutionContext context);
}

/**
 * Main entry point of the extension
*/
public interface CustomCommandFactory {

    /**
     * Generate the available command groups.
     */
    List<CustomCommandGroup> commandGroups(CommandExecutionContext context);
}

@Builder
@Getter
public final class ParameterDescriptor {
    @Builder.Default
    private List<String> names = new ArrayList<>();
    private boolean mainParameter; // set to true if you want this command to contain the main argument of the command
    @Builder.Default
    private String description = "";
    private ParameterType type = ParameterType.STRING;
    private  boolean required = false;
}

Sample code for a custom command:

public class CommandFactory implements CustomCommandFactory {
  @Override
  public List<CustomCommandGroup> commandGroups(CommandExecutionContext context) {
    return Arrays.asList(
        new CustomCommandGroup() {
          @Override
          public String name() {
            return "customgroup";
          }

          @Override
          public String description() {
            return "My group";
          }

          @Override
          public List<CustomCommand> commands(CommandExecutionContext context) {
            return Arrays.asList(
                new CustomCommand() {
                  @Override
                  public String name() {
                    return "customcommand";
                  }

                  @Override
                  public String description() {
                    return "Run something";
                  }

                  @Override
                  public List<ParameterDescriptor> parameters() {
                    return Arrays.asList(
                        ParameterDescriptor.builder()
                            .description("Op type")
                            .type(ParameterType.STRING)
                            .names(Arrays.asList("--type", "-t"))
                            .required(true)
                            .build(),
                        ParameterDescriptor.builder()
                            .description("Topic")
                            .type(ParameterType.STRING)
                            .mainParameter(true)
                            .names(Arrays.asList("topic"))
                            .required(true)
                            .build());
                  }

                  @Override
                  public boolean execute(
                      Map<String, Object> parameters, CommandExecutionContext context)
                      throws Exception {
                    System.out.println(
                        "Execute: " + parameters + " properties " + context.getConfiguration());
                    String topic = parameters.getOrDefault("topic", "").toString();
                    TopicStats stats = context.getPulsarAdmin().topics().getStats(topic);
                    System.out.println("Topic stats: " + stats);
                    return false;
                  }
                });
          }
        });
  }
}

Implementation

In the Pulsar Admin CLI codebase we will add the code to use NarClassLoader and to bootstrap the Factories and implement the API.
Unfortunately JCommander relies on Java Class and Field annotations. We are going to use JavaAssist (that is already used by Pulsar) to generate dynamically some small bridge classes to make JCommander happy.
If in the future JCommander will change or we will drop JCommander we can change the bridge code easily.

Alternatives

There is no viable alternative. Third party extensions have to build their own tools like the JMS CLI (https://github.com/datastax/pulsar-jms/tree/master/pulsar-jms-cli).

[eolivelli]

merged

[asafm]

The design document is written very clearly, thanks for that 💯