PIP-181: Pulsar Shell

[nicoloboschi]

Screen.Recording.2022-06-28.at.9.46.45.AM.online-video-cutter.com.mp4

Screen.Recording.2022-06-28.at.2.25.52.PM.online-video-cutter.com.mp4

Motivation

Currently Pulsar comes with a couple of utility scripts with the goal of managing an existing cluster, test behaviours and verify performances: these tools are available as SH script inside the bin directory.
The pulsar-admin is the CLI tool supposed to help users and operators to configure the system, operate over policies, install functions and much else.

This proposal basically aims to solve two different problems:

1. pulsar-admin is terribly slow. Every time the script is triggered, a new JVM process is spawned. The JVM process creation is heavy and most of the time is spent by the JVM initialization process. A very common use case for cluster operators is to create scripts with several commands with the goal of initialize the cluster, initialize a specific tenant (namespaces, topics, policies, functions..); in this case, one JVM is initialized for each scripts leads to waste of time and resources.

2. User experience. The current design of the Pulsar CLIs can be improved. There are a couple of aspects that may be annoying for a user and can discourage a user to use Pulsar.
       1. Poking around available commands and options in a CLI tool (pulsar-admin for instance, but it's the same for pulsar-perf and pulsar-client) is slow and hard. In order to discover commands and options you need to use -h option and, since the performance issue pointed at 1., it can be annoying and time-consuming. Autocomplete feature could be a real game-changer in this context.
       2. Different CLI tools. There are a couple of different shell scripts. They have different goals and it's okay to keep them separated. However, they raise a barrier for a non Pulsar expert that doesn't have a convenient entry-point.

Goal

Address all the issues in the previous section with a single solution.

API Changes

A new shell script bin/pulsar-shell will be introduced. bin/pulsar shell could be a valid alternative but it's not findable for a newbie user since no direct file exists.

Implementation

Concepts

The new script pulsar-shell will differ from the existing for the following reasons:

1. It's a shell. When you start it, it will wait for commands to be executed. After the command has been executed, despite its result, the shell session will not be destroyed and it will wait for another command.
2. Unifies all the CLI scripts. In pulsar-shell you'll be able to run all the existing CLI commands. This will be done in a way that when a new command/option is added, the pulsar-shell will be updated accordingly.
3. It comes with sophisticated autocompletion and command history to highly improve the UX.
4. Performance. Since JVM is initiated once, it will gain on performance thanks to the JVM warmup and internal libraries bootstraps.
5. It will accept a file or a list of commands (parameter and stdin) to start a shell, run the commands and close the shell. We'll call it the non-interactive mode and it will ease the cluster operations automations.

Note that existing tools will not be removed/changed.

Implementation

The shell implementation will be developed in Java, using a well-known library called JLine 3 for the shell support.
There will be a new main class that will extends the existing class tools (e.g. PulsarAdminTool)

Configuration

The configuration file taken by default will be client.conf, like current CLI tools. The env setup will be exactly the same as for pulsar-admin and pulsar-client.

Autocompletion

JLine3 has great support for autocompletion. The shell java class will translate current JCommander tools to the JLine3 completion API. This will ensure that all APIs will be up-to-date and covered.

History

JLine3 has built-in support for history, both in-memory and persisted (on local FS). By default the history will be persisted in the user home directory. This feature can be turned off for security concerns with a configuration property.

List of tools

The proposal is to get the following tools:

• bin/pulsar-admin
• bin/pulsar-client
• bin/pulsar-perf

Distribution

At the moment the only way to use the CLI (outside of the docker image) is to download the entire tarball and extract it and then run the CLI command.

With this proposal there will be a new artifact (along with pulsar-io, pulsar-offloaders) called pulsar-shell.
This artifact will be a tarball containing only the needed lib for running pulsar-shell.

Rejected Alternatives

• Create a shell mode for pulsar-admin. This won't be a flexible solution because then we may need the same for other tools. Also it doesn't cover the CLI unification part.

• Create a brand-new shell mode. Not compatible with current CLI and another tool to maintain.

• Use another shell library. There are a couple of valid alternatives to JLine3 but it's wide-spread and the autocompletion API is much more flexible than others since it's not done with annotations (the shell needs to translate current JCommander definitions from existing tools)

• Basic script

• Add non-interactive mode

• Add pulsar-admin

• Add pulsar-client

• Multi cluster management

• Distribute the tool

Future works

• Create a native image for running the shell using GraalVM. This will make the shell even faster and it will remove the JDK requirement.
• Distribute the shell with the most used tools (homebrew, sdkman...)

[MonicaMagoniCom]

In our enviroment based on Kubernetes we deploy pulsar with Helm; authentication and authorization are enabled and this means that when the broker is created it will run some pulsar admin commands in order to:

• create the tenant
• create the token
• create the namespace
• grant permission
  In version 2.9.3, it seems that these commands are really slow (the broker takes up to 6 minutes to be ready); until 2.7.2 they were quite fast (in the order of 2 minutes max). any suggestion?

[codelipenghui]

Close since all the PRs get merged.