Securing your Pulsar Cluster with Vault_Chris Kellogg
•
1 like•1,060 views
Learn how to secure a Pulsar cluster with Hashicorp Vault and deploy it on Kubernetes. Vault provides a secure way to generate tokens and store sensitive data and Pulsar has a pluggable architecture for authentication, authorization and secret management. This talk will walk through how to create custom plugins for Vault, integrate them with Pulsar and then deploy a Pulsar cluster on Kubernetes.
Report
Share
![Function Secret Configuration
tenant: "public"
namespace: "default"
name: “secrets-printer"
className: “secrets_printer”
inputs: ["public/default/secrets-trigger"]
autoAck: true
parallelism: 1
resources:
cpu: 0.5
ram: 536870912
disk: 536870912
secrets:
username:
path: "internal/data/database/config"
key: username
password:
path: "internal/data/database/config"
key: password
customRuntimeOptions: >-
{
"serviceAccountName": "pf-secrets-printer"
}
Used by the VaultKubernetesCustomizer to
add annotations for vault token and secret
injection](https://cdn.statically.io/img/image.slidesharecdn.com/ck-pulsar-summit-final-200624064318/85/Securing-your-Pulsar-Cluster-with-Vault_Chris-Kellogg-33-320.jpg)
Securing your Pulsar Cluster with Vault_Chris Kellogg
- 1. @cckellogg #PulsarSummit Securing Your Pulsar Cluster with Vault &
- 2. Chris Kellogg Software Engineer at Splunk Contributor to Apache Pulsar and Apache Heron committer cckellogg You can find me on: cckellogg
- 3. Agenda • Vault Overview • Why Pulsar and Vault • Pulsar Authentication/Authorization Model • Creating Custom Plugins • Packaging Custom Plugins • Kubernetes Integration • Demo
- 4. https://www.vaultproject.io “Vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, or certificates. Vault provides a unified interface to any secret, while providing tight access control and recording a detailed audit log.” What is Vault?
- 5. Vault Features Secret Management Authentication and Identity Data Encryption
- 6. Why Vault • Single source to manage secrets and tokens • Dynamic and Revokable tokens and secrets • Audit tracking for secrets and token • Merges identities across providers - LDAP, Okta, Kubernetes, AWS, GCP • Cloud friendly
- 7. Why Pulsar and Vault • No more forever tokens • Revokable tokens • Secure secret management for functions and connectors • Supports authenticating against many trusted sources of identity - LDAP, Okta, Kubernetes, AWS, GCP, GitHub • Central location for all security
- 9. Default is No Security • Produce and consume from any topic • Modify any tenant, namespace, topic or function • Function/Connector secrets stored as plain text in configs • No auditing of actions
- 10. Pulsar Security Features • TLS Encryption for traffic • Authentication - validate identity • Authorization - can user perform an action • Data encryption between producers and consumers
- 11. Pulsar Authentication • Responsible for determining identity of clients • Plugin System • Built-in Plugins - TLS - JWT - Authenz - Kerberos
- 12. Pulsar Authorization • Determines if a client has permission to perform an action • Plugin System • Built-in Plugin - Role based system backed by Zookeeper - SuperUsers - Tenant Admins - Actions: produce/consume/functions
- 14. Building Plugins Best Practices • Minimize third party dependencies • Use your own executor and threads for remote requests • Cache responses
- 15. public class VaultAuthenticationProvider implements AuthenticationProvider { void initialize(ServiceConfiguration config) throws IOException {}; String getAuthMethodName() { return "token" }; boolean authenticateHttpRequest(HttpServletRequest req, HttpServletResponse resp) throws Exception { throw new AuthenticationException("Not supported"); } String authenticate(AuthenticationDataSource authData) throws AuthenticationException { // Implement code to authenticate the client with vault } AuthenticationState newAuthState(AuthData authData, SocketAddress remoteAddress, SSLSession sslSession) throws AuthenticationException { // Implement code to authenticate the client with vault - // Used in binary connections for challenges } } Vault Authentication Plugin
- 16. ### --- Authentication --- ### # Enable authentication authenticationEnabled=true # Autentication provider name list, which is comma separated list of class names authenticationProviders=org.apache.pulsar.vault.authentication.VaultAuthentictionProvider # Interval of time for checking for expired authentication credentials authenticationRefreshCheckSeconds=60 Configuring Auth Plugin broker.conf
- 17. Client BROKER Vault Authentication Provider Authentication Service 1 5 2 4 6 3 1. Client Request with Vault Token 2. Authenticate Client 3. Token pass to Vault for Authentication 4. Vault token info returned 5. Return user identity 6. Return result to client Pulsar Vault Authentication
- 19. Pulsar Secret Plugins Secrets Provider • Run in the instance • Provides secrets through the function context api Secrets Configurator • Runs on the server (Broker or Function Worker) • Determines the Secret Provider the instance should use
- 20. public interface SecretsProvider { // Initialize the SecretsProvider. default void init(Map<String, String> config) {} // Fetches a secret String provideSecret(String secretName, Object pathToSecret); } public class MySecretFunction implements Function<String, Void> { @Override public Void process(String input, Context context) throws Exception { final String password = context.getSecret("password"); context.getLogger().info("read secret password=" + password); return null; } } Example code SecretsProvider - Client Side Plugin
- 21. public interface SecretsProviderConfigurator { default void init(Map<String, String> config) {} void configureKubernetesRuntimeSecretsProvider(V1PodSpec ps, String container, Function.FunctionDetails details; void configureProcessRuntimeSecretsProvider(ProcessBuilder pb, Function.FunctionDetails detailsetails); Type getSecretObjectType(); default void doAdmissionChecks(AppsV1Api appsV1Api, CoreV1Api coreV1Api, String ns, Function.FunctionDetails details) {} String getSecretsProviderClassName(Function.FunctionDetails details); Map<String, String> getSecretsProviderConfig(Function.FunctionDetails details); } SecretsProviderConfigurator - Server Side Plugin Highlighted methods are used to setup secrets plugins on the instances
- 22. ######################## # Secrets ######################## secretsProviderConfiguratorClassName: org.apache.pulsar.vault.secrets.VaultSecretsProviderConfigurator secretsProviderConfiguratorConfig: vaultAddress: http://localhost:8200 tokenPath: /etc/auth/token Configuring Secret Plugins Secrets Configurator functions_worker.yml
- 23. Secrets Provider public class VaultSecretsProviderConfigurator implements SecretsProviderConfigurator { @Override public String getSecretsProviderClassName(Function.FunctionDetails details) { if (!isEmpty(functionDetails.getSecretsMap())) { if (Function.FunctionDetails.Runtime.JAVA == details.getRuntime()) { return "org.apache.pulsar.vault.secrets.VaultSecretsProvider"; } else if (Function.FunctionDetails.Runtime.PYTHON == details.getRuntime()) { return "python_secret_provider"; } } return null; } @Override public Map<String, String> getSecretsProviderConfig(Function.FunctionDetails details) { final Map<String, String> secrets = new HashMap<>(); secrets.put("vaultAddress", "http://localhost:8200"); secrets.put("tokenPath", "/var/auth/token"); return secrets; } Configuring Secret Plugins
- 24. Java Function Instance User Code final String password = context.getSecret("password"); Vault Secret Provider 1 2 3 4 1. Request secret from code 2. Secret request with token 3. Secret returned to plugin 4. Return secret value Vault Secret Provider
- 25. Pulsar Kubernetes Plugins Kubernetes Manifest Customizer • Runs on the server (Broker or Function Worker) • Enables customization to the K8s function specs Kubernetes Function Auth Provider • Runs on the server (Broker or Function Worker) • Determines the auth params passed to the instances
- 26. public interface KubernetesManifestCustomizer extends RuntimeCustomizer { default V1StatefulSet customizeStatefulSet(Function.FunctionDetails funcDetails, V1StatefulSet statefulSet) { return statefulSet; } default V1Service customizeService(Function.FunctionDetails funcDetails, V1Service service) { return service; } default String customizeNamespace(Function.FunctionDetails funcDetails, String currentNamespace) { return currentNamespace; } } KubernetesManifestCustomizer - Server Side Plugin
- 27. public interface KubernetesFunctionAuthProvider extends FunctionAuthProvider { public void configureAuthDataStatefulSet(V1StatefulSet sts, Optional<FunctionAuthData> o) {} public void configureAuthenticationConfig(AuthenticationConfig config, Optional<FunctionAuthData> o) { ** configures the client auth for the function instances } public Optional<FunctionAuthData> cacheAuthData(Function.FunctionDetails details, AuthenticationDataSource s) throws Exception { ** Optional<FunctionAuthData> returned is used in configureAuthenticationConfig } public Optional<FunctionAuthData> updateAuthData(Function.FunctionDetails details, Optional<FunctionAuthData> o, AuthenticationDataSource s) throws Exception { ** Optional<FunctionAuthData> returned is used in configureAuthenticationConfig } public void cleanUpAuthData(Function.FunctionDetails details, Optional<FunctionAuthData> o) throws Exception {} } KubernetesFunctionAuthProvider - Server Side Plugin
- 28. Java Function Instance User Code final String password = context.getSecret("password"); Vault Secret Provider 1 2 3 1. Request secret from code 2. Read secret from file 3. Return secret value Vault Secret Provider with Vault Agent
- 29. Packaging Plugins Where do my plugins go? pulsar/ instances/ lib/ authentication.jar secret-configurator.jar secret-provider.jar deps/ kubernetes-plugins.jar
- 31. pulsar functions vault zookeeper brokers bookies proxy
- 32. Pulsar Kubernetes Pod Pulsar Process Vault Agent Kubernetes JWT 4 3 2 1 1. Service Account JWT passed to Vault for Authentication 2. Vault Token auth returned 3. Write token to file 4. Pulsar process reads token from file Pulsar Vault Kubernetes Integration
- 33. Function Secret Configuration tenant: "public" namespace: "default" name: “secrets-printer" className: “secrets_printer” inputs: ["public/default/secrets-trigger"] autoAck: true parallelism: 1 resources: cpu: 0.5 ram: 536870912 disk: 536870912 secrets: username: path: "internal/data/database/config" key: username password: path: "internal/data/database/config" key: password customRuntimeOptions: >- { "serviceAccountName": "pf-secrets-printer" } Used by the VaultKubernetesCustomizer to add annotations for vault token and secret injection
- 34. Function Vault Annotations vault.hashicorp.com/role: pf-secrets-printer vault.hashicorp.com/agent-inject: 'true' vault.hashicorp.com/agent-inject-token: 'true' vault.hashicorp.com/agent-inject-secret-password: secret-path vault.hashicorp.com/agent-inject-template-password: | '{{- with secret "secret-path" }}{{ .Data.data.password }}{{ end }}'
- 35. Demo
- 36. Future Enhancements • Vault for certificate management • Pulsar Vault authorization plugin • Vault for data encryption
- 37. Resources • https://pulsar.apache.org/docs/en/security-overview/ • https://pulsar.apache.org/docs/en/security-authorization/ • https://pulsar.apache.org/docs/en/security-extending/ • https://github.com/hashicorp/vault-k8s • https://www.vaultproject.io/docs/platform/k8s/helm • https://www.vaultproject.io/docs/platform/k8s/injector • https://learn.hashicorp.com/vault/kubernetes/k8s-reference-architecture Pulsar Vault
- 38. Questions