Access Management

You need an application to read from or write to Kafka, and Kafka will not let
it in until that application has its own user, the right permissions, and the
credentials to present. This article covers the four jobs involved: granting
permissions (ACLs), getting credentials into your application, connecting a
local client for debugging, and managing topics.

Kafka on PushOps uses SASL authentication and ACL-based authorisation.
Every application that produces or consumes messages needs a dedicated user
whose permissions are scoped to only the topics it actually uses.

1. Granting permissions with ACLs

ACLs are declared in your Kafka ACL Component's configuration in SVIEW.
Each entry ties one user to one topic with a role of producer, consumer, or
admin. Consumers also name the consumer group they will join.

acls:
  - topic: my-service.events
    role: producer
    user: my-service-producer

  - topic: my-service.events
    role: consumer
    user: my-other-service-consumer
    group: my-consumer-group

Save the Component and the platform applies the change to Kafka automatically.

Adding a new consumer to an existing topic is a routine change you make
yourself. Creating a brand-new Kafka user is not self-service — raise a
request with the username and the topics it needs.

2. Getting credentials into your application

Credentials are injected into your application Component as environment
variables through its dependency block. You never copy a password by hand.

dependency:
  kafka:
    instance: <kafka-cluster-name>
    user: my-service-producer
    keys:
      - secretKey: username
        envKey: KAFKA_USERNAME
      - secretKey: password
        envKey: KAFKA_PASSWORD
      - secretKey: brokers
        envKey: KAFKA_BROKERS

Your application then reads KAFKA_USERNAME, KAFKA_PASSWORD, and
KAFKA_BROKERS from its environment at startup.

3. Connecting a local Kafka client for debugging

This is for ad-hoc inspection from your own machine. It requires Kubernetes
namespace access to the namespace the Kafka cluster runs in.

1. Request Kubernetes namespace access in SVIEW under Access requests.
2. Log in and connect to the cluster through Teleport:

tsh login --user <your-email> --proxy https://teleport.manage.cto2b.eu
tsh kube login <your-cluster-name>

3. Forward the Kafka service to a local port:

kubectl port-forward svc/kafka-service 9092:9092 -n <kafka-namespace>

4. Point your local Kafka client at localhost:9092 and authenticate with the
   same SASL credentials your application uses.

4. Managing topics

Topics are managed declaratively through the Kafka Topics application in
the Catalog. One Kafka Topics Component can manage any number of topics on
a single Kafka cluster.

To set it up:

1. In the Catalog, select the Kafka Topics application and choose the
   environment and Service.
2. Set the namespace to the namespace where your Kafka cluster is deployed.
3. Set the deployment name to match your Kafka cluster's deployment name.
4. List the topics and their configuration in the Component's values.

Replica and partition counts cannot be changed in place. Changing either
requires recreating the topic: remove it from the list, save, then re-add it
with the new values. Plan for the loss of that topic's data when you do.

Automatic topic creation is disabled on all clusters, so an application that
expects to create its own topic on first use will fail with a "topic not found"
error. Declare the topic here first.

Related articles

• BigQuery Connector
• Debezium CDC Connector (GCP Cloud SQL PostgreSQL)
• Access Requests