Kafka Access, ACLs and Topics

You need an application to read from or write to Kafka, and Kafka will not let
it in until that application has its own user, the right permissions, and the
credentials to present. This article covers the four jobs involved: granting
permissions (ACLs), getting credentials into your application, connecting a
local client for debugging, and managing topics.

Kafka on PushOps uses SASL authentication and ACL-based authorisation.
Every application that produces or consumes messages needs a dedicated user
whose permissions are scoped to only the topics it actually uses.

1. Granting permissions with ACLs

ACLs are declared in a Kafka ACL Component in SVIEW. Each Kafka ACL
Component defines one Kafka user, and that user automatically has full rights on
topics and consumer groups named <stage>.<tenant>.<user>.*. To let another
application in the same tenant read or write those topics, add it under acls:

tenant: my-tenant
user: orders
acls:
  - topic: orders.events          # becomes <stage>.my-tenant.orders.events
    role: reader
    user: billing                 # becomes <stage>-my-tenant-billing-user

Roles are reader, writer and admin for one topic, and multireader,
multiwriter and multiadmin for every topic with a prefix. How the names are
built, all the values, and what each role allows are in
Kafka Users, Topic Names and ACL Roles.

Save the Component and the platform applies the change to Kafka automatically.

Adding a new consumer to an existing topic is a routine change you make
yourself. Creating a brand-new Kafka user is not self-service — raise a
request with the username and the topics it needs.

2. Getting credentials into your application

Credentials are injected into your application Component as environment
variables through its dependency block. You never copy a password by hand.

dependency:
  kafka:
    instance: <kafka-cluster-name>
    user: my-service-producer
    keys:
      - secretKey: username
        envKey: KAFKA_USERNAME
      - secretKey: password
        envKey: KAFKA_PASSWORD
      - secretKey: brokers
        envKey: KAFKA_BROKERS

Your application then reads KAFKA_USERNAME, KAFKA_PASSWORD, and
KAFKA_BROKERS from its environment at startup.

3. Connecting from your own machine for debugging

Kafka accepts only SASL_SSL connections on port 9093, and a local client needs
the cluster's CA certificate, a user's JAAS configuration, and a forward to
every broker, not just the bootstrap service. The Kafka debug pod skips all of
that setup. Both ways are in
Connect to Kafka From Your Computer.

4. Managing topics

Topics are managed declaratively through the Kafka Topics application in
the Catalog. One Kafka Topics Component can manage any number of topics on
a single Kafka cluster.

Removing a topic from the list deletes it

🛑
The topic list is synchronised to the cluster in both directions. Topics you
add are created, and topics you remove from the list are deleted from the
cluster, along with their data. Treat a removal with the same care as
dropping a database table.

Set up the Kafka Topics Component

In the Catalog, select the Kafka Topics application and choose the
environment and Service.
Set the namespace to the namespace where your Kafka cluster is deployed.
Set the deployment name to match your Kafka cluster's deployment name.
List the topics and their configuration in the Component's values:
```
default_partitions: 3
default_replicas: 3
kafka_topics:
  - topic_name: dev.my-tenant.orders.events
    partitions: 6
  - topic_name: dev.my-tenant.orders.audit
    config:
      retention.ms: "604800000"   # 7 days
```
A topic without partitions or replicas takes the defaults. replicas
cannot be higher than the number of brokers, so a single-broker
non-production cluster needs default_replicas: 1. Name topics
<stage>.<tenant>.<user>.… so the owning Kafka user's default rights cover
them — the Component does not enforce this.

Replica and partition counts cannot be changed in place. Changing either
requires recreating the topic: remove it from the list, save, then re-add it
with the new values. Plan for the loss of that topic's data when you do.

Automatic topic creation is disabled on all clusters, so an application that
expects to create its own topic on first use will fail with a "topic not found"
error. Declare the topic here first.

1. Granting permissions with ACLs

2. Getting credentials into your application

3. Connecting from your own machine for debugging

4. Managing topics

Removing a topic from the list deletes it

Set up the Kafka Topics Component

Related articles