Jump to Content

Debezium CDC Troubleshooting

Your Debezium CDC connector is not running, has stopped producing events, or is
failing with an error you do not recognise. This article lists the errors that
come up most often, what each one actually means, and how to fix it. For
initial setup, see Kafka — Debezium CDC Connector (GCP Cloud SQL
PostgreSQL).

Common errors

Error	Cause	Fix
`password authentication failed for user "debezium"`	Wrong password, or the `DirectoryConfigProvider` path wrongly includes the filename in the directory part	Use `${dir:...:DB_PASSWORD}` — the directory ends at `db-credentials`, the filename comes after the second colon
`insufficient privileges to start walsender`	The Debezium database user is missing the `REPLICATION` privilege	Grant replication privileges to the user. On Cloud SQL: `GRANT cloudsql_superuser TO debezium;`
`UNKNOWN_TOPIC_OR_PARTITION` in producer logs	The topic does not exist, and brokers do not auto-create topics	Add `topic.creation.default.replication.factor` and `topic.creation.default.partitions` to the connector config
`Not authorized to describe topic(s) '__debezium-heartbeat.*'`	Your ACLs do not cover the heartbeat topic prefix	Add a `rawAcls` entry for it, or override `heartbeat.topics.prefix` so it falls under a prefix you already grant
`Creation of replication slot failed`	Missing replication privilege, or the connector points at a read replica	Grant replication privileges, and point the connector at the primary instance
`permission denied for table <view>`	`GRANT ON ALL TABLES` also covered views, which Debezium cannot read	Grant `SELECT` per table instead of schema-wide
`replication slot "debezium_x" already exists`	A slot was left behind by a previous run	Run `SELECT pg_drop_replication_slot('debezium_x');` then restart the connector
`Snapshot: relation "myschema.x" does not exist`	The table is not part of the publication	Run `ALTER PUBLICATION debezium_pub ADD TABLE myschema.x;`
Automatic restarts stop retrying	The `maxRestarts` limit was reached	Annotate the connector with `strimzi.io/restart="true"` to reset it, and omit `maxRestarts` so retries are unlimited

Checking connector state

Start in SVIEW: open the Kafka Connector Component and check whether the
connector and its tasks report a running state, and look at the Kafka Connect
Component's logs for the failure message.

For a closer look you need Kubernetes namespace access to the Kafka
namespace. Request it in SVIEW under Access requests, then connect through
Teleport.

# All connectors and their state
kubectl get kafkaconnector -n <kafka-namespace>

# Full status including the error trace
kubectl get kafkaconnector cdc-mydb-psql -n <kafka-namespace> \
  -o jsonpath='{.status.connectorStatus}' | python3 -m json.tool

# Restart a failed connector
kubectl annotate kafkaconnector cdc-mydb-psql -n <kafka-namespace> \
  strimzi.io/restart="true" --overwrite

Checking replication lag on the database

Run this against the source database to see how far behind the connector is. A
lag that grows steadily means the connector is not consuming, and the database
cannot clean up its write-ahead log in the meantime.

SELECT slot_name, confirmed_flush_lsn, pg_current_wal_lsn(),
       (pg_current_wal_lsn() - confirmed_flush_lsn) AS lag_bytes
FROM pg_replication_slots WHERE slot_name = 'debezium_mydb';

Preventing the problems that recur

Set heartbeat.interval.ms so the replication slot stays current during
quiet periods.
Leave autoRestart.enabled: true and omit maxRestarts, so a transient
failure does not leave the connector stopped.
If the connector will be down for an extended period, drop the replication
slot rather than leaving it inactive and letting the database's storage fill.

Still having issues?

Raise a request including the connector name, the environment, the exact error
text from the connector status, and roughly when events stopped arriving.

Related articles

Updated about 6 hours ago

Did this page help you?