Secret Not Found or Not Reaching Your Application

If your application reports a secret as not found, a pod will not start because a secret is missing, or a key you just added (or changed, rotated or updated) is not being read, this article walks you through the checks you can do yourself.

Most of these problems come from one of four places: the secret is not where the application looks for it, the application does not ask for the key, the application has not been restarted since the value changed, or the stored JSON is not valid. Work through the checks below in order.

Symptoms

• A pod does not start, and its events show an error similar to this one:

  Text

  MountRotationFailed
  provider mount err: rpc error: code = Unknown desc = Failed to fetch secret from all regions: /<stage>/<environment>/<namespace>/<secret-name>

• The pod starts, but the application logs say an environment variable is empty or a setting is missing.

• The pod is in CrashLoopBackOff because the application exits when a required value is missing.

• You added a new key to a JSON secret, the secret looks correct in Secret Manager, but the application still does not see the key.

• You changed a secret value, but the application keeps using the old one.

Check 1: the secret exists where the application looks

Failed to fetch secret from all regions means no secret exists at the path shown in the error. The path is built from the Stage, Environment and Namespace of the application, so a secret created in a different one is invisible to it.

1. Copy the path from the error message.
2. Go to Secret Manager in the sidebar and find the secret.
3. Compare its Stage, Environment and Namespace with the path in the error. They must match exactly.
4. If the secret is for one application (not Namespace-wide), check that the Application name matches the application that reads it.

If there is no matching secret, create it in the right place, as described in Secrets Manager. A secret created in the development Environment is not copied to staging or production: create it in each Environment that needs it.

Check 2: the application asks for the key

Creating a secret does not deliver it to any application. The application only receives the keys that are listed in its Component configuration. This is also true for a Namespace-wide secret.

1. Open the application Component and find the Secrets card.
2. Click Copy YAML snippet on the secret.
3. Compare the snippet with the dependency block in the Component's configuration. Every key the application needs must be listed.
4. If a key is missing, add it, keeping the indentation of the lines around it (spaces, never tabs), and click Save. The Component redeploys.

A new key in a JSON secret must be added to this list too. Adding the key to the secret in Secret Manager updates the stored value, but the application only receives keys that are listed in its configuration. This is the most common reason a new key "is synced but not read".

Also check the spelling and case of each key. Unless followKeysFormat is set to true, key names are converted to upper case before they reach your application, so smtp_password arrives as SMTP_PASSWORD. See Inject Secrets from Parameter Store for all the options.

Check 3: the application has restarted since the change

Most applications read environment variables once, when they start. If you change a secret value, or add a key, while the application is running, it keeps the values it read at startup until its pods restart.

• Saving the Component configuration redeploys it, so a fix made in Check 2 also restarts the pods.

• If you changed only the secret value and not the configuration, restart the application so that it reads the new value. SVIEW has no restart button: if you have namespace access with a role that allows changes, restart it with kubectl, or raise a request in your dedicated Slack channel and CTO2B restarts it. See Restart an Application or a Single Pod for the steps.

  Shell

  kubectl rollout restart deployment/<deployment-name> -n <your-namespace>

For secrets that come from 1Password, a change can take up to 2 minutes to be synchronised into your namespace. Wait 2 minutes after the change, then restart the application.

Check 4: the JSON secret is valid

When JSON format is on, all keys are stored as one JSON object. If that JSON is not valid, the keys cannot be read.

• Values that contain double quotes (") or backslashes (\) must be escaped as \" and \\. A certificate or private key must have its line breaks written as \n, on one line.

• Check the value with a JSON validator before you save it. If you have jq installed, a quick check is:

  Shell

  jq . my-secret.json

• Cloud secret stores limit the size of one secret. If a JSON secret has grown very large, for example because it holds several certificates, split it into more than one secret. On AWS, a Secrets Manager secret holds up to 64 KB. SVIEW creates Parameter Store secrets as standard parameters, which hold up to 4 KB each; with JSON format on, that limit applies to the whole JSON object, because all keys are stored together.

What you can and cannot see

Seeing a secret's value, and creating or editing secrets, need specific Secret Manager roles — see Who can see and change secrets. If you cannot open or edit a secret, ask your own administrator for the role first.

Still having issues?

If a secret exists in the right place, the keys are listed in the Component configuration, and the application has been restarted, raise a request in your dedicated Slack channel with:

• the application name, Stage, Environment and namespace
• the name of the secret, and whether it is per-application or Namespace-wide
• whether it is stored in AWS Secrets Manager, Parameter Store or 1Password
• the dependency block from the Component configuration
• the exact error message, and where you saw it (pod events, application logs, SVIEW)
• what you have already checked from this page

Related articles

• Secrets Manager — creating secrets and connecting them to an application
• Inject Secrets from Parameter Store — the ssm configuration reference
• Inject Secrets from 1Password — the onepassword configuration reference
• Variables and Shared Config — for values that are not secret
• Pod Keeps Restarting (CrashLoopBackOff) — when the application exits at startup