Get Alerted When a Scheduled Job Fails

This page shows you how to get an alert — by e-mail or Slack — when a scheduled job fails or silently stops running, so a broken nightly export or cleanup does not go unnoticed for days.

How it works

Alerts your team receives are set up in your own Grafana, which comes with a dedicated observability stack. CTO2B's own platform monitoring does not send your team a message when one of your jobs fails, so without your own alert a failure can go unnoticed.

Your stack's Grafana includes a Cronjobs dashboard with the job metrics you alert on. It has no alert rules set up for you: you choose which jobs matter and who hears about them.

Before you start

• A dedicated observability stack. If you do not have one, raise a request — see Observability.
• Grafana Admin access, requested in SVIEW under Access requests — see Observability for the steps.

Set up the alert

1. In Grafana, go to Alerting → Contact points and add where alerts should go — an e-mail address or a Slack webhook.

2. Go to Alerting → Alert rules and create a rule on the job metrics from the Cronjobs dashboard. Alerting on how long ago the job last succeeded catches both failures and runs that never happened — for example a job due every day that has not succeeded for longer than a day plus a margin:

   Text

   time() - kube_cronjob_status_last_successful_time{namespace="your-namespace", cronjob="nightly-export"} > 93600

   Replace the namespace, the job name and the number of seconds with your own: one schedule interval plus some margin.

3. Set the rule to alert when there is no data for critical jobs, so a missing metric does not look like success.

4. Point the rule at your contact point and save it.

Why "last success" rather than "failed"

A finished job's pod is removed shortly after it ends, so an alert that waits a few minutes for a failed job to appear can miss it. A rule on the time since the last success does not depend on the failed job still being there. The same removal means the failed run's output is no longer on the pod: read it from your logs — see Find Your Application Logs.

A run that exceeds its maximum run time is stopped and counts as a failure.

Want CTO2B to set it up?

Raise a request with:

• the jobs and namespaces, and the Environment (usually production only);
• who should receive alerts, and how (e-mail addresses or a Slack channel);
• what counts as a failure — a failed run, a missed run, or a run longer than a given time;
• whether alerts should go out around the clock or in business hours only.

Related articles

• CronJobs — creating and running scheduled jobs
• Observability — the dedicated stack and Grafana access
• Find Your Application Logs — reading a job's output after it has finished
• Event-Driven Autoscaling with KEDA — dashboards and alerts for event-driven jobs