Pod Stuck in Pending (FailedScheduling)

If a pod never starts and stays in Pending, with an event such as 0/5 nodes are available: 1 Insufficient cpu, Kubernetes cannot find a node to place it on. This page explains how to read the message, what each part means on this platform, and what to change.

Symptoms

• The pod status is Pending and never moves on to ContainerCreating or Running.
• The pod shows as progressing rather than healthy in SVIEW, and the Component has no logs, because no container has started yet.
• The pod events show a FailedScheduling warning with a message like this one:

0/10 nodes are available: 1 Insufficient cpu, 1 Insufficient memory, 2 node(s) had untolerated taint(s), 3 node(s) didn't match Pod's node affinity/selector, 3 node(s) didn't match pod anti-affinity rules. preemption: 0/10 nodes are available: 5 No preemption victims found for incoming pod, 5 Preemption is not helpful for scheduling.

Read the scheduling message

The message is a tally: for every node in the cluster, it gives the reason that
node was rejected. Read it as "no node passed every check", not as "the cluster
is full". The reasons often differ node by node, and the fix is usually the one
reason that applies to the nodes your pod is meant to run on. The preemption:
part only says that no lower-priority pod could be moved aside, and you can
ignore it.

In SVIEW. Go to Applications → Deployments and open the Component. In
the All resources card, click the pod. The Kubernetes manifest window
shows the live pod; under status.conditions, the PodScheduled condition
carries reason: Unschedulable and the full scheduling message.

With kubectl. If you have namespace access,
the events at the bottom of this command show the same message:

kubectl describe pod <pod-name> -n <your-namespace>

How new nodes are added

A FailedScheduling event on its own is normal for a few seconds. When no
existing node fits, the node autoscaler starts a new node sized for the pod's
resource requests, and the pod is scheduled onto it — usually within a couple
of minutes. See Cluster Autoscaling.

A pod that is still Pending after about five minutes means the autoscaler
could not add a node that fits either. The sections below cover why.

Insufficient cpu or Insufficient memory

The pod's requests are larger than the free capacity on the existing nodes.
Normally a new node is added for it. It stays Pending when:

• The request is larger than any node size the pool can start. A pod must
  fit on a single node, after the platform's own system pods on that node have
  taken their share. A request for more CPU or memory than the largest node
  size in the pool never schedules, no matter how long it waits.
• The node pool has reached its maximum size. Each node pool has an upper
  limit on its total CPU and memory. Once the pool is at that limit, no further
  nodes are added.

To fix it:

1. Open the Component, go to the Values tab, and lower resources.requests
   for CPU and memory. Requests are what the scheduler plans around, so size
   them to what the application actually uses, not to its peak. The limit can
   stay higher than the request. See
   Workload Resources and Health Checks for the format and for VPA
   sizing recommendations.
2. Click Save. The Component redeploys with the new requests.
3. If a large request is genuinely needed, or you suspect the pool is at its
   maximum, raise a request (see below). A larger node size or a higher pool
   limit is a change we make, and it adds cost.

node(s) didn't match Pod's node affinity/selector

The pod asks for a node with a particular label, through nodeSelector or
affinity in the Component's values, and those nodes did not match. Common
causes:

• A typo in the label key or value, or a pool name that does not exist in this
  Environment. The label that selects a node pool is pool-type, with values
  such as main, data, interruptible-jobs and uninterruptible-jobs.
• Values copied from another Environment that has a pool this one does not.

Correct or remove the nodeSelector or affinity block on the Values tab
and click Save. The job pools are described in
Cluster Autoscaling.

node(s) had untolerated taint(s)

Some nodes carry a taint that keeps ordinary workloads off them — the platform's
own system nodes, and the dedicated job pools. Seeing this for some nodes is
normal and harmless: those nodes were never meant for your pod.

It matters only when your pod is meant to run on a tainted pool. A pod
pinned to interruptible-jobs or uninterruptible-jobs needs both the node
affinity and the matching toleration; with the affinity alone, every node it
selects rejects it with this reason. Add the toleration shown in
Cluster Autoscaling, or, for a CronJob, use
dedicatedNodes: true as described in CronJobs.

node(s) didn't match pod anti-affinity rules

The pod has a required anti-affinity rule — for example, no two replicas on
the same node or in the same zone — and every eligible node already runs a pod
it must avoid. The platform's default spreading is soft and never blocks
scheduling on its own, so a required rule comes from the Component's values.

Either lower the replica count to what the rule can satisfy, or change the rule
from requiredDuringSchedulingIgnoredDuringExecution to
preferredDuringSchedulingIgnoredDuringExecution so that it spreads pods where
it can without blocking them.

pod has unbound immediate PersistentVolumeClaims

The pod uses a persistent volume claim that has no volume yet. Look at the
claim itself:

kubectl get pvc -n <your-namespace>
kubectl describe pvc <claim-name> -n <your-namespace>

• The claim is Pending with a storage class error. The storage class
  named in the Component's values does not exist in this cluster. Remove the
  storage class setting to use the default, or correct its name.
• The claim was just created. Provisioning can take a minute; the pod
  schedules once the volume exists.

A related message, node(s) had volume node affinity conflict, means the volume
already exists in one availability zone and no node that fits the pod is in
that zone. Block volumes cannot move between zones, so the pod can only run in
the zone its volume was created in. A new node in that zone is normally added
for it; if the pod still stays Pending, raise a request. For storage that
several pods in different zones share, see
Shared Storage (EFS).

Prevention

• Set requests from real usage, and keep the largest single pod well below the
  size of one node.
• Test nodeSelector, affinity and anti-affinity changes in a non-production
  Environment first, with the same replica count as production.

Still having issues?

Raise a request in your dedicated Slack channel with the namespace, the
Component name, and the full FailedScheduling message. Raise it straight away,
without changing values first, when:

• the pod has been Pending for more than about five minutes and its requests
  already fit on one node — the node pool may have reached its maximum size;
• you need a single pod larger than the current node sizes allow.

Related articles

• Cluster Autoscaling — node pools, job pools and pinning
• Workload Resources and Health Checks — resource requests and limits
• Pod Keeps Restarting (CrashLoopBackOff) — the pod starts but then fails
• Pod Killed with OOMKilled — the pod exceeded its memory limit
• Pod Restarted During Node Maintenance — a restart with no change on your side
• Namespace Access — for kubectl describe
• Troubleshooting Common Issues — find your symptom