← All checklists

Checklist 03 of 10 · September 2026

Kubernetes probe checklist for AI services

Keep a slow dependency from restarting every pod

A liveness probe that checks a dependency turns a slow database into a restart storm: every replica fails the same probe in the same window, and the kubelet restarts them all together. The demo never shows it, because the demo ran one pod. Every default and behaviour below is from the Kubernetes documentation; nothing here is a vendor claim. Five checks; the first two are an afternoon's work.

Prove it in two commands

kubectl get pods                 # RESTARTS rises on every replica in the same minute
kubectl describe pod <pod>       # Events: "Liveness probe failed: ..." - then open the probe's handler

The checks

Everything starts ticked. Untick what you will not do; the choice is saved on this device only, and Print keeps only what you kept.

Why. The kubelet restarts a container when its liveness probe fails three times in a row, probing every ten seconds with a one-second timeout by default. Point that probe at an endpoint that calls your database, vector store or model server, and one thirty-second stall fails it on every replica at once, so every replica restarts together. Capacity goes to zero while the dependency was only slow.

The check.

  • Open every livenessProbe. If its handler touches anything over the network, move that check out.
  • Liveness answers one question: can this process respond at all?

Source: Kubernetes documentation, Configure Liveness, Readiness and Startup Probes

Why. A failing readiness probe marks the pod unready and takes it out of Service traffic; it does not restart the container. For an AI service that holds a model in memory, that is the difference between a pause and a cold start.

The check.

  • Give the pod two endpoints: /livez (process only) and /readyz (database, vector store, model loaded).
  • The pod leaves traffic while a dependency is down and returns the moment it is back, model still loaded.
livenessProbe:  { httpGet: { path: /livez,  port: 8080 } }   # process only, no I/O
readinessProbe: { httpGet: { path: /readyz, port: 8080 } }   # database, vector store, model loaded

Source: Kubernetes documentation, Configure Liveness, Readiness and Startup Probes

Why. Liveness and readiness checks do not start until the startup probe has succeeded. The documentation's rule: set failureThreshold x periodSeconds long enough to cover the worst-case startup time (its example allows 30 x 10 = 300 seconds).

The check.

  • Time your worst cold start, weights downloaded, loaded and warmed, and size the startup probe to cover it.
  • Do not stretch the liveness probe's initial delay instead.

Source: Kubernetes documentation, Configure Liveness, Readiness and Startup Probes

Why. An HTTP probe that answers after its timeout counts as a failure, whatever it would have returned. The default timeout is one second.

The check.

  • Keep the liveness handler constant-time and free of I/O.
  • If it cannot answer inside the timeout under peak load, fix the handler before you raise the timeout.

Source: Kubernetes documentation, Configure Liveness, Readiness and Startup Probes

Why. One restart is noise; every replica restarting inside the same minute usually means something they all share, and the events say whether it was the liveness probe. kube-state-metrics exposes restarts per container as kube_pod_container_status_restarts_total.

The check.

  • Alert when restarts rise on several replicas of one Deployment within the same window.
  • Link the alert to the two commands on this page.

Source: kube-state-metrics (pod metrics documentation in the repository)

Work it in this order

#ControlEffortImpact
1Move dependency checks out of every liveness probe1-2 hoursCritical
2Add a readiness endpoint that checks the dependencies2-4 hoursCritical
3Size a startup probe to the worst cold start2-4 hoursHigh
4Alert on restarts across replicashalf a dayHigh
5Time the probe handlers under peak loadhalf a dayMedium

Effort figures are planning estimates for a team that already runs Kubernetes, not measurements. Everything else on this page is from the Kubernetes documentation.

Take this list with youThe PDF is the handout: this list on one or two pages. The Markdown is for your own docs, your wiki, or an assistant you want to hand the list to. Print gives you this list only, without the navigation.

Something in your production AI broke in a way that is not on this list? Tell me what broke.

New checklists, as they are published

One list at a time, each built the same way: a dated failure, the control that stops it, and the primary source so you can check it. Nothing else goes to this address, no digest and no newsletter.