Kubernetes CronJob concurrency policy for paid API jobs

A Kubernetes CronJob's concurrencyPolicy decides what happens when a run comes due while the previous Job is still running: Allow, the default, runs both; Forbid skips the new run; Replace replaces the running Job with a new one. When the Job calls a paid API that runs for minutes, such as AI video generation, use Forbid and make the call idempotent, because stopping a pod doesn't stop work the API has already accepted.

Kubernetes facts come from its CronJob, Jobs, and kubectl create job pages; Sume facts come from Create a run, Runs and results, and Errors and spend. All were read on 2026-09-27. Sume has no Kubernetes operator: the pod makes one plain HTTPS call.

What does each concurrencyPolicy value do to a paid API call?

The policy only applies to Jobs created by the same CronJob. Kubernetes also warns that a CronJob creates a Job only approximately once per scheduled time: in some circumstances it creates two, or none, so the Jobs you define should be idempotent.

Does Replace or activeDeadlineSeconds stop the API spend?

No. Both act on Jobs and their pods. Once a Job reaches activeDeadlineSeconds, Kubernetes terminates its running pods and marks it failed with reason DeadlineExceeded; Replace swaps the Job. Neither reaches the API. On Sume, abandoning a poll loop does not stop the run or its spend, and a timeout does not cancel it. A Format run ends on its own at the latest 90 minutes after created_at, when Sume force-finalizes it as failed.

• To stop spend, cancel the run itself: POST /v1/format-runs/{run_id}/cancel with formats:write. Generation the run completed before the cancel is still billed, and a canceled run sends no webhook.
• Generation jobs, such as a POST /v1/videos job, cancel only before generation starts. After that, cancel answers 409 job_generation_already_started and the job runs to completion.

How do I stop a retried or duplicate Job from paying twice?

Build the Idempotency-Key and the body from the scheduled slot, such as the UTC date for a daily run, and nothing else. Every attempt at that slot then sends the identical request: a pod retry (a Job's backoffLimit defaults to 6, with back-off delays of 10 s, 20 s, 40 s and so on, capped at six minutes), a second Job for the same time, or a manual rerun. Keys are scoped to one Format and are up to 255 characters.

• Same key, same body: 200 with the original receipt and idempotency_hit: true. No second run, no second charge.
• Same key, different body: 409 idempotency_conflict, and nothing runs. Never put a timestamp or a random value in the body.
• Same key, two requests at the same moment: one wins; the other gets 409 idempotency_key_in_use, which is retryable after about a second.
• A create that failed (402, 503, …) released the key, so the pod's retry can still start the slot's run.

#!/bin/sh
# submit-nightly.sh: one Format run per UTC day, then exit
set -eu
SLOT=$(date -u +%Y-%m-%d)  # UTC, like the CronJob's timeZone
curl --fail-with-body -sS -X POST \
  "https://api.sume.com/v1/formats/acme/daily-recap/runs" \
  -H "Authorization: Bearer $SUME_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: daily-recap-$SLOT" \
  -d "{
    \"input\": { \"date\": \"$SLOT\" },
    \"on_active_run\": \"skip\",
    \"communication\": { \"webhook_url\": \"https://example.com/hooks/sume\" }
  }"

Should the pod wait for the video to finish?

No: submit and exit. A pod that waits keeps its Job running for the whole render, so under Forbid a render that outlasts the interval skips the next slot, and any deadline kills only the waiting. With communication.webhook_url, Sume POSTs the run's receipt once, when it completes or fails, to a receiver you run as an ordinary Deployment; the receipt's trigger.idempotency_key tells it which slot finished. --fail-with-body makes curl return an error on a status of 400 or more while still printing Sume's error, so a refused submit fails the pod.

A pod that exits in seconds also means Forbid no longer sees the render, so guard overlap on Sume's side too. A Format run's on_active_run defaults to allow; skip records a skipped run and starts nothing, and reject answers 409 format_run_in_progress. Prevent overlapping AI agent runs compares the two.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: daily-recap
spec:
  schedule: "0 2 * * *"
  timeZone: "Etc/UTC"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 3
      activeDeadlineSeconds: 300  # bounds the submit, not the video
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: submit
              image: registry.example.com/submit-nightly:1  # sh, date, curl
              command: ["/bin/sh", "/app/submit-nightly.sh"]
              env:
                - name: SUME_API_KEY
                  valueFrom:
                    secretKeyRef:
                      name: sume-api
                      key: api-key

How do I trigger a CronJob manually, and in which time zone?

kubectl create job daily-recap-manual --from=cronjob/daily-recap creates a Job from the CronJob's template; CronJob is the only resource --from supports. Because the key comes from the slot, a manual run on a day that already ran replays that day's run and starts nothing. To force a fresh one, add a version you bump to the key, which is how Sume's docs say to request a deliberate re-run.

Set .spec.timeZone, stable since Kubernetes v1.27, to a zone name such as Etc/UTC. Without it, the kube-controller-manager reads the schedule in its own local time zone, and the pod's slot and the schedule can disagree around midnight.