Partition Creation with Kubernetes CronJobs

When the application already runs on Kubernetes, the scheduler is right there — and the failure modes are different from cron on a host. This guide runs partition maintenance as one CronJob per shard, with concurrency policy, history limits, credentials handled properly and, most importantly, alerting that does not depend on the job’s own exit status. It is the container-native counterpart to the approaches in Automated Partition Creation Workflows, part of Partitioning Implementation Patterns & Routing.

Prerequisites

Step 1 — One CronJob per shard, generated from a template

# partition-maintenance.yaml — rendered once per shard by kustomize or helm
apiVersion: batch/v1
kind: CronJob
metadata:
  name: partition-maintenance-shard3
  labels: {app: partition-maintenance, shard: shard3}
spec:
  schedule: "17 * * * *"            # hourly, off the hour
  timeZone: "Etc/UTC"
  concurrencyPolicy: Forbid          # never two runs against one shard
  startingDeadlineSeconds: 900       # skip rather than pile up after an outage
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 10
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 600
      template:
        spec:
          restartPolicy: Never
          serviceAccountName: partition-maintenance
          containers:
            - name: maintenance
              image: registry.internal/db-maintenance:2026.08.1
              args: ["--shard", "shard3", "--tables", "events,metrics"]
              env:
                - name: PGHOST
                  value: shard3.internal
                - name: PGPASSWORD
                  valueFrom:
                    secretKeyRef: {name: shard3-maintenance, key: password}
              resources:
                requests: {cpu: 50m, memory: 64Mi}
                limits:   {cpu: 500m, memory: 256Mi}

Operational note: concurrencyPolicy: Forbid plus the advisory lock inside the procedure gives two independent guarantees. Either alone is adequate; both together mean a change to one does not silently remove the protection.

DBA tip: startingDeadlineSeconds matters after a control-plane outage. Without it, Kubernetes may try to run every missed schedule at once, sending a burst of DDL at a shard that has just come back.

One job per shard makes the failure legible A single CronJob looping over eight shards fails as a whole when one shard is unreachable, leaving the other seven ambiguous and requiring a full re-run. Eight per-shard CronJobs fail independently: the Kubernetes job list shows exactly which shard is behind, retries affect only that shard, and the other seven completed successfully. One CronJob looping over shards One CronJob per shard maintenance job status: Failed shard0 ✓ shard1 ✓ shard2 ✓ shard3 ✗ unreachable shard4 ? shard5 ? shard6 ? shard7 ? the loop aborted at shard3; shards 4–7 were never attempted, and the job status says only "Failed" a retry re-runs everything, including the shards that already succeeded shard0✓ shard1✓ shard2✓ shard3✗ retrying shard4✓ shard5✓ shard6✓ shard7✓ seven shards are provably current; one is retrying on its own backoff, and kubectl get jobs is the status dashboard Generating eight manifests from one template costs nothing and turns "the maintenance job failed" into "shard3 is behind", which is the difference between an investigation and a fact.

Step 2 — Make the container do one thing, idempotently

#!/usr/bin/env python3
# maintain.py — the whole container entrypoint
import argparse, os, sys, psycopg

def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--shard", required=True)
    ap.add_argument("--tables", required=True)
    args = ap.parse_args()

    with psycopg.connect(autocommit=False) as conn:
        with conn.cursor() as cur:
            cur.execute("CALL maintain_partitions()")     # idempotent, advisory-locked
            cur.execute("""SELECT table_name,
                                  (newest_bound - current_date) AS days_ahead,
                                  pending_retire
                           FROM v_partition_health""")
            rows = cur.fetchall()
        conn.commit()

    push_metrics(args.shard, rows)          # to the push gateway
    for table, days_ahead, pending in rows:
        print(f"{args.shard} {table}: {days_ahead}d ahead, {pending} pending retire")
        if days_ahead < 21:
            print(f"WARNING {table} horizon is only {days_ahead} days", file=sys.stderr)
    return 0

if __name__ == "__main__":
    sys.exit(main())
Missed schedules: replayed in a burst, or skipped During a four-hour control-plane outage, four hourly runs are missed. Without startingDeadlineSeconds, Kubernetes attempts all four as soon as it recovers, sending a burst of DDL at a shard that has just come back. With a fifteen-minute deadline, the missed runs are skipped and the next scheduled run proceeds normally — which is safe because the job is idempotent and converges. No startingDeadlineSeconds startingDeadlineSeconds: 900 control plane unavailable — 4 runs missed control plane unavailable — 4 runs missed 4 runs fire at once next normal run next normal run — missed ones skipped Skipping is correct because the job converges: one run creates every partition the missed runs would have. A job doing incremental work would need the opposite setting.

Operational note: The job pushes the health metrics itself. That is what makes an absent job detectable — the metric goes stale, and staleness is alertable, whereas a job that never ran produces no event at all.

SRE tip: Keep the image small and the entrypoint single-purpose. A maintenance container that also runs migrations, backups and reports becomes a thing nobody dares to schedule hourly.

Step 3 — Alert on the database, not on the job

# primary alert: the horizon, published by the job, per shard and table
min by (shard, table) (partition_days_ahead) < 21

# staleness: the job stopped publishing at all
time() - max by (shard) (push_time_seconds{job="partition-maintenance"}) > 7200

# secondary: Kubernetes noticed a failure
kube_job_status_failed{job_name=~"partition-maintenance-.*"} > 0
Three signals, and the failure each one is blind to The job-failure signal catches a run that errored but not one that was never scheduled. The metric-staleness signal catches a suspended or deleted CronJob but not one that runs and silently does nothing. The horizon signal catches every case where partitions are not being created, regardless of cause, which is why it is the primary alert. Signal Catches Blind to kube_job_status_failed secondary a run that errored a CronJob that was suspended, deleted, or never applied metric staleness secondary the job stopped running at all a job that runs and does nothing because its role lost permissions partition_days_ahead primary every cause of "partitions are not being created" nothing that matters — it measures the outcome rather than the process Keep all three, but page only on the horizon. The other two are diagnostic: they tell you why the horizon stopped moving, which is useful at 3 a.m. and not worth waking someone for on its own.

Operational note: A CronJob removed by a bad kubectl apply --prune produces no failures, no events and no logs. Only a measurement taken from the database notices.

DBA tip: Publish the metric with a shard and table label so one alert rule covers the fleet and the notification names the affected shard.

Verification

Prove the whole chain, including the parts that only matter when something is wrong:

# 1. run it now rather than waiting for the schedule
kubectl create job --from=cronjob/partition-maintenance-shard3 pm-manual-1
kubectl logs job/pm-manual-1
shard3 events: 92d ahead, 0 pending retire
shard3 metrics: 87d ahead, 1 pending retire

# 2. confirm concurrency is actually forbidden
kubectl create job --from=cronjob/partition-maintenance-shard3 pm-manual-2
# the advisory lock makes the second run a no-op; check its logs say so
# 3. simulate the invisible failure
kubectl patch cronjob partition-maintenance-shard3 -p '{"spec":{"suspend":true}}'
# the staleness alert must fire within its window; the horizon alert follows days later

The third check is the one worth doing. A suspended CronJob is the most common real failure and the only one that produces no error anywhere.

Where This Sits Relative to the Other Schedulers

Running maintenance as a Kubernetes CronJob is not better or worse than pg_cron or an external workflow engine — it is a different set of trade-offs, and the right one depends on what the team already operates.

Against pg_cron, the CronJob wins on visibility and loses on proximity. A CronJob’s history, logs and failures land in the same place as every other workload, and the maintenance image is deployed with the same pipeline as the application. What it gives up is being inside the database: pg_cron cannot be blocked by a network partition, does not need credentials, and keeps working when the Kubernetes control plane is unavailable.

Against Airflow or a similar engine, the CronJob wins on simplicity and loses on orchestration. There is no dependency graph, no backfill semantics and no cross-shard coordination — which is exactly right for a job whose steps are idempotent and independent, and exactly wrong if partition maintenance is one node in a larger data pipeline that must run in order.

The practical guidance is to match the scheduler to the on-call rotation. Whoever is paged when partitions stop being created should already be fluent in the tool that creates them, because the first debugging step is always “did it run, and what did it say?”.

Failure mode table

Failure mode Root cause SRE mitigation
Maintenance silently stops the CronJob was suspended or pruned by a deployment change alert on metric staleness and on the partition horizon; both are independent of the job’s own status
A burst of runs after a control-plane outage missed schedules were replayed once the control plane recovered set startingDeadlineSeconds so missed runs are skipped rather than queued
Two runs collide during a slow maintenance pass concurrencyPolicy left at the default Allow set Forbid, and keep the advisory lock inside the procedure as the second layer

FAQ

One CronJob for all shards or one per shard?

One per shard, generated from the same template. A single job that loops over shards fails as a unit, hides which shard is behind, and cannot be retried selectively. Per-shard jobs make the Kubernetes UI itself the status dashboard and let one unreachable shard fail without affecting the others.

Why not rely on the CronJob's failure status for alerting?

Because the most common failure is that the job did not run at all — suspended, deleted by a bad apply, or scheduled on a node pool that no longer exists. A failed job is visible; an absent job is not. Alert on the partition horizon measured from the database, and treat job failures as a secondary signal.

How should database credentials reach the job?

Through a mounted secret or a workload identity that the job exchanges for a short-lived token, never as a literal in the manifest. The job needs CREATE on the schema and ownership of the partitioned tables, which is a strictly smaller grant than the application’s role — use a dedicated maintenance role rather than reusing the app credentials.

Back to Automated Partition Creation Workflows