CronJob Scheduling

A CronJob does not run anything itself directly - its entire job is to create ordinary Jobs (as described above) on a recurring schedule, using the same familiar * cron syntax. Because scheduled runs can sometimes take longer than the interval between them, concurrencyPolicy controls what happens if a new scheduled run is due while the previous one is still active: Allow (the default) lets them overlap and run simultaneously, Forbid skips the new run entirely if the old one has not finished, and Replace kills the still-running old Job and starts the new one in its place. startingDeadlineSeconds bounds how late a run is allowed to start (for example, if the CronJob controller itself was briefly down) before that scheduled run is abandoned entirely rather than started dangerously late, and the successfulJobsHistoryLimit/failedJobsHistoryLimit fields simply cap how many old completed Jobs are kept around for inspection versus garbage-collected. None of these settings are cosmetic - time zone configuration, brief controller downtime, and schedules that land close to a boundary (like exactly midnight, or a daylight-saving transition) all require you to explicitly reason through what "correct" behavior looks like, rather than assuming the defaults will do the sensible thing.

Analogy: Kubernetes is a thermostat, not a remote control. You declare the temperature; independent controllers measure reality and keep acting. Debugging means finding which sensor, rule or actuator prevents convergence.

Read the object as evidence

kubectl get RESOURCE NAME -n NAMESPACE -o yaml
kubectl describe RESOURCE NAME -n NAMESPACE
kubectl get events -n NAMESPACE --sort-by=.metadata.creationTimestamp
kubectl explain RESOURCE.spec

Do not memorize these as a ritual. The first command exposes desired and observed state, the second connects conditions and events, the third supplies a timeline, and the fourth checks the server's schema. Compare what the controller was asked to do with what it reports doing. Then test the narrowest hypothesis.

Scenario: A daily report job is scheduled to run hourly by mistake, and on a particularly busy day it takes 70 minutes to finish - longer than the hour between scheduled runs. With concurrencyPolicy: Allow, the next hour's copy starts anyway while the first is still running, so now two copies contend for the same resources and possibly write conflicting output. Switching to Forbid would skip that overlapping run entirely, and Replace would kill the first run outright - but notice that neither setting actually fixes the underlying problem: if the report generation logic is not itself safe to interrupt or rerun (non-idempotent), both options just trade one flavor of corrupted output for another.

Treat schedules as triggers, not clocks

CronJobs create Jobs according to controller observation, time zone configuration and startingDeadlineSeconds. concurrencyPolicy Allow permits overlap, Forbid skips while an old run remains active, and Replace terminates the old Job for a new one. Set history limits and TTL to control object growth. Design every scheduled task for delayed, duplicate or missed execution.

Scenario: A daily billing job runs longer after month end. Allow overlaps two writers and duplicates invoices; Forbid skips the next schedule. A lease plus idempotent business keys handles both timing outcomes explicitly.
Warning: Suspending a CronJob stops new Jobs, not already-created ones. Inspect active Jobs before assuming the workload has stopped.

Production reasoning

Ask four questions: Who owns this object? What dependency must become ready next? Which controller reports the blocking condition? What evidence would disprove my current theory? This prevents symptom-driven changes. Record the context, namespace, object generation, image digest and recent rollout before mutation; a recreated Pod may erase the evidence you needed.

Warning: Running is not the same as ready, healthy, durable or correct. Kubernetes status is layered. Confirm the application-level outcome as well as the object state.
Goal: Put this model into practice in the Kubernetes lab cronjob-concurrency-deadline. Open /labs/kubernetes, choose cronjob-concurrency-deadline, predict the failure path before changing anything, then use the simulator's check command to validate the finished state.

Deliberate practice

Before the lab, write the expected object relationship and the first three commands you will run. Afterward, explain why the fix converged and name one tempting change that would only mask the symptom. Repeat using an explicit namespace and a structured output format. This prediction-observation-explanation loop is what turns command familiarity into production judgment.

45-minute investigation

  1. Map (5 min): draw the owner-to-child chain and mark every namespace, selector, identity and dependency involved.
  2. Predict (5 min): write one expected status condition, one likely event and one log or metric signal before opening the lab.
  3. Observe (10 min): collect YAML, describe output and ordered events. Do not mutate state. Record which observation disproves your first theory.
  4. Repair (15 min): make the smallest declarative correction, watch the responsible controller converge, and verify the user-facing path rather than stopping at Running.
  5. Stress (10 min): change one relevant constraint - replica count, label, readiness, resource value or placement rule - predict the outcome, observe it, then restore the known-good declaration.
Tip: Keep a short incident note with symptom, evidence, hypothesis, change, result. Across four sections this produces a reusable runbook instead of a pile of remembered commands.