k8s jobs are great at running something once but terrible at running things in order. For example, if you have a pipeline of jobs that need to run sequentially, you cannot just apply them all at once and expect them to run in order. B will start even if A has not finished. B will start even if A has failed. B will start even if A has not started yet.
This article goes through the four approaches I have used to run sequential work on Openshift or k8s.
- Plain jobs sequenced by hand
- initContainers pulling double duty
- A cronjob that folds the chain into one script
- A workflow using Argo Workflows
This demo fetches the latest Argo Workflows release metadata from the Github, validate it with real python logic and then report. All though the demo is small, this is a realistic demo of a job that needs to run sequentially.
Setup
[ashutosh@lima-default ~]$ oc new-project ashutoshpatole-dev[ashutosh@lima-default ~]$ oc apply -f content/kubernetes/run-jobs-in-order/00-namespace.yamlapiVersion: v1kind: Namespacemetadata: name: ashutoshpatole-devEvery manifest below carries a restricted-v2 compliant securityContext (runAsNonRoot, RuntimeDefault seccomp, drop: ["ALL"]), so they run on OpenShift 4.18 without touching an SCC.
Part 1: Plain Jobs
Step one fetches with real curl flags on a UBI Minimal image.
apiVersion: batch/v1kind: Jobmetadata: name: release-fetch namespace: ashutoshpatole-dev labels: step: fetchspec: backoffLimit: 2 activeDeadlineSeconds: 300 template: metadata: labels: step: fetch spec: restartPolicy: Never securityContext: runAsNonRoot: true seccompProfile: type: RuntimeDefault containers: - name: fetch image: registry.access.redhat.com/ubi9/ubi-minimal:latest command: ["sh", "-c"] args: - | set -e echo "[fetch] querying GitHub releases API..." curl -fsSL --retry 3 --retry-delay 5 \ -H "Accept: application/vnd.github+json" \ https://api.github.com/repos/argoproj/argo-workflows/releases/latest \ -o /tmp/release.json echo -n "[fetch] latest release: " grep -o '"tag_name": *"[^"]*"' /tmp/release.json securityContext: allowPrivilegeEscalation: false runAsNonRoot: true capabilities: drop: ["ALL"]Step two switches images to python:3.12-slim and asserts on the real payload with stdlib only, so there is no pip install and no extra network to fail:
apiVersion: batch/v1kind: Jobmetadata: name: release-validate namespace: ashutoshpatole-dev labels: step: validatespec: backoffLimit: 2 activeDeadlineSeconds: 300 template: metadata: labels: step: validate spec: restartPolicy: Never securityContext: runAsNonRoot: true seccompProfile: type: RuntimeDefault containers: - name: validate image: python:3.12-slim command: ["sh", "-c"] args: - | set -e python3 - <<"EOF" import json, sys, urllib.request url = "https://api.github.com/repos/argoproj/argo-workflows/releases/latest" req = urllib.request.Request(url, headers={ "Accept": "application/vnd.github+json", "User-Agent": "release-check-demo", }) try: with urllib.request.urlopen(req, timeout=30) as resp: rel = json.load(resp) except Exception as exc: print(f"[validate] fetch failed: {exc}") sys.exit(1) tag = rel.get("tag_name", "") assert tag.startswith("v"), f"unexpected tag format: {tag!r}" assert rel.get("published_at"), "release has no published_at" assert rel.get("assets"), "release has no downloadable assets" print(f"[validate] {tag} published {rel['published_at']} " f"with {len(rel['assets'])} assets - OK") EOF securityContext: allowPrivilegeEscalation: false runAsNonRoot: true capabilities: drop: ["ALL"]Step three reports. In production this POSTs to Slack, so the webhook call ships in the file, commented, with the secret reference next to it:
apiVersion: batch/v1kind: Jobmetadata: name: release-report namespace: ashutoshpatole-dev labels: step: reportspec: backoffLimit: 2 activeDeadlineSeconds: 300 template: metadata: labels: step: report spec: restartPolicy: Never securityContext: runAsNonRoot: true seccompProfile: type: RuntimeDefault containers: - name: report image: registry.access.redhat.com/ubi9/ubi-minimal:latest command: ["sh", "-c"] args: - | set -e TAG=$(curl -fsSL --retry 3 \ -H "Accept: application/vnd.github+json" \ https://api.github.com/repos/argoproj/argo-workflows/releases/latest \ | grep -o '"tag_name": *"[^"]*"' | cut -d'"' -f4) echo "[report] release $TAG validated - ready for rollout review" # Production version: create the secret, uncomment the env and # the POST below, and the same pod reports to Slack. # curl -fsSL -X POST -H "Content-Type: application/json" \ # -d "{\"text\":\"release $TAG validated\"}" \ # "$SLACK_WEBHOOK_URL" # env: # - name: SLACK_WEBHOOK_URL # valueFrom: # secretKeyRef: { name: slack-webhook, key: url } securityContext: allowPrivilegeEscalation: false runAsNonRoot: true capabilities: drop: ["ALL"]The correct run looks like this:
[ashutosh@lima-default ~]$ oc apply -f 01-job-fetch.yamljob.batch/release-fetch created [ashutosh@lima-default ~]$ oc wait --for=condition=complete job/release-fetch -n ashutoshpatole-dev --timeout=120sjob.batch/release-fetch condition met [ashutosh@lima-default ~]$ oc apply -f 02-job-validate.yamljob.batch/release-validate created [ashutosh@lima-default ~]$ oc wait --for=condition=complete job/release-validate -n ashutoshpatole-dev --timeout=120sjob.batch/release-validate condition met [ashutosh@lima-default ~]$ oc apply -f 03-job-report.yamljob.batch/release-report created [ashutosh@lima-default ~]$ oc get pods -n ashutoshpatole-devNAME READY STATUS RESTARTS AGErelease-fetch-jtph6 0/1 Completed 0 56srelease-report-2rfmh 0/1 Completed 0 10srelease-validate-9c8h8 0/1 Completed 0 26sThis works untile someone tries this in hurry. If you apply all three at once, the cluster starts everything in parallel so the report can print its verdict on a release that was never validated. You can try yourself, change the last assertion in 02 to require prerelease (the latest stable release is not one, so validation fails), apply all three, and watch the report run anyway. The dependency is a promise you make to yourself, not something the cluster enforces.
Manual oc wait sequencing is fine for a one-off, but it is not a pipeline.
Part 2: InitContainers
The first fix most people reach for is stuffing the stages into one Pod as initContainers. They run sequentially, and the app container only starts if all of them succeed:
# Init containers run sequentially and a failure stops the chain,# so this LOOKS like ordering done right.apiVersion: v1kind: Podmetadata: name: release-check namespace: ashutoshpatole-devspec: restartPolicy: Never securityContext: runAsNonRoot: true seccompProfile: type: RuntimeDefault initContainers: - name: fetch image: registry.access.redhat.com/ubi9/ubi-minimal:latest command: ["sh", "-c"] args: - | set -e echo "[fetch] querying GitHub releases API..." curl -fsSL --retry 3 --retry-delay 5 \ -H "Accept: application/vnd.github+json" \ https://api.github.com/repos/argoproj/argo-workflows/releases/latest \ -o /tmp/release.json echo -n "[fetch] latest release: " grep -o '"tag_name": *"[^"]*"' /tmp/release.json securityContext: allowPrivilegeEscalation: false runAsNonRoot: true capabilities: drop: ["ALL"] - name: validate image: python:3.12-slim command: ["sh", "-c"] args: - | set -e python3 - <<"EOF" import json, sys, urllib.request url = "https://api.github.com/repos/argoproj/argo-workflows/releases/latest" req = urllib.request.Request(url, headers={ "Accept": "application/vnd.github+json", "User-Agent": "release-check-demo", }) try: with urllib.request.urlopen(req, timeout=30) as resp: rel = json.load(resp) except Exception as exc: print(f"[validate] fetch failed: {exc}") sys.exit(1) tag = rel.get("tag_name", "") assert tag.startswith("v"), f"unexpected tag format: {tag!r}" assert rel.get("published_at"), "release has no published_at" assert rel.get("assets"), "release has no downloadable assets" print(f"[validate] {tag} looks good") EOF securityContext: allowPrivilegeEscalation: false runAsNonRoot: true capabilities: drop: ["ALL"] - name: report image: registry.access.redhat.com/ubi9/ubi-minimal:latest command: ["sh", "-c"] args: - | set -e TAG=$(curl -fsSL --retry 3 \ -H "Accept: application/vnd.github+json" \ https://api.github.com/repos/argoproj/argo-workflows/releases/latest \ | grep -o '"tag_name": *"[^"]*"' | cut -d'"' -f4) echo "[report] release $TAG validated - ready for rollout review" securityContext: allowPrivilegeEscalation: false runAsNonRoot: true capabilities: drop: ["ALL"] containers: # Placeholder for your real app. It cannot start until all three # initContainers succeed, which is exactly the problem with this pattern. - name: app image: registry.access.redhat.com/ubi9/ubi-minimal:latest command: ["sleep", "infinity"] securityContext: allowPrivilegeEscalation: false runAsNonRoot: true capabilities: drop: ["ALL"][ashutosh@lima-default ~]$ oc apply -f 04-pod-initcontainers.yaml[ashutosh@lima-default ~]$ oc get pod release-check -n ashutoshpatole-dev[ashutosh@lima-default ~]$ oc logs release-check -n ashutoshpatole-dev -c validateEventhough it works, do not ship it because of five reasons:
- Init containers are for setup, not real workloads. They are designed for quick prep tasks before an app starts, like checking if a database is reachable or running schema migrations. Heavy batch jobs or report generation do not belong here, and anyone reading your code will not expect to find business workflows hidden inside an init container.
- If one step fails, everything restarts from zero. Pods do not let you retry a single container. If step three fails, the entire pod restarts from step one and runs everything again from scratch. Kubernetes Jobs, on the other hand, let each stage retry independently using
backoffLimit. - Locked to a single machine. Every init container must run on the exact same server (node) as the pod. If that node reboots, runs out of memory, or gets drained for maintenance, the entire sequence crashes and has to start all over again. Independent Jobs can be scheduled anywhere in the cluster.
- No automatic cleanup. Kubernetes Jobs automatically delete or clean up old runs after they finish. Regular pods do not; completed pods stay in the cluster indefinitely until someone or an automated script manually deletes them.
- Your main app is blocked. The primary container cannot start until every single init container finishes. If a batch step gets stuck or takes an hour to complete, your web app cannot boot or serve user traffic. Live web services and background batch tasks should never share a pod.
Part 3: CronJobs
Another common workaround is combining all three steps into a single script and running it on a schedule using a Kubernetes CronJob:
apiVersion: batch/v1kind: CronJobmetadata: name: nightly-chain namespace: ashutoshpatole-devspec: schedule: "0 2 * * *" concurrencyPolicy: Forbid successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 3 jobTemplate: spec: backoffLimit: 1 activeDeadlineSeconds: 600 template: metadata: labels: step: chain spec: restartPolicy: Never securityContext: runAsNonRoot: true seccompProfile: type: RuntimeDefault containers: - name: chain image: python:3.12-slim command: ["sh", "-c"] args: - | set -e python3 - <<"EOF" import json, urllib.request url = "https://api.github.com/repos/argoproj/argo-workflows/releases/latest" print("[fetch] querying GitHub releases API...") req = urllib.request.Request(url, headers={ "Accept": "application/vnd.github+json", "User-Agent": "release-check-demo", }) with urllib.request.urlopen(req, timeout=30) as resp: rel = json.load(resp) tag = rel.get("tag_name", "") print(f"[fetch] latest release: {tag}") print("[validate] checking metadata...") assert tag.startswith("v"), f"unexpected tag format: {tag!r}" assert rel.get("published_at"), "release has no published_at" assert rel.get("assets"), "release has no downloadable assets" print(f"[validate] {tag} looks good") print(f"[report] release {tag} validated - ready for rollout review") print("[chain] nightly run complete - PASS") EOF securityContext: allowPrivilegeEscalation: false runAsNonRoot: true capabilities: drop: ["ALL"]This runs automatically every night at 2 AM, and the steps run in order. But because the entire workflow is hidden inside one script, Kubernetes has no visibility into what is actually happening. This leads to three main problems:
- No step-by-step visibility: Kubernetes only knows whether the whole script passed or failed. You cannot see which specific stage broke without digging through one long, mixed stream of logs.
- All-or-nothing retries: If the final reporting step fails due to a network glitch, Kubernetes restarts the whole container. That means step one (fetching) and step two (validating) must run all over again.
- Tight coupling: If you need to change how the validation step works, you have to touch and re-test the entire monolithic script rather than updating an isolated task.
In short, a CronJob solves when work starts, but it does not manage dependencies between steps. To Kubernetes, your pipeline is still just one black box.
Part 4: Argo Workflows
Instead of writing custom scripts to handle execution order, a workflow engine lets you declare the exact sequence directly in Kubernetes manifests. This is where Argo Workflows comes in.
Argo Workflows adds native workflow support to your cluster (installed via the Argo Workflows operator). The step containers below use the exact same images and commands from Part 1, but now Kubernetes manages the execution order:
apiVersion: argoproj.io/v1alpha1kind: Workflowmetadata: generateName: release-check- namespace: ashutoshpatole-devspec: entrypoint: pipeline ttlStrategy: secondsAfterCompletion: 86400 onExit: notify templates: - name: pipeline steps: - - name: fetch template: fetch - - name: validate template: validate - - name: report template: report - name: fetch container: image: registry.access.redhat.com/ubi9/ubi-minimal:latest command: ["sh", "-c"] args: - | set -e echo "[fetch] querying GitHub releases API..." curl -fsSL --retry 3 --retry-delay 5 \ -H "Accept: application/vnd.github+json" \ https://api.github.com/repos/argoproj/argo-workflows/releases/latest \ -o /tmp/release.json echo -n "[fetch] latest release: " grep -o '"tag_name": *"[^"]*"' /tmp/release.json - name: validate # Flaky validation gets its own retry budget. The fetch is NOT re-run. retryStrategy: limit: 2 container: image: python:3.12-slim command: ["sh", "-c"] args: - | set -e python3 - <<"EOF" import json, sys, urllib.request url = "https://api.github.com/repos/argoproj/argo-workflows/releases/latest" req = urllib.request.Request(url, headers={ "Accept": "application/vnd.github+json", "User-Agent": "release-check-demo", }) try: with urllib.request.urlopen(req, timeout=30) as resp: rel = json.load(resp) except Exception as exc: print(f"[validate] fetch failed: {exc}") sys.exit(1) tag = rel.get("tag_name", "") assert tag.startswith("v"), f"unexpected tag format: {tag!r}" assert rel.get("published_at"), "release has no published_at" assert rel.get("assets"), "release has no downloadable assets" print(f"[validate] {tag} published {rel['published_at']} " f"with {len(rel['assets'])} assets - OK") EOF - name: report container: image: registry.access.redhat.com/ubi9/ubi-minimal:latest command: ["sh", "-c"] args: - | set -e TAG=$(curl -fsSL --retry 3 \ -H "Accept: application/vnd.github+json" \ https://api.github.com/repos/argoproj/argo-workflows/releases/latest \ | grep -o '"tag_name": *"[^"]*"' | cut -d'"' -f4) echo "[report] release $TAG validated - ready for rollout review" - name: notify container: image: registry.access.redhat.com/ubi9/ubi-minimal:latest command: ["sh", "-c"] args: - | echo "[notify] pipeline {{workflow.status}} ({{workflow.name}})"The reason why its better than shell scripts or init containers:
- Guaranteed order without custom scripting. Each step runs only after the previous step succeeds. The
reportstep only starts oncevalidatefinishes, andvalidateonly starts oncefetchfinishes. - Independent retries for each step. With
retryStrategy, a flaky network call invalidatewill only retry the validation step. It will not waste time rerunning thefetchstep. - Guaranteed notifications on exit. The
onExit: notifyhandler runs whether the workflow succeeds or fails. If the pipeline breaks halfway through, you still get a notification or failure alert.
Submit it and watch:
[ashutosh@lima-default ~]$ oc create -f 06-workflow-sequential.yaml -n ashutoshpatole-dev[ashutosh@lima-default ~]$ oc get workflows -n ashutoshpatole-devIf you simulate a failure (for example, by requiring prerelease in the validation step and resubmitting), watch what happens:
- The
validatestep fails after attempting its configured retries. - The
reportstep never starts because its prerequisite failed. - The workflow status updates to
Failed. - The
notifystep still runs to report the failure.
That is the biggest advantage: Kubernetes now natively understands the dependencies between your steps instead of blindly running commands.
Summary: Which approach should you choose?
Here is a quick cheat sheet:
- One-off commands from your terminal: Run standalone Jobs and chain them with
kubectl wait(oroc wait). - Setup work before an app starts: Use
initContainers(for things like database migrations before your web server boots). - Simple recurring tasks with no step dependencies: Use a standard
CronJobrunning a single script. - Multi-step pipelines where step B depends on step A: Use a workflow engine like Argo Workflows. If validation must pass before reporting, or a build must pass before deployment, define the dependencies explicitly in a Workflow manifest.
The moment correctness depends on order, that order belongs in declarative Kubernetes manifests, not hidden inside custom shell scripts or ad-hoc terminal commands.
When complex jobs or initContainers fail to complete as expected, pods can easily get trapped in ambiguous initialization states. For practical diagnostics when containers refuse to reach ready status or pod specs cannot be mutated, see our troubleshooting teardowns in Pod Is Running but Not Ready and The Pod That Refused to Be Edited.