Five ways to run a workflow: Airflow, Dagster, Prefect, Temporal and Argo

"Workflow orchestration" covers tools with genuinely different models of what a workflow is. Comparing them feature-by-feature produces a matrix where everything scores well, because each tool has grown the features its competitors advertise. The more useful comparison is architectural: what is the unit of work, what is the scheduler's job, and what happens when something fails halfway through.

This is a survey, not a recommendation. All five are mature, all five are in production at large organisations, and the right choice depends far more on your workload than on any ranking. Details change release to release — verify current behaviour against each project's own documentation before committing.

Apache Airflow

The unit is a task in a DAG, and the organising idea is scheduled batch. You define a directed graph in Python, the scheduler decides which tasks are runnable, and executors run them. Airflow's model is strongest when work is periodic, the graph is known ahead of time, and tasks are coarse — "run this transform, then that export."

Its longevity is a real asset: a very large operator ecosystem, and a labour market where hiring someone who knows it is easy. The historical criticisms concern the scheduling model itself — the tight coupling of scheduling to execution intervals, and awkwardness with dynamic graphs whose shape depends on runtime data. Airflow 3, generally available since 2025, addressed a good deal of this — notably event-driven scheduling, so a DAG can be triggered by an external event such as a file landing in object storage rather than only on an interval. If your mental model is "a cron that understands dependencies," Airflow is the canonical implementation.

Dagster

The unit is an asset — a table, a file, a model — and the graph is derived from what produces what. This inverts Airflow: instead of describing tasks and inferring the data, you describe data and infer the tasks.

For data platforms this framing pays off. Lineage is native rather than reconstructed, "what is stale and needs rebuilding" is a first-class query, and the type and partition system catches a class of error before a run starts. Local development and testing are deliberate design goals, which teams coming from harder-to-test orchestrators tend to notice immediately.

The asset framing is also the constraint. Work that isn't really about producing data — a long-running approval process, a per-customer state machine — fits less naturally than it does in a general task runner.

Prefect

The unit is a Python function, and the pitch is that your existing code becomes a workflow with decorators rather than a rewrite. Dynamic control flow — loops, branches, graph shapes decided at runtime — is native rather than a workaround, because the graph is whatever the code does.

That makes it approachable for teams whose pipelines are already Python and who want scheduling, retries, observability and failure handling without adopting a new structural vocabulary. The flip side of "it's just Python" is that the framework enforces less: the discipline Dagster imposes through asset definitions is left to you.

Temporal

The odd one out, and the most different. The unit is a durable execution — a workflow function whose entire state, including its position in the code, survives process death, deploys and machine failure. It works by recording every side effect to an event history and replaying it deterministically on recovery.

This solves a problem the others don't attempt: long-running, stateful processes measured in days or months. Order fulfilment, subscription lifecycles, human approval chains, multi-step sagas with compensation. If you've ever built state machines in a database with a cron sweeping for stuck rows, Temporal is the general form of that.

The cost is a genuine programming model, not a configuration format. Workflow code must be deterministic; side effects go through activities; the replay semantics need to be understood rather than absorbed by osmosis. Teams report a real learning curve. Temporal is also not a data pipeline tool — using it for nightly batch ETL is possible and usually the wrong instinct.

Argo Workflows

The unit is a container, and the scheduler is Kubernetes. Workflows are custom resources; each step is a pod. If your platform is already Kubernetes and your steps are heterogeneous — some Python, some Go, some a vendor's image — the model is a clean fit, and you inherit the cluster's scheduling, autoscaling and resource isolation for free.

The trade is that everything is a container. Per-step overhead makes fine-grained tasks expensive, the authoring experience is YAML rather than a programming language, and the operational burden is Kubernetes' burden. Off Kubernetes, it isn't a candidate.

Choosing between them

The questions that actually discriminate:

Is your graph known before the run starts? If yes, Airflow and Argo are comfortable. If the shape depends on data, Prefect and Dagster handle it more naturally.

Is the output data, or an outcome? Producing tables points toward Dagster. Driving a process to completion points toward Temporal.

How long does one run last? Minutes to hours: any of them. Days to months, surviving deploys: Temporal is in a category of its own here.

What runs the code? Already all-in on Kubernetes with polyglot steps: Argo. Python shop that wants minimal ceremony: Prefect.

Who operates it? A platform team that can run a scheduler, a database and workers has every option. A team without that capacity should weigh managed offerings — all five have them, with varying maturity — or a hosted platform, above any architectural preference.

A closing caveat worth taking seriously: most orchestration pain is not caused by picking the wrong tool. It's caused by tasks that aren't idempotent, by pipelines with no backfill story, and by failures nobody is alerted to. Those problems follow you across every migration.