Data pipeline orchestration in 2026: revisit Apache Airflow, Dagster and Prefect before you schedule another DAG

Manoj Shukla
October 13, 2026

Picture a two-wheeler loan company in Coimbatore with about ninety branches (an invented example, not a real client). Its data team of five runs about 140 cron jobs on two Linux VMs: shell scripts that pull collections files from a payment gateway SFTP, Python scripts that clean them, and SQL that loads a PostgreSQL reporting database every night. A few newer pipelines live on an Airflow 2.10 instance someone set up two years ago. Month-end is painful. When the gateway file arrives late, the 2 AM job runs on yesterday’s data, the 3 AM job builds the branch dashboard on top of it, and on the first working day someone reruns six scripts by hand in the right order and hopes nothing loaded twice. Airflow 2 has now reached end of life, so the team must move anyway. One engineer wants to upgrade to Airflow 3 because everybody knows Airflow. A second wants Dagster because the problem is really about tables, not jobs. A third wants Prefect because the scripts are already plain Python and she does not want to rewrite them into DAG files.

All three are open-source orchestrators for batch and ETL work: they decide when a piece of pipeline code runs, in what order, with what retries, and they record what happened. They are not data processing engines (Spark, dbt or SQL do the heavy lifting) and not durable execution engines for per-request business workflows. This post compares Apache Airflow, Dagster and Prefect for a team about to schedule new pipelines or migrate old ones: history, architecture, tasks versus assets, scheduling semantics, backfills, retries, local testing, Kubernetes, metadata databases, access control, licences and security. Facts come from release notes, official documentation, licence files and CVE records, checked on 13 October 2026. The drill near the end is my own run on this Linux box.

The short version

An orchestrator does not make a pipeline safe to rerun. Decide first what one run is responsible for (a date, a partition, a file) and make every load idempotent for that key. Then pick a tool.
Apache Airflow 3.3.2 (17 September 2026, 4:40 PM IST) is the current release. Airflow 2 reached end of life on 22 April 2026, so 2.x users have a real migration in front of them: new airflow.sdk imports, a separate DAG processor, an API server between workers and the metadata database, catchup off by default and manual runs without a logical date.
Dagster 1.13.26 (10 October 2026, 4:38 AM IST) is the current release, with integration libraries at 0.29.26. It is built around software-defined assets. Dagster OSS is Apache 2.0; authentication, role-based access control and audit logs are listed as Dagster+ (the commercial service) additions.
Prefect 3.8.8 (6 October 2026, 9:59 PM IST) is the current release and the last to support Python 3.10. Flows and tasks are decorated Python functions; deployments, work pools and workers move them to real infrastructure. The self-hosted server offers basic authentication only; RBAC, SSO and audit logs are Prefect Cloud features.
Security in 2026: GitHub lists 60 advisories for the apache-airflow and apache-airflow-core packages published this year, including a critical serialised DAG deserialisation issue fixed in 3.3.0 and twelve fixes in 3.3.1. Dagster fixed an SQL injection in its database I/O managers; Prefect fixed several authentication bypasses in its self-hosted server. Run the latest patch release of whichever you choose.
In my drill on one VM, all three ran the same three-step pipeline, retried an injected failure once and loaded the same eight rows. Airflow 3 also showed why the semantics matter: two manually triggered runs had no logical date, so my date-keyed load wrote duplicate rows.

Where things stand on 13 October 2026

Dates are from each project’s GitHub release page unless marked PyPI; times are converted from UTC to IST. Licences are from the LICENSE files at the tagged release. The Airflow 2 dates come from the official “Supported versions” page.

Project

Release

Date (IST)

Notes

Licence

Apache Airflow

3.3.2

17 September 2026, 4:40 PM

Current release; Task SDK 1.3.2

Apache 2.0

Apache Airflow

3.3.1

12 August 2026, 3:20 PM

Twelve security fixes (GitHub advisories)

Apache 2.0

Apache Airflow

3.3.0

6 July 2026, 8:46 PM

Asset partition mappers, task and asset state store, pluggable retry policies, experimental Java and Go task SDKs

Apache 2.0

Apache Airflow

3.2.0

7 April 2026, 8:21 PM

Asset partitioning, multi-team deployments (experimental), Python 3.14 support, SQLAlchemy 2.0

Apache 2.0

Apache Airflow

3.0.0

22 April 2025, 7:44 PM

Service-oriented architecture, Task SDK, React UI, assets, scheduler-managed backfills

Apache 2.0

Apache Airflow

2.11.2

14 March 2026, 11:37 PM (PyPI)

Last 2.x patch; 2.x limited maintenance from 22 October 2025, end of life 22 April 2026

Apache 2.0

Airflow Helm chart

1.22.0

14 June 2026, 5:07 AM

Official chart for Airflow 2.11+ and 3.0+

Apache 2.0

Dagster

1.13.26

10 October 2026, 4:38 AM

Current release (libraries 0.29.26); weekly cadence

Apache 2.0

Dagster

1.13.0

10 April 2026, 12:27 AM

Partitioned asset checks, virtual assets (preview), state-backed components by default

Apache 2.0

Prefect

3.8.8

6 October 2026, 9:59 PM

Current release; last release supporting Python 3.10

Apache 2.0

Prefect

3.8.0

24 July 2026, 3:19 AM

V2 UI default for self-hosted servers

Apache 2.0

Prefect Helm charts

2026.10.6163015

6 October 2026, 10:00 PM

prefect-server and prefect-worker charts

Apache 2.0

All three declare Python 3.10 to 3.14 support on PyPI; Prefect’s 3.8.8 notes say later releases will need Python 3.11.

Timeline

Date (IST)

Event

October 2014

Maxime Beauchemin starts Airflow at Airbnb (Airflow project page)

13 April 2015

apache/airflow repository created on GitHub; Airbnb announces Airflow in June 2015

March 2016

Airflow enters the Apache Incubator

30 April 2018

dagster-io/dagster repository created

12 June 2018

First Dagster package on PyPI (0.1.0rc1)

30 June 2018

PrefectHQ/prefect repository created

8 January 2019

The ASF announces Apache Airflow as a Top-Level Project

24 March 2019

First Prefect package on PyPI (0.5.0)

17 December 2020

Airflow 2.0.0

24 February 2022

Prefect 1.0.0

28 July 2022

Prefect 2.0.0 on PyPI, a rewrite around plain Python flows

5 August 2022

Dagster 1.0.0

3 September 2024

Prefect 3.0.0

22 April 2025

Airflow 3.0.0

22 October 2025

Airflow 2 enters limited maintenance (security and critical fixes only)

7 April 2026

Airflow 3.2.0

10 April 2026

Dagster 1.13.0

22 April 2026

Airflow 2 end of life

Three tools, three ages. Airflow is the oldest and is governed by the Apache Software Foundation. Dagster and Prefect started in 2018, each backed by a company (Dagster Labs and Prefect Technologies) that also sells a hosted product built on the open-source core.

First question: what does one run own?

Before comparing features, write down what a single run of each pipeline is responsible for. For the Coimbatore team, the nightly collections job owns “collections for business date D”. The branch dashboard owns “the dashboard table as of D”. The month-end RBI report owns “month M”. Once a run owns a key like this, three things become easy in every tool: a rerun replaces data for that key instead of adding to it, a backfill is just “run these keys”, and a late file means “run D again” rather than “run six scripts in order”.

Cron hides this question because every job runs “now”. Orchestrators name it differently:

Airflow calls it the logical date and the data interval of a DAG run, and in Airflow 3 it also offers asset partitions.
Dagster calls it a partition of an asset, such as one day of a daily-partitioned table.
Prefect leaves it to you, usually as a flow parameter such as business_date, with an optional cache key on tasks.

If your current scripts cannot be rerun for a given date without side effects, fix that first. No orchestrator will fix it for you, and my drill below shows how easily it goes wrong.

What these three tools actually are

Apache Airflow 3.3.2

Airflow describes workflows as DAGs (directed acyclic graphs) of tasks written in Python. Airflow 3 changed the runtime more than the authoring style. The main parts now are:

Scheduler: decides which DAG runs and task instances are due and hands them to an executor. The executor logic runs inside the scheduler process.
DAG processor: parses DAG files and stores serialised DAGs in the metadata database. In Airflow 3 it must be started separately, even locally (airflow dag-processor).
API server: replaces the old webserver. It serves the React UI, the REST API (/api/v2) and the Task Execution API that workers use. Workers no longer talk to the metadata database directly; the upgrade guide calls the API server the sole access point for tasks and workers.
Triggerer: runs deferrable operators and event triggers asynchronously.
Executors: LocalExecutor is the default and works with SQLite for development; CeleryExecutor and KubernetesExecutor are the common production choices, and several executors can now be configured at once. SequentialExecutor, CeleryKubernetesExecutor and LocalKubernetesExecutor were removed in 3.0.
Task SDK: DAG authors import dag, task, Asset and friends from airflow.sdk. Since 3.3.0 an experimental coordinator layer lets individual tasks run in Java or Go while the DAG stays in Python.

Airflow 3 also brought DAG versioning, scheduler-managed backfills, assets (the renamed and redesigned datasets), and DAG bundles that replace --subdir. Since 3.2 it has asset partitioning and an experimental multi-team mode.

Dagster 1.13.26

Dagster starts from the data, not the job. You declare software-defined assets: a Python function decorated with @asset that produces a table, file or model, with its upstream assets as parameters. Dagster builds the lineage graph from those parameters and runs the steps needed to materialise the assets you select. Ops and jobs still exist for task-style work, but most new code is written as assets.

A Dagster OSS deployment has three long-running services, according to its deployment architecture page:

dagster-webserver: the UI and GraphQL API; can have several replicas.
dagster-daemon: runs schedules, sensors, run queuing and run monitoring; only one replica is supported.
Code location servers: gRPC servers that load your definitions, one per code location, each with its own Python environment. This is how separate teams ship separate dependency sets to one Dagster instance.

Storage, executors and run launchers are configurable; the Kubernetes run launcher starts each run as a Kubernetes job. dagster dev starts the webserver and daemon together for local work, and the newer dg CLI handles project scaffolding and launches.

Prefect 3.8.8

Prefect’s unit is the flow: a Python function decorated with @flow, calling functions decorated with @task. You can run a flow by calling it, exactly like a script, and Prefect records the run against an API. That API is either Prefect Cloud or a self-hosted Prefect server (FastAPI plus a database). If no server is configured, Prefect starts a temporary local one, as it did in my drill.

To run flows on a schedule or on other machines you create a deployment. A deployment stores when, where and how a flow should run. Where is decided by a work pool: a hybrid pool needs a worker you run (process, Docker, Kubernetes, ECS and others), which polls the pool and starts flow runs; push and managed pools submit runs to serverless infrastructure without a worker. For simple cases, flow.serve() runs a long-lived process that both schedules and executes a flow without a work pool.

Tasks, assets and flows: three ways to think

Idea

Apache Airflow 3

Dagster

Prefect 3

Unit you write

Task (operator or @task) in a DAG

Asset (@asset), or op in a job

Task (@task) inside a flow (@flow)

How dependencies are declared

Task wiring (>> or TaskFlow return values), plus asset schedules between DAGs

Function parameters name upstream assets

Normal Python calls and futures

What a run is keyed on

Logical date and data interval, or asset partition

Asset partition (time, static, dynamic, multi)

Whatever parameters you pass

Data-aware scheduling

schedule=[Asset(...)], asset watchers for external events

Declarative automation, sensors, freshness and asset checks

Event triggers and automations on deployments

Passing data between steps

XCom (small values; custom XCom backends for objects)

I/O managers store and load asset values

Return values in memory; result persistence optional

Lineage view

Asset graph and DAG graph

Asset graph is the main UI

Flow and task run graph

In Airflow you mostly think “run these tasks in this order for this interval”. Airflow 3 assets let one DAG run when another DAG updates an asset, and 3.3 adds partition mappers so a daily asset can feed a weekly rollup, but the DAG is still the unit you schedule. In Dagster you think “keep these tables fresh”, and a job is something Dagster derives from an asset selection. In Prefect you think “run this Python function reliably”, and any data model is yours to impose.

The Coimbatore team’s nightly work is mostly tables, which fits Dagster closely, but its SFTP pulls and API calls fit Airflow tasks or Prefect flows better. Each tool can do both; ask which style your team will keep clean after a year.

Scheduling semantics: data intervals, catchup and time zones

This is where most migration bugs live, so read it slowly.

Airflow 3 changed two defaults. catchup_by_default is now False, so a new DAG with an old start_date no longer creates a run for every missed interval when you unpause it. And create_cron_data_intervals is now False, so a bare cron string uses CronTriggerTimetable instead of CronDataIntervalTimetable. With the old timetable, a daily run at midnight processed the previous day and its logical date was the start of that interval. With the trigger timetable the logical date is the time the run fires. The upgrade guide says to set create_cron_data_intervals=True before the upgrade if your tasks rely on data_interval_start, data_interval_end, ds or ts meaning the old thing; changing it after Airflow 3 runs exist skips one scheduled run.

Manual runs changed too. Airflow 3 lets DAG runs have logical_date=None, and the REST API defaults to that when triggering a new run. The upgrade guide warns not to assume a manual run’s data interval matches a supplied logical date. Several context variables (execution_date, prev_ds, next_ds, yesterday_ds and others) were removed. If your Airflow 2 tasks build file names or DELETE statements from ds, test a manual trigger before you trust it.

Time zones need one more line. ds is rendered from the logical date in UTC. In my drill, a @daily DAG with an Asia/Kolkata start date fired at midnight IST, its logical date was 18:30 UTC on the previous day, and ds came out as the previous calendar date. That can be right or wrong for you, but it must be a decision, not a surprise.

Dagster schedules are cron-based and evaluated by the daemon. For time-partitioned assets you usually build the schedule from the partitions definition, so each tick targets one partition key (for example 2026-10-12), and the partition key, not the wall clock, decides what the code processes. The daemon writes a heartbeat you can see on the Deployment page; if it was down, check which partitions were missed and launch a backfill for them rather than trusting the schedule to have caught up.

Prefect deployment schedules (cron, interval or RRule, each with a time zone) are handled by the server’s scheduler service, which creates upcoming runs ahead of time. The docs describe the limits: no more than 100 runs, no more than 100 days ahead, at least three runs and at least one hour ahead by default. There is no data interval concept; if a run should process yesterday, pass business_date as a parameter or compute it from the scheduled start time in a stated time zone.

Backfills, retries and dynamic work

Need

Apache Airflow 3

Dagster

Prefect 3

Backfill a date range

airflow backfill create or UI; scheduler-managed since 3.0; reprocess behaviour none, failed or completed; max_active_runs; run backwards; dry run

Backfill over a partition range from UI or CLI; one run per partition, or a single run for a range with BackfillPolicy.single_run()

No Airflow-style backfill command in the docs I read; create parameterised runs for each date (CLI, API or a parent flow)

Step retries

retries, retry_delay, exponential backoff; pluggable retry policies since 3.3.0

RetryPolicy(max_retries, delay, backoff, jitter) or raise RetryRequested

retries, retry_delay_seconds (number or list), exponential_backoff, retry_condition_fn

Whole-run retries

Clear and rerun; DAG-level retry via callback

Run retries via run_retries in dagster.yaml (handled by the daemon)

@flow(retries=...)

Fan-out

Dynamic task mapping with .expand(); max_map_length default 1024

Dynamic outputs in ops; dynamic partitions for assets

Ordinary Python loops, task.map() and task runners

Rerun with which code

Bundle version: original for clear and rerun, latest for backfills by default (rerun_with_latest_version, 3.3.0)

Code location version at launch

Whatever the deployment points to at run time

Two notes from the docs. Airflow’s backfill will not create a run for a date whose latest run is queued or running, whatever reprocess behaviour you choose. Dagster’s single-run backfill only works if your asset code reads a partition range (partition_key_range or partition_time_window) instead of a single partition_key; otherwise you get one run per partition.

Retries only help if a step is safe to repeat: a retry after a half-finished INSERT doubles rows in every tool. Make each step atomic (load a staging table, then swap) or keyed (delete and insert the run’s key in one transaction).

Local development and testing

Task

Apache Airflow 3

Dagster

Prefect 3

Run one pipeline locally

airflow dags test <dag> <date> or dag.test() in Python

dagster asset materialize (now nudges towards dg launch --assets) or materialize() in tests

Call the flow function: python flow.py

Full local stack

airflow standalone (API server, scheduler, DAG processor, triggerer, SQLite)

dagster dev (webserver and daemon)

prefect server start, or nothing at all (temporary server)

Unit tests

Test plain functions; dag.test() for integration; check imports with airflow dags reserialize

Assets are plain functions; materialize() with test resources

Flows and tasks callable directly; prefect_test_harness

Upgrade linting

ruff check --select AIR301 (and AIR302, AIR311, AIR312) for Airflow 3 changes

Changelog breaking changes; deprecation warnings

Release notes

My impression from the drill: Prefect has the least friction for a team with existing Python scripts, because the script stays a script. Dagster needs more rethinking up front but gives the clearest local picture of dependencies. Airflow’s dags test is good for one DAG, but a realistic local stack is four processes plus an auth manager, and airflow standalone needs the virtualenv’s bin directory on PATH because it starts its components as subprocesses.

Running in production: Kubernetes, databases and observability

Area

Apache Airflow 3

Dagster OSS

Prefect server (OSS)

Official Helm chart

apache-airflow/airflow chart 1.22.0; Local, Celery and Kubernetes executors, plus AWS and Edge executors through providers; KEDA autoscaling for Celery; optional bundled PostgreSQL and PgBouncer

dagster/dagster chart with webserver, daemon, user code deployments and an optional PostgreSQL subchart

prefect/prefect-server and prefect/prefect-worker charts from prefect-helm

Metadata database

PostgreSQL 13 to 17 or MySQL 8.0, 8.4 and Innovation; SQLite 3.15+ for development only; MariaDB not supported

SQLite default; PostgreSQL or MySQL storage for production

SQLite default; PostgreSQL (14.9+ for multi-server), plus Redis for multi-server

Processes to keep alive

API server, scheduler, DAG processor, triggerer, workers (per executor)

Webserver, one daemon, code location servers, run workers

API server, background services, workers per work pool

Metrics and traces

StatsD or OpenTelemetry metrics (timers as histograms since 3.3.0); OpenTelemetry traces

Run and asset events in the event log; run monitoring

Events, automations and logs in the server and UI

Logs

Remote task logging to object storage or Elasticsearch, via providers

Compute logs stored locally or in S3, GCS or Azure

Flow and task logs sent to the API

Plan the metadata database like any production database. Airflow’s upgrade guide asks you to back it up and trim it with airflow db clean before migrating, because schema changes on a large database take time. XComs, event logs and run history grow without limit in all three tools unless you set retention.

The data itself should not live in the orchestrator. Airflow XComs and Prefect results are for small values; Dagster I/O managers write real data, but to your storage. If your nightly jobs write Parquet to object storage, the table format decision is separate, and I covered it in the lakehouse table formats post.

Access control and multi-tenancy

Airflow 3 ships the Simple auth manager as default, meant for development; it logs a warning when the deployment looks like production. For real use you install the FAB auth manager (the Flask AppBuilder provider, with LDAP or OAuth) or the Keycloak auth manager. Per-DAG permissions exist. The security model is explicit that DAG authors are trusted: their code runs on workers and in the DAG processor and triggerer, which can still reach the metadata database. Multi-team mode, added in 3.2, lets several teams share one deployment with team-scoped variables, connections, pools and executors (DAGs belong to a team through their DAG bundle), but the docs mark it experimental and “different from multi-tenancy”: all teams share the scheduler and the metadata database.

Dagster OSS has no built-in login or role model. Its Dagster+ page lists authentication, role-based access control and audit logs among the things Dagster+ adds. Self-hosted teams usually put the webserver behind an authenticating proxy and give each team its own code location for dependency isolation, which is not the same as permission isolation.

Prefect server can be protected with basic authentication: one admin:password string set on the server (PREFECT_SERVER_API_AUTH_STRING) and on every client. There are no users, roles or workspaces in the open-source server. The Prefect docs say RBAC, audit logs and SSO are Prefect Cloud features.

In short: if several teams with different trust levels must share one orchestrator, Airflow is the only one of the three with an open-source permission model, and even there the DAG author is a trusted role. For Dagster and Prefect, either buy the hosted product, or run separate instances per trust boundary behind your own SSO proxy.

Licences, history and governance

Apache Airflow

Dagster

Prefect

Licence (LICENSE file)

Apache 2.0

Governance

Apache Software Foundation project (TLP since 8 January 2019); PMC and committers from many companies

Company-led (Dagster Labs)

Company-led (Prefect Technologies)

Commercial offering by the maintainers

None from the ASF; several vendors and clouds sell managed Airflow

Dagster+ (Serverless and Hybrid)

Prefect Cloud

Features held back from open source

None by the project itself

Auth, RBAC, audit logs, Insights, branch deployments, catalogue features (Dagster+)

RBAC, SSO, audit logs, multiple workspaces (Prefect Cloud)

All three are permissively licensed today; the difference is who decides the roadmap. Airflow changes through Apache votes and AIPs (Airflow Improvement Proposals), slower but hard for one company to redirect. Dagster and Prefect ship weekly or near-weekly and keep governance features for their paid services. That is a fair model; just know which side of the line your needs fall on.

Security and CVEs

Scores below are what the CVE records carry. For Apache Airflow, the Apache CNA writes a textual severity and the CVSS 3.1 score comes from CISA’s ADP container; for Dagster and Prefect the scores come from the CNA (GitHub, huntr or VulDB).

Apache Airflow

GitHub’s advisory database lists 60 advisories for apache-airflow or apache-airflow-core published between 1 January and 13 October 2026 (3 critical, 19 high, 35 medium, 3 low). Many are authorisation and secret-masking fixes in the new API server, which is what you would expect after a large rewrite. Selected ones:

CVE

Score

Issue

Fixed in

CVE-2026-33264

9.8 (ADP)

Unrestricted class import while deserialising a serialised DAG; a DAG author could get code execution in the scheduler or API server

3.3.0

CVE-2026-67587

8.8 (ADP)

Task SDK callback deserialisation could import arbitrary modules in the scheduler (3.3.0 only)

3.3.1

CVE-2026-68968

7.5 (ADP)

Backfill API parser mismatch allowed cross-DAG authorisation bypass

3.3.1

CVE-2026-68971

6.5 (ADP)

Asset materialise and XCom wait endpoints ignored the team in multi-team mode

3.3.1

CVE-2026-49298

8.8 (ADP)

KubernetesExecutor passed the Execution API JWT as pod command-line arguments

3.2.2

CVE-2026-42252

9.1 (ADP)

Official docs showed dag_run.conf templated straight into bash_command; copied code allowed command injection by users who can trigger

3.2.2 (docs and guidance)

CVE-2025-57735

9.1 (ADP)

JWT not invalidated at logout

3.2.0

CVE-2024-56373

8.4 (ADP)

Airflow 2: DAG author could get code execution in the webserver through log template history

2.11.1

The lesson from Airflow’s list: the 3.x boundary is “DAG author code must not run in the scheduler or API server”, and several 2026 fixes closed gaps in exactly that boundary. Treat DAG authoring rights as code deployment rights. If you are still on 2.x, 2.11.1 (February 2026) carried several of these fixes, but 2.x is past end of life now and later issues may not be fixed there.

Dagster

CVE

Score

Issue

Fixed in

CVE-2026-41490

8.3 (CNA)

DuckDB, Snowflake, BigQuery and DeltaLake I/O managers interpolated dynamic partition keys into SQL; a user able to add dynamic partitions could inject SQL

Core 1.13.1, libraries 0.29.1

CVE-2025-51481

6.6 (ADP)

Local file inclusion through the gRPC notebook data call (path traversal)

1.10.16

Dagster’s own gRPC code servers should never be reachable from outside the cluster, and the SQL injection is a reminder that partition keys are user input when partitions are dynamic.

Prefect

CVE

Score

Issue

Fixed in

CVE-2026-3514

7.5 (CNA)

Authentication middleware exempted any path ending in health or ready

3.6.22.dev7 per GHSA (3.6.22 and later)

CVE-2026-7723

7.3 (CNA, CVSS 3.1)

Unauthenticated event injection through the /api/events/in WebSocket

3.6.14

CVE-2026-3515

8.5 (CNA)

Argument injection into git clone through the reference field of the prefect-github GitHubRepository block

No patched version listed in the GitHub advisory on 13 October 2026; check prefect-github releases

CVE-2024-8183

7.6 (CNA)

CORS misconfiguration on the server

3.0.3 and 2.20.17

Two of these only matter if you enabled basic authentication, which tells you something: without it, a self-hosted Prefect server has no authentication at all. Never expose one on a public address.

Practical controls

Run the latest patch: Airflow 3.3.2, Dagster 1.13.26, Prefect 3.8.8 on 13 October 2026. Subscribe to each project’s security advisories.
Treat “can write DAGs, assets or flows” as “can run code on your workers and read their secrets”. Gate it with code review and CI.
Keep the Airflow API server, Dagster webserver and Prefect server off the public internet; put SSO in front.
Do not template user-supplied run parameters into shell commands. Pass them as environment variables or arguments and validate them.
On Kubernetes, give worker pods a least-privilege service account and check what pod specs reveal.
Back up the metadata database and test restoring it before every major upgrade.

Migrating from Airflow 2 to Airflow 3

If you stay with Airflow, plan this as a project, not an upgrade command. From the official upgrade guide:

Be on Airflow 2.7 or later (latest 2.x recommended) and a supported Python.
Back up the metadata database with Airflow stopped, run airflow db clean, and make sure airflow dags reserialize runs without errors.
Run ruff check dags/ --select AIR301 (and AIR302, AIR311, AIR312) and fix imports: airflow.decorators to airflow.sdk, airflow.datasets.Dataset to airflow.sdk.Asset, airflow.models.Variable to airflow.sdk.Variable and so on.
Install apache-airflow-providers-standard; BashOperator, PythonOperator, FileSensor and ExternalTaskSensor moved there.
Remove direct metadata database access from task code. Use the Airflow Python client against the REST API instead.
Run airflow config update --fix; in the Helm chart, webserver values become apiServer, and the DAG processor runs on its own.
Decide create_cron_data_intervals and catchup before the first Airflow 3 run, and review every task that uses ds, data_interval_start or removed context variables.
Replace SubDAGs with task groups or assets, SLAs with deadline alerts, and /api/v1 clients with /api/v2; install the FAB provider if you need FAB login, and move OAuth redirect URLs under /auth.
Check xcom_pull() calls without task_ids: in Airflow 3 they read only from the current task.

Run old and new side by side for at least one month-end before you switch off Airflow 2.

Decision guide for the Coimbatore scenario

If your situation is

Lean towards

Why

Existing Airflow 2 estate, many operators and providers in use, several teams

Airflow 3

Migration is real work but cheaper than a rewrite; the largest provider ecosystem; open-source permission model

Mostly tables and models (dbt, warehouse loads), want lineage and freshness as first-class

Dagster

Assets and partitions match the problem; strong local feedback

Mostly Python scripts and API calls, small team, want the least ceremony

Prefect

Scripts become flows with decorators; easy to run anywhere

Several teams with different trust levels on one self-hosted instance

Airflow 3, or separate Dagster or Prefect instances per team

Dagster OSS and Prefect server have no user or role model

Per-order or per-payment workflows that must survive crashes

None of these; see durable execution

Batch orchestrators are not built for millions of small long-lived workflows

If a “pipeline” is really a business transaction (one loan disbursal, one refund), look at the engines in the durable execution post. The tools here are for the nightly and hourly data work around those transactions.

For the invented Coimbatore team, I would do this:

Write down the run key for every job (business date for nightly loads, month for RBI reports) and make each load idempotent for it, while still on cron.
Shortlist Airflow 3 (existing knowledge and DAGs) and Dagster (table-shaped work). Keep Prefect in mind if the team shrinks to two people.
Build one pipeline (collections file to branch summary) in both, with a late-file test, a failed-step retry, a manual rerun and a seven-day backfill.
Choose on how the month-end rerun feels. With Airflow, set create_cron_data_intervals and time zones deliberately and key loads on the logical date or an explicit parameter, never on ds alone.

Checklist before you schedule another DAG

Every pipeline has a written run key (date, partition or file), and reruns for that key replace data instead of adding to it.
Time zone of schedules and of the run key is stated in code, and someone has checked what midnight IST means in UTC.
Retries are set per step, with delays, and every retried step is idempotent.
A backfill of seven days has been run in staging and checked against the source.
Late and missing source files have a defined behaviour (sensor, skip, fail with alert), not “process yesterday’s file”.
The metadata database is PostgreSQL (or MySQL for Airflow), backed up, with retention for old runs, logs and XComs.
The UI and API are behind SSO or at least an authenticating proxy, and not on a public address.
Secrets come from a secrets backend; nothing sensitive is in DAG files, Dagster config or plain Prefect blocks.
You run the latest patch release and someone owns reading the security advisories.
Alerts go to people on call for failures, not to a shared inbox nobody reads.

Common mistakes

Migrating cron jobs one to one without deciding what each run owns.
Using ds or “today” in a task instead of the run’s logical date or partition key.
Upgrading to Airflow 3 without deciding create_cron_data_intervals.
Assuming manual runs in Airflow 3 have a logical date.
Expecting Airflow 3 to catch up missed intervals when catchup is now off by default, or turning it on with an old start_date and flooding the workers.
Passing large data frames through XCom or Prefect results instead of storage.
Forgetting the Dagster daemon, or running two, and wondering why schedules do nothing.
Exposing a Prefect server or Dagster webserver publicly because “it is only the UI”.
Letting the metadata database grow for years, then attempting a major upgrade on it.
Putting business logic inside orchestrator code so it cannot run or be tested outside it.

A drill I ran on 13 October 2026

Environment: shared Linux VM (Debian 13, kernel 6.12), 8 vCPU, about 15.6 GiB RAM with about 4 GiB available, local overlay filesystem, no root, no Docker. Between about 2:20 and 2:40 PM IST I installed the three tools from PyPI with uv into separate Python 3.13 virtualenvs and ran the same pipeline in each. Other work shares this VM, so treat every number as indicative only.

What ran: Airflow 3.3.2 with the official Python 3.13 constraints file (128 packages), first airflow dags test, then airflow standalone on port 18093 (my first standalone attempt failed because the venv bin directory was not on PATH); Dagster 1.13.26 with dagster-webserver (68 packages) via dagster asset materialize, no webserver or daemon; Prefect 3.8.8 (104 packages) via python prefect_flow.py with no server configured.

The pipeline was identical logic in all three: extract writes 5,000 synthetic order rows to CSV, transform sums amounts for eight cities and raises an error on its first attempt, load deletes and inserts the run date’s rows in a local SQLite table. Each step had one retry with a two-second delay, set in the tool’s own way.

One run per tool, local SQLite everywhere (smoke test, not a benchmark)

Dagster 1.13.26  asset materialize, multiprocess executor (one subprocess per step)
  exit 0, wall 10.61 s incl. CLI start-up
  city_totals: STEP_UP_FOR_RETRY -> STEP_RESTARTED -> STEP_SUCCESS; 8 rows loaded
  CLI notes: SupersessionWarning (use 'dg launch --assets'); telemetry notice

Prefect 3.8.8    python prefect_flow.py, temporary local server
  server start about 7.9 s before the flow began; exit 0, wall 14.98 s
  transform: "Retry 1/1 will start 2 second(s) from now" -> Completed; 8 rows loaded

Airflow 3.3.2    airflow dags test orders_daily 2026-10-13
  db migrate 3.29 s; dags test exit 0, wall 9.58 s
  transform: try 1 failed -> up_for_retry -> try 2 success; 8 rows loaded

Airflow 3.3.2    airflow standalone (LocalExecutor, SQLite)
  API health 200 after 7.1 s; unpause created ONE scheduled run (catchup=False)
    run for 13 Oct 00:00 IST: logical_date 2026-10-12T18:30Z, ds = 2026-10-12
    injected failure hit this run: try 1 failed, try 2 success
  manual trigger to success: 7.8 s
  idle RSS: api-server ~274 MiB, dag-processor ~166, scheduler ~157, triggerer ~151 MiB

Rows in Airflow's SQLite after all runs:
  run_date 2026-10-13: 8   run_date 2026-10-12: 8   run_date NULL: 16  <- two manual runs, ds = None

What this shows:

All three retried the injected failure exactly once and produced the same eight totals.
Prefect needed only decorators and no server setup; the cost was a few seconds of temporary server start.
Dagster turned the three functions into a lineage of three assets and recorded each retry as an event in its event log. Its CLI also printed a telemetry notice: Dagster OSS collects usage statistics by default, and the docs show how to switch it off in dagster.yaml.
Airflow 3 behaved as documented: catchup off created only the latest interval, and manual runs had no logical date. My load used ds as its key, so each manual run wrote eight rows with a NULL date that the next run’s delete could not match. That is my pipeline’s bug, and it is exactly the kind an Airflow 2 to 3 migration exposes.

Limits of the drill: one VM, one run per tool, SQLite only, no PostgreSQL, no Celery or Kubernetes executors, no Dagster webserver or daemon (so no schedules or sensors), no Prefect worker, work pool or deployment, no Helm, no load. Install times were not recorded reliably, so I do not quote them. Timings include Python start-up. Artefacts (README, scripts, logs, results) are under /workspace/revisit/drill34/. All processes were stopped; virtualenvs and caches were deleted, and the generated Airflow admin password was removed.

Unlearn / re-learn

Unlearn

A cron line is a pipeline.
Retries make a job reliable.
ds is “the date this run is about”, and it is in your time zone.
Airflow 3 is Airflow 2 with a new UI.
Open source means every feature you see in a demo is in the open-source edition.

Re-learn

Give every run a key and make every load idempotent for it.
Choose a mindset (tasks, assets or plain flows) that matches how your team will think in a year.
Read the scheduling semantics of your version, especially defaults that changed.
Treat DAG or flow authoring as code deployment, with review and least privilege.
Practise a manual rerun, a retry and a seven-day backfill before you call a pipeline done.

Closing

Data orchestration in 2026 comes down to three mature, Apache 2.0 licensed tools with different centres of gravity. Learn their models: Airflow’s DAG runs, data intervals and new API-server architecture, Dagster’s assets, partitions and code locations, Prefect’s flows, deployments and work pools. Unlearn the habit of treating a scheduled script as a pipeline, and of trusting ds and old defaults after an upgrade. Re-learn the current versions: Airflow 3.3.2 with Airflow 2 past end of life, Dagster 1.13.26 with its access control in Dagster+, Prefect 3.8.8 with basic authentication only in the open-source server. Practise with a small drill like the one above: one pipeline, one forced failure, one manual rerun, one backfill. Apply one rule before you schedule another DAG: every run owns a key, and running it twice gives the same result.

Sources

Apache Airflow

Dagster

Prefect

Security (CVE Program records and GitHub advisories)

Internal Revisit.Tech links used in the body

comments powered by Disqus

Releted Posts

Feature flags in 2026: revisit Unleash, Flagsmith and GrowthBook before you hard-code another release toggle

Manoj Shukla
October 12, 2026

Picture a test-preparation and online classes platform in Jaipur (an invented example, not a real client). Students use an Android app, an iOS app and a React web app.

Manoj Shukla
October 12, 2026

Picture a payments and lending analytics platform in Hyderabad (an invented example, not a real client). Card authorisations and EMI schedules still land in PostgreSQL 16.

Manoj Shukla
October 12, 2026

Picture an ecommerce and last-mile logistics platform in Bengaluru (an invented example, not a real client). Order capture and inventory still run on PostgreSQL 16.