Data ingestion in 2026: revisit Airbyte, Meltano and dlt before you write another extract script

Manoj Shukla
October 30, 2026

Almost every data team has a folder of extract scripts. One pulls orders from the shop platform every night, one reads the payment gateway’s settlement API, one copies a few tables from the ERP database, and one downloads a CSV that somebody uploads to a shared drive. Each script was written in an afternoon. Each one works, until the API changes its pagination, a column is renamed, a token expires, or the person who wrote it moves to another team. Then the dashboard is quietly wrong for a week.

Connector-based tools promise to replace that folder. You pick a source, pick a destination, set a schedule, and the tool handles pagination, retries, incremental state and schema changes. This post compares the three tools Indian teams most often look at for self-hosted extract and load in 2026: Airbyte, Meltano and dlt. We also mention Sling, CloudQuery and Estuary Flow briefly. We look at versions, licences (Airbyte’s has changed in a way many people have not noticed), architecture, how connectors are built and maintained, incremental state, schema evolution, secrets, upgrades and footprint, and when a plain script is still the honest answer.

This post sits between several earlier ones, and it tries not to repeat them. Change data capture in 2026 covers streaming row changes from your own databases with Debezium, Flink CDC and PeerDB; here we deal mostly with batch pulls from SaaS APIs and databases. Data pipeline orchestration in 2026 is about scheduling and dependencies; the tools here are what those schedulers often run. Lakehouse table formats and real-time OLAP are about where the data lands. In-process analytics in 2026 explains why DuckDB is often a fine first destination. Data catalogs in 2026 covers what happens after the data arrives. Workflow automation in 2026 is the tool people reach for when they want a webhook to trigger a business process, which is a different job from filling a warehouse. And since ingestion tools hold dozens of API keys and database passwords, Kubernetes secrets delivery and Vault and OpenBao matter here too.

We end with a small drill: dlt loading a synthetic local order table into DuckDB, testing incremental merges, a new column, a type change, a frozen schema contract and state recovery, and a short Meltano run with a Singer tap. We say clearly what we did not run.

A scenario to keep us honest

Imagine a knitwear and hosiery maker in Ludhiana. The company is invented for this post. It makes sweaters, thermals and socks in two factories, sells to about 300 distributors across Punjab, Haryana, Delhi, Rajasthan and Uttar Pradesh, and has grown a direct-to-consumer business through its own online store and two marketplaces. The winter season, from September to January, brings most of the year’s revenue. The engineering team has seven people, and one of them, the “data person”, also looks after the reporting database.

The setup today:

An order and dispatch system on PostgreSQL 16, built in-house, in a cloud region in Mumbai.
An online store on a hosted e-commerce platform, a payment gateway, a courier aggregator for shipping labels and tracking, an accounting SaaS for invoices and GST, and an ad account for online campaigns.
A reporting database: a second PostgreSQL instance, filled every night by eleven Python scripts run by cron on one VM.
Dashboards in an open source BI tool on top of the reporting database.

Last winter three things went wrong. The courier aggregator changed its API pagination, and the tracking script silently fetched only the first page for nine days, so the “delivered but not paid” report under-counted cash-on-delivery money stuck with couriers. A developer added a discount_type column in the order system, and the nightly copy script failed on the insert, but cron only mailed a root inbox nobody reads. And when the data person was on leave for a family wedding, the ad account token expired and nobody knew which script used it or where the token was stored.

Now the finance head wants a daily view of sell-through by distributor, cash pending with couriers, and ad spend against online orders, ready before the 10 AM sales call. The owner’s son, back from a stint at a startup, says “use Airbyte, it has every connector”. The data person wants to stay in Python and has read about dlt. A consultant suggests Meltano because “it is all config in Git”.

We will come back to Ludhiana in the worked plan.

The short version

Airbyte is a full platform with a UI, an API, a scheduler, a connector catalogue and a Kubernetes-based runtime. The latest self-managed version is 2.4, released on 9 October 2026 according to its release notes. The platform has been under the Elastic License 2.0 (ELv2) since 2021, and since July 2025 Airbyte’s own connectors are under ELv2 as well, not MIT. ELv2 is source-available, not OSI open source. For internal use it costs nothing; it does not allow you to offer Airbyte to others as a managed service.
Meltano is a command-line tool, MIT-licensed, that runs Singer taps (extractors) and targets (loaders) from a meltano.yml project kept in Git. The latest release is 4.4.1 (7 October 2026). The project has changed hands: by March 2026, Meltano’s own blog said Matatika had acquired the Meltano open source project, and the company now runs as Meltano with a paid Meltano Cloud beside the MIT core.
dlt (data load tool) is an Apache-2.0 Python library, not a service. You write sources as Python generators or declarative REST API configs, and dlt handles schema inference, normalisation, incremental cursors, state and loading into a destination. The latest release is 1.32.0, published on 30 October 2026 at 5:05 AM IST.
The real cost of ingestion is not the first load. It is incremental state, schema changes and connector maintenance over years. Choose the tool whose failure modes your team can see and fix.
For a small team with Python skills and a dozen sources, dlt plus the scheduler you already have is often enough. Airbyte earns its weight when many people need a UI and the connector catalogue covers most of your sources well. Meltano suits teams who want config in Git and are comfortable owning Singer taps.

Where things stand on 30 October 2026

Airbyte. The self-managed release notes list 2.4 (9 October 2026), 2.3 (15 September), 2.2 (10 August) and 2.1 (3 April 2026), after 2.0 on 14 October 2025. Note that GitHub releases stop at v2.0.0; for 2.1 onwards, read the release notes on the documentation site, not the GitHub releases page. Several of these releases carry action items. Version 2.1 dropped Helm chart V1 entirely and turned authentication on by default for new installs. Version 2.2 is a one-way upgrade: its database migrations drop tables that 2.1 needs, so the notes say to take a backup first, and that rolling back is not supported. The same release stopped accepting HTTP Basic credentials on the API for Community deployments, which breaks scripts that call the API with curl -u. Version 2.3 moved the bundled MinIO image to airbyte/minio because the upstream image was no longer publicly pullable, with a manual step for existing installs. Version 2.4 fixed CVE-2026-80049, a cross-workspace authorisation flaw. The 2.1, 2.2 and 2.3 release notes also tell Self-Managed Enterprise customers not to upgrade and to talk to Airbyte about moving to its Cloud plans, which is a clear signal about where the company puts its paid self-hosted effort.

Meltano. Version 4.0.0 came in October 2025 with breaking changes in logging and the meltano config arguments. Then 4.1.0 (January 2026), 4.2.0 (April), 4.3.0 (21 September), 4.4.0 (29 September) and 4.4.1 (7 October 2026). A 3.9 maintenance line also got fixes until July 2026. The 4.3 and 4.4 releases added meltano cloud auth login, Hub requests authenticated with a Meltano Cloud account, meltano state export and import, and a fix so that an empty state payload from a crashed target no longer wipes the bookmark and forces a full refresh. On the company side, a post on Meltano’s blog dated 1 March 2026 says Arch, the company that had been behind Meltano, has shut down, and that Matatika has acquired the Meltano open source project. A community announcement on 12 March 2026 says Matatika is moving everything to the Meltano name. The docs now describe “Meltano Open” (MIT) and “Meltano Cloud” (managed).

dlt. Releases are frequent: 1.30.0 (11 August 2026), 1.31.0 (7 October) and 1.32.0 (30 October 2026). Both recent minors list breaking changes at the top of their notes, mostly around timestamps and time zones: 1.31.0 requires pendulum 3, changes how datetime cursors are typed and makes Incremental.last_value always the current cursor; 1.32.0 stops treating naive SCD2 timestamps as UTC and rewrote the Jira and Salesforce verified sources. dlt supports Python 3.10 to 3.14.

The others. Sling 1.6.5 came on 9 October 2026. CloudQuery’s CLI reached 6.49.2 on 22 October. Estuary Flow published 0.6.15 on 30 October 2026.

The first question: what kind of extract is this?

Before comparing tools, list your sources and put each in one of four groups. The groups need different things from a tool.

Group 1: SaaS APIs with a well-known schema

The store platform, the payment gateway, the ad account, the CRM. Here the main value of a connector tool is that someone else has already written pagination, rate-limit handling, authentication and the stream definitions. The question is: does a well-maintained connector exist for this exact source, and who maintains it?

Group 2: your own databases

The order system on PostgreSQL. Here you control the schema. You need incremental reads by an updated_at column, or CDC from the write-ahead log if deletes matter or the tables are large. If you need low-latency streams of every change, that is CDC territory and a different post. For a nightly or hourly copy, a cursor column is usually fine.

Group 3: niche or regional APIs

The courier aggregator, the accounting SaaS used mostly in India, a distributor portal. Often no connector exists, or one exists in a community catalogue with three stars and a last commit two years ago. Here the question is how easy it is to write and own your own connector.

Group 4: files

CSV exports, Excel uploads, files on SFTP or in object storage. Every tool handles these. The hard part is usually the people sending the files, not the tool.

A quick test

Count the sources in each group. If most are in Group 1 and good certified connectors exist for them, a platform with a large catalogue saves real work. If most are in Groups 2, 3 and 4, you will be writing or adapting connectors anyway, and the tool that makes writing connectors easy matters more than the size of its catalogue.

What each tool is

Airbyte

Airbyte is a data movement platform started in 2020. It has a web UI, a public API, a Terraform provider, a scheduler, a connector registry and a Connector Builder for writing declarative API connectors without code. Connectors are Docker images that speak the Airbyte Protocol: a source emits records and state messages, a destination writes them. The platform runs on Kubernetes. For a single machine, the abctl tool creates a local Kubernetes cluster inside Docker and installs Airbyte into it; for production, the Helm chart V2 deploys it into a cluster you run.

The quickstart says to run Airbyte on at least 4 CPUs and 8 GB of memory, or 2 CPUs and 8 GB with a low-resource mode that disables the Connector Builder, and that installation can take up to 30 minutes.

# single machine trial (sketch; not run in our drill, which had no Docker)
curl -LsfS https://get.airbyte.com | bash -
abctl local install --low-resource-mode
abctl local credentials        # prints the generated login
# UI on http://localhost:8000

A sync in Airbyte is a “connection” between a source and a destination, with a sync mode per stream:

Incremental Append reads new records since the last cursor value and appends them.
Incremental Append + Deduped appends and then builds a final table deduplicated by primary key.
Full Refresh Overwrite, Full Refresh Append and Full Refresh Overwrite + Deduped read everything each time.

Incremental reads use either a cursor field or, for some database sources, CDC. Airbyte keeps the state for each stream in its own database, not in the destination.

Connectors come in support levels. The documentation lists Airbyte-maintained connectors (labelled certified in the connector metadata), Enterprise connectors that need an extra licence and are not available to open source users, Marketplace connectors maintained by the community with no SLA, and custom connectors you build yourself. It also says Airbyte removes low-use, unmaintained connectors from time to time. Airbyte’s marketing says it moves data from “700+ sources”; that is the vendor’s count across all levels.

PyAirbyte, a separate Python library (0.71.3 on 26 October 2026), lets you run some Airbyte source connectors from Python without the platform. It is worth knowing about, but for this post we focus on the platform.

Meltano

Meltano is a command-line tool for running Singer pipelines. Singer is an open specification for data extraction, published by Stitch; its specification repository dates from October 2016. A Singer tap is a program that writes JSON lines to standard output: SCHEMA messages describing a stream, RECORD messages with rows, and STATE messages with bookmarks. A Singer target reads those lines from standard input and writes them somewhere. Taps and targets are ordinary Python packages, each in its own virtual environment, joined by a pipe.

Meltano adds a project structure around this: a meltano.yml file lists plugins (extractors, loaders, mappers, dbt and utilities), their settings and their environments; meltano install creates the virtual environments; meltano run tap-x target-y runs a pipeline and stores state; the Meltano Hub is a catalogue of plugin definitions with many variants per source. The company says the Hub lists “600+” connectors; that is the vendor’s count, and quality varies widely. Many taps are built with the Meltano Singer SDK (0.54.9, Apache-2.0), which handles pagination, typing and state conventions.

# meltano.yml (sketch for 4.4; partly based on the drill project)
version: 1
default_environment: prod
send_anonymous_usage_stats: false
plugins:
  extractors:
  - name: tap-postgres
    variant: meltanolabs
    pip_url: meltanolabs-tap-postgres==<pin a version>
    config:
      host: 10.20.0.15
      user: ingest_ro
      database: orders
    select:
    - public-orders.*
    - public-dispatches.*
    metadata:
      public-orders:
        replication-method: INCREMENTAL
        replication-key: updated_at
  loaders:
  - name: target-postgres
    variant: meltanolabs
    pip_url: meltanolabs-target-postgres==<pin a version>
state_backend:
  uri: s3://ludhiana-ingest-state/meltano

Passwords do not go in this file. Meltano reads them from environment variables or a .env file that is not committed. Running it is one command:

meltano --environment=prod run tap-postgres target-postgres

State is stored per pipeline under a state ID. The default backend is Meltano’s system database (SQLite in .meltano/meltano.db unless you configure another database). The docs also describe local file, S3, Google Cloud Storage and Azure backends, with locking. Two details from the docs deserve attention: bookmarks in state take precedence over start_date, and some targets emit state only when they finish, so a failed run may restart from further back than you expect.

dlt

dlt is a Python library from dltHub, started in 2022, Apache-2.0, with 1.0.0 released in September 2024. There is no server. You pip install dlt, write a source, and call pipeline.run() from a script, a notebook, a cron job, a serverless function or an orchestrator task. dltHub also sells a commercial platform, but the library stands alone.

A dlt resource is a Python generator that yields dictionaries, data frames or Arrow tables. dlt infers a schema, flattens nested JSON into child tables, normalises names and types, and loads into a destination: DuckDB, PostgreSQL, Snowflake, BigQuery, ClickHouse, object storage files, Iceberg or Delta tables and others through optional extras. It ships ready-made sources for SQL databases (with SQLAlchemy, PyArrow, pandas or ConnectorX backends), REST APIs (declarative, with paginators and incremental parameters) and filesystems, plus a set of verified sources for popular SaaS apps.

Here is the shape of the resource we used in the drill, reading an order table incrementally and merging by primary key:

import dlt, sqlite3

@dlt.resource(name="orders", primary_key="id", write_disposition="merge")
def orders(updated_at=dlt.sources.incremental(
        "updated_at", initial_value="2026-01-01T00:00:00+00:00")):
    con = sqlite3.connect("src_orders.sqlite")
    con.row_factory = sqlite3.Row
    q = "select * from orders where updated_at >= ? order by updated_at"
    for row in con.execute(q, (updated_at.last_value,)):
        yield dict(row)

pipeline = dlt.pipeline(pipeline_name="ludhiana_orders",
                        destination="duckdb", dataset_name="raw_erp")
print(pipeline.run(orders()))

And a sketch of a declarative REST API source for a courier tracking API, so pagination and the incremental parameter are configuration rather than code:

# sketch; field names follow the dlt rest_api docs, the API is invented
from dlt.sources.rest_api import rest_api_source

courier = rest_api_source({
    "client": {
        "base_url": "https://api.courier.example/v2/",
        "auth": {"type": "bearer", "token": dlt.secrets["courier_token"]},
        "paginator": {"type": "json_link", "next_url_path": "paging.next"},
    },
    "resource_defaults": {"primary_key": "awb", "write_disposition": "merge"},
    "resources": [{
        "name": "shipments",
        "endpoint": {
            "path": "shipments",
            "params": {"updated_since": {"type": "incremental",
                                         "cursor_path": "updated_at",
                                         "initial_value": "2026-09-01T00:00:00Z"}},
        },
    }],
})

dlt keeps pipeline state (including incremental cursors) in a local working directory, and also writes it into a _dlt_pipeline_state table in the destination. If the working directory is lost, as it is on a fresh CI runner or container, dlt restores state from the destination on the next run. Credentials come from environment variables, secrets.toml and config.toml, or vault providers.

Sling, CloudQuery and Estuary Flow, briefly

Sling is a Go single binary under GPL-3.0 that copies data between databases, files and object storage using YAML or JSON replication configs, with wildcards for many tables. It is a good fit for database-to-warehouse and file copies, and less for SaaS APIs. The README calls it a free CLI for small to medium volume pipelines.

CloudQuery describes itself today as a cloud asset inventory for platform teams: it syncs cloud infrastructure and some SaaS metadata into a warehouse. The README says the framework, SDK, CLI and “some integrations” are open source under MPL-2.0, and that some code has moved to closed source. Check each plugin’s licence and terms before you plan on it.

Estuary Flow is a managed streaming and batch data movement service. Its repository carries Apache-2.0 and Business Source License files, and the README points to the BSL for use outside the managed offering. It is closer to the CDC post than to this one.

The Singer ecosystem itself is wider than Meltano: many taps and targets live under the singer-io, MeltanoLabs and individual GitHub accounts, and any Singer runner can use them.

Side by side

Licence and governance

Airbyte: platform under ELv2 since 27 September 2021 (version 0.30.0). On 23 July 2025, a commit titled “migrate licenses to elv2” moved the remaining MIT code, including Airbyte’s connectors, to ELv2; connector metadata files now say license: ELv2 for connectors such as the PostgreSQL, HubSpot and Shopify sources. The Airbyte Protocol stays MIT, and the Python CDK repository and PyPI package still show MIT. In a blog post, Airbyte’s CEO explains the change as protection against competitors reselling the connectors as a managed service, and says past contributions keep their original licence. ELv2 allows internal production use, modification and redistribution, but not offering the software to third parties as a hosted or managed service, and not circumventing licence-key features. Contributions need a CLA. Governance is by the company.
Meltano: MIT. The company stewarding it is now Matatika, which acquired the project and operates under the Meltano name since March 2026. Plugin licences vary: each tap and target has its own licence, chosen by its author.
dlt: Apache-2.0, company governance by dltHub. Verified sources are in a separate repository; check their licence too.

If you only run ingestion for your own company, all three are free to use. If you are a consultancy or SaaS company that wants to offer “connect your apps” to your clients, read ELv2 carefully with your legal team before you build on Airbyte; Airbyte’s FAQ says building an analytics product that uses Airbyte behind the scenes is allowed, but exposing Airbyte’s UI or API to customers is not.

Many blog posts and comparison pages still say “Airbyte connectors are MIT”. That was true until mid-2025. Unlearn it.

Architecture

Airbyte: a set of services on Kubernetes (server, workers, workload API, a Temporal-based scheduler, a database, object storage for logs and state, Keycloak for auth in some setups), and each sync runs connector containers as pods. Powerful and isolated, and also many moving parts.
Meltano: a CLI process that launches a tap and a target as subprocesses joined by a pipe, with a system database for state and run history. Scheduling is external: cron, Airflow, Dagster or the Meltano Cloud.
dlt: a library in your process. Extract, normalise and load are stages in one Python program, with files in a working directory between stages. Scheduling and isolation are whatever runs the script.

Connectors and maintenance

This is where the long-term cost lives.

Airbyte: the largest catalogue, with clear support levels. Certified connectors are maintained by Airbyte and get frequent versions; connector versions can be pinned, and connector upgrades sometimes come with breaking-change migrations. Community connectors may lag. The Connector Builder and the low-code declarative YAML make simple REST connectors fast to build.
Meltano: the Singer catalogue is large but uneven. A popular source may have five variants, each a fork maintained by a different person. Picking the right variant, pinning it, and being ready to fork it yourself is part of the job. MeltanoLabs hosts community-maintained taps and targets with a clearer ownership model. Taps built on the Singer SDK are usually easier to fix.
dlt: a smaller set of verified sources, but writing your own is ordinary Python, and the REST API source covers many JSON APIs with configuration alone. Your connectors live in your repository and get your code review and tests. The flip side is that you maintain them. Note that dlt itself rewrote two verified sources in 1.32.0, so even “verified” sources can change shape.

The honest comparison: Airbyte’s catalogue reduces the work for popular sources, but for niche Indian SaaS tools all three tools leave you writing something. Ask “who fixes it when the API changes, and how fast?” for each of your top ten sources.

Incremental state

Airbyte: state per stream, stored in the platform database, visible and editable in the UI (the docs have a page on modifying connection state). Resetting or refreshing a stream is a UI action. Losing the Airbyte database means losing state, so it needs backups like any other PostgreSQL.
Meltano: Singer STATE messages, stored per state ID in a configurable backend. Taps decide what bookmark they keep, and some taps have no incremental support at all; in our drill, the CSV tap advertised only catalog and discover capabilities and re-read the whole file each run. The 4.4.0 fix for empty state payloads shows that state handling is still being hardened. meltano state get, set, export and import give you control.
dlt: cursors declared in code with dlt.sources.incremental, de-duplication of boundary rows, state saved locally and in the destination. Because state travels with the data, a lost VM does not mean a full reload, which we confirmed in the drill. Be careful with time zones in cursors after the 1.31.0 and 1.32.0 changes.

For all three, the same rule applies: a cursor on updated_at misses hard deletes and rows updated without touching that column. If deletes matter, use soft deletes, periodic full refresh of small tables, or CDC.

Schema evolution

Airbyte: per connection, choose “propagate field changes only”, “propagate all field and stream changes”, “approve all changes myself” or “stop future syncs”. Self-managed Airbyte checks the source schema before a sync at most once every 24 hours, so a change may take a day to be seen. Removed columns stay in the destination with nulls for new rows. A type change that does not fit produces row errors recorded in the airbyte_meta field. Removing a primary key or cursor field is a breaking change and pauses the connection.
Meltano: the tap emits a SCHEMA message, and what happens next depends on the target. Some targets add columns, some fail, some widen types. You can override key properties and replication keys in metadata, and select or exclude fields. Test schema changes per target.
dlt: the schema evolves by default: new columns are added, and values that cannot be coerced to an existing column type go to a “variant” column such as quantity__v_text. Schema contracts let you choose per table, column or data type: evolve, freeze (fail the load), discard_row or discard_value. In the drill, a frozen column contract stopped the load when a new column appeared, as designed.

The useful question is not “does it handle schema changes?” but “what do I want to happen, and will someone be told?” For finance tables, a frozen contract plus an alert is often better than silent evolution.

Secrets and security

Airbyte: connector specs mark fields as secrets; the platform stores them at secret coordinates in a configured secret persistence layer, does not return them from the API and does not log them. You can use an external secret manager. Security history needs attention: GHSA-g35w-6r6v-h4r7, published on 17 October 2026 and rated critical, describes remote code execution through Connector Builder custom components in versions before 2.2.0, even when unsafe code was disabled; any Workspace Reader could exploit it in 2.x, and on 1.x with the legacy Helm chart the Connector Builder did not inherit authentication at all. Airbyte says 1.x will not get a fix. CVE-2026-80049 (CVSS 3.1 score 8.8 at NVD) let a member of one workspace act on sources, destinations and connections in other workspaces; the 2.4 notes say it is fixed. Earlier, CVE-2024-38363 was a server-side template injection RCE. Keep Airbyte current, keep it off the public internet, and treat every workspace member as able to reach every credential the platform holds until proven otherwise.
Meltano: secrets in environment variables or .env, never in meltano.yml. The risk is supply chain: every tap and target is a pip package, often installed from a Git branch. In our drill, the Hub definition for the CSV tap used git+https://github.com/MeltanoLabs/tap-csv.git with no tag or commit. Pin versions or commits, and review what you install. The 4.4.1 fix that stopped sending the Meltano Cloud login token to an explicitly configured Hub is a reminder to read the release notes for credential handling.
dlt: secrets in environment variables, secrets.toml (kept out of Git) or vault providers. Your connectors are your code, so security is your code review, your dependency pinning and where you run the script. No GitHub security advisories were published for dlt or Meltano when we checked.

For all three: give every source a read-only account, and every destination a writer account that can touch only the raw schema.

Operations and upgrades

Airbyte: the heaviest. A Kubernetes deployment with its own database and object storage, Helm values that change between versions, one-way database migrations (2.2), image repository changes (2.3), and API authentication changes. Plan upgrades as small projects: read every release note between your version and the target, back up the database, test on a copy. Upgrading connectors is separate from upgrading the platform.
Meltano: upgrade the CLI with pip, then upgrade each plugin. 4.0 changed CLI arguments, and in our drill the old positional form meltano add extractor tap-csv no longer worked in 4.4.1. Run history and state live in the system database, so back it up or use a remote state backend.
dlt: upgrade a pip package, but read the breaking changes carefully; both 1.31.0 and 1.32.0 changed timestamp behaviour. Pin the version, run a test load to a scratch dataset, and compare row counts before upgrading production.

Footprint

Airbyte: the quickstart’s minimum is 2 CPUs and 8 GB of memory in low-resource mode, with 4 CPUs recommended, and in production a Kubernetes cluster with room for connector pods.
Meltano: one Python process plus one virtual environment per plugin; disk grows with plugins rather than memory.
dlt: one Python process. The drill loaded 1,000 rows in about a third of a second on this box, which says little about scale but shows how little there is to run.

We have no benchmark numbers we can stand behind. Airbyte’s 2.0 notes say faster sync speed makes syncs “4-6 times faster, on average”; that is the vendor’s own figure.

When a script is still the right answer

A script is not always technical debt. Keep a script when:

There is one source and one destination, and it is unlikely to grow.
The source is an odd one, like a government portal download or a partner’s SFTP drop, and no tool will have a connector.
The logic is part of the business process, such as reconciling a settlement file and marking payments, rather than copying data for analysis.

But if you keep scripts, steal the ideas from these tools: store the cursor somewhere durable, load into a raw schema with a load ID column, fail loudly on schema changes, and alert a channel people read. That is roughly what dlt packages as a library, which is why many teams move from scripts to dlt with little friction.

A worked plan for Ludhiana

Here is how the Ludhiana team could replace eleven cron scripts with something the finance head can rely on before the winter season.

List every source and owner. For each of the eleven scripts: source, tables or endpoints, volume, how often it must refresh, who needs it, where its credential lives. They find two scripts nobody uses and switch them off.
Group the sources. Group 1 (SaaS with a known schema): online store, payment gateway, ad account. Group 2 (own database): the order and dispatch system. Group 3 (niche): the courier aggregator and the accounting SaaS. Group 4 (files): distributor stock sheets uploaded weekly.
Check connectors honestly. For Group 1, they find Airbyte-maintained connectors for the store platform and the ad account, a community-level one for the payment gateway, and several Singer variants of uneven age. For Group 3, there is nothing well maintained in any catalogue.
Choose the backbone by team, not catalogue. Seven engineers, all comfortable with Python, no Kubernetes cluster, and one data person. Running Airbyte would mean adopting Kubernetes and a platform with one-way upgrades for three useful connectors. They choose dlt as the backbone, scheduled by the orchestrator they already use for other batch jobs, with each source as a small Python module in one repository.
Start with the database. The order system uses dlt’s SQL database source with an updated_at cursor and merge on primary key, reading from the replica with a read-only role. They add an updated_at trigger on two tables that did not reliably update it, and a weekly full refresh of the small distributors table to catch deletes.
Do the niche APIs declaratively. The courier and accounting APIs become rest_api_source configs with paginators and incremental parameters. The courier pagination bug from last winter is now a configuration line with a test: a fixture with three pages must load three pages.
Use existing connectors where they are clearly better. For the ad account, they try the dlt verified source and, as a comparison, PyAirbyte running Airbyte’s certified source from Python. They keep whichever is more stable after two weeks, and note that the Airbyte connector is under ELv2, which is fine for internal use.
Set schema contracts per table. Finance tables (payments, settlements, invoices) get columns: freeze so a new or changed column stops the load and alerts; marketing and tracking data is allowed to evolve. Variant columns are checked in a daily data test.
Put secrets and alerts in place. Every credential moves to the secrets manager and reaches the job as environment variables; each has an owner and an expiry date in a shared sheet. Failed loads and frozen-contract failures go to the team’s chat channel, not to a root mailbox.
Land raw, model later. dlt loads into a raw_* schema per source in the reporting PostgreSQL, with load IDs. SQL models build the sell-through and courier cash views on top. Nothing reads raw tables directly from dashboards.
Review after the season. If the number of sources grows past 30, or non-engineers need to add connections themselves, they revisit Airbyte with a real cluster and a budget for operating it, or Meltano if they prefer config files in Git over Python modules.

When each is better

Airbyte is better when many sources are popular SaaS apps with certified connectors, analysts or operations staff need a UI to add and monitor connections, you already run Kubernetes well, and you are using it for your own company. It is also good for building simple REST connectors without code.
Meltano is better when you want everything as configuration in Git, you are comfortable choosing, pinning and occasionally forking Singer taps, you like the tap and target pipe model, and you want an MIT core with no platform to run.
dlt is better when your team writes Python, many sources are your own databases or niche APIs, you want ingestion to live in the same repository as the rest of your code, you want to run it anywhere from a laptop to an orchestrator task, and you want schema contracts and state recovery without a server.
A script is better for one-off, one-source jobs and for process logic that is not really ingestion.

When not to adopt each

Do not adopt Airbyte to move three tables, if nobody can operate Kubernetes, if you plan to resell it as a service, or if you cannot keep up with platform upgrades. A stale Airbyte with known critical advisories holding all your API keys is worse than eleven scripts.
Do not adopt Meltano if the taps you need are abandoned forks and nobody on the team will maintain them, or if your team wants a UI.
Do not adopt dlt if nobody on the team writes Python comfortably, or if you need non-engineers to manage connections.
Do not adopt any of them for streaming CDC at low latency. Use the tools in the CDC post.

A short history

Extract, transform, load (ETL) tools were for decades large commercial suites. Cloud warehouses made it cheaper to load raw data first and transform it in SQL afterwards, which became ELT, and dbt made the “T” a SQL project. Stitch published the Singer specification around 2016 and 2017, giving taps and targets a simple shared language of JSON lines. Meltano started inside GitLab and became an independent company in 2021, building a CLI and Hub around Singer; by early 2026 Matatika had acquired the project. Airbyte launched in 2020 with its own protocol and containerised connectors, moved its platform to ELv2 in 2021 and its connectors in 2025, and has since pushed its Cloud and hybrid offerings while telling Self-Managed Enterprise users not to upgrade. dlt came in 2022 with a different idea: ingestion as a Python library rather than a platform, reaching 1.0 in September 2024. In the same years, CloudQuery narrowed towards cloud asset inventory, and Estuary combined batch and streaming under a BSL licence.

Decision guide

Start with your sources

Mostly popular SaaS apps with certified connectors: Airbyte’s catalogue saves real work.
Mostly your own databases: dlt’s SQL source, Sling, or a Singer tap; consider CDC if deletes and latency matter.
Mostly niche APIs: dlt’s REST API source or Airbyte’s Connector Builder; plan to own the connector.
Mostly files: any of them, or a script.

Then match the team

Python engineers and an existing scheduler: dlt.
Config-in-Git culture, happy with CLI and forks: Meltano.
Many non-engineer users and a Kubernetes platform team: Airbyte.

Then check the licence for your use

Internal ingestion for your own company: all three are fine.
Offering ingestion as a service to clients: Airbyte’s ELv2 does not allow it; Meltano (MIT) and dlt (Apache-2.0) do, subject to each connector’s own licence.

Checklist

Every source is listed with an owner, a refresh need and a credential location.
Each connector has a named maintainer, and you know who fixes it when the API changes.
Connector and tool versions are pinned (image tags, pip versions or commits), not branches.
Incremental cursors are chosen deliberately, and deletes are handled by soft deletes, periodic full refresh or CDC.
State is stored durably and backed up: Airbyte’s database, Meltano’s state backend, or dlt’s destination state.
Schema change behaviour is set per table, with freezing for finance-critical tables.
Failed loads, paused connections and contract violations alert a channel people read.
Source accounts are read-only; destination accounts can write only to raw schemas.
Secrets live in a secrets manager or environment, never in Git or config files.
Upgrade notes are read for every version you skip, with a backup before one-way migrations.
For Airbyte: version 2.4 or later, auth enabled, not exposed to the internet, workspace access limited.
The licence is checked against how you use the tool, especially if clients will use it.

Common mistakes

Choosing by connector count. Seven hundred connectors do not help if your three most important sources are not among the well-maintained ones.
Believing old licence summaries. Airbyte connectors are ELv2 since July 2025, not MIT.
Treating the first load as the project. The real work is the hundredth incremental run after an API change.
Unpinned plugins. A tap installed from a Git branch can change under you overnight.
Silent schema evolution on money tables. A new nullable column is harmless; a changed amount type is not.
Cursor columns that do not move. If the application does not update updated_at on every change, incremental loads miss rows.
Ignoring pending or failed state. In our drill, a failed dlt load left a pending package that blocked the next run until we aborted it.
Letting the ingestion tool become the secrets store nobody audits.

The drill we ran

The drill was small and local and ran on 30 October 2026 between about 10:22 AM and 10:24 AM IST, inside a user folder. Nothing listened on the network, and we did not install packages system-wide, use Docker or touch system directories. These are results from one run on this box, not a benchmark.

We created a Python 3.13 virtual environment and installed dlt 1.32.0 with the DuckDB extra, which brought DuckDB 1.5.6.
We generated a synthetic SQLite “order system” with a fixed seed: 1,000 orders with distributor, SKU, quantity, amount and updated_at. We loaded it into a local DuckDB file with the incremental merge resource shown above.
Initial load: 1,000 rows in about 0.35 seconds.
Incremental merge: after updating 50 existing rows and adding 100 new ones, the next run extracted 150 rows, and the destination had 1,100 rows and 1,100 distinct IDs, so the updates merged instead of duplicating.
No-change rerun: 0 rows. Our query used >= on the cursor, and dlt de-duplicated the boundary row.
New column: when the source gained a gst_rate column, dlt added it in DuckDB without any change in code.
Type change: when one row arrived with quantity as the text “12 pcs”, dlt kept the integer column and created a variant column quantity__v_text holding “12 pcs”, with the integer column null for that row. Nothing failed, which is exactly why you need a data test for variant columns.
Frozen contract: with schema_contract={"columns": "freeze"}, a new channel column made the run fail at the normalise step, and nothing was loaded.
A snag worth knowing: in our first attempt, the next run did not extract new data at all. dlt warned that it would first load the pending package from the failed run, and that package failed again. We had to abort it (the drop_pending_packages method we used is deprecated since 1.30.0 in favour of abort_packages; the CLI has dlt pipeline <name> abort-packages). After that, a run with discard_value loaded the row without the channel value.
State recovery: we deleted the local pipeline working directory and added one new row. The next run restored state from the _dlt_pipeline_state table in DuckDB and extracted only that one row, not the whole table. We also noticed that the discard_value setting from the previous run was still in effect, so the channel column was not created even though the code no longer set a contract. It looks like the contract was saved with the stored schema; we did not dig further, but it is a reminder to check what your stored schema says.
Meltano: in a separate virtual environment we installed Meltano 4.4.1, created a project, and added the MeltanoLabs CSV tap (version 1.2.0, built on Singer SDK 0.54.9) and a JSONL target. The older form meltano add extractor tap-csv failed with “Utility ’extractor’ is not known to Meltano”; meltano add tap-csv worked. The Hub definitions installed the tap from a Git branch with no pin. The first run wrote 120 rows. After we appended one row, the second run re-read the whole file and the append-only target ended with 241 lines for 121 distinct IDs, because this tap has no incremental state. Meltano stored the (empty) bookmark in its SQLite system database.
A pgrep check at the end showed no drill process running.

What we did not run: Airbyte (it needs Docker and a local Kubernetes cluster through abctl, or Helm on Kubernetes, and Docker is not allowed on this box), PyAirbyte, Sling, CloudQuery and Estuary Flow. We did not test real SaaS APIs, CDC, cloud warehouses, large volumes, concurrency, or upgrades. A useful next step is the same incremental and schema-change tests against an Airbyte 2.4 install with the PostgreSQL source.

Unlearn, re-learn

Unlearn “Airbyte connectors are MIT”. Re-learn that Airbyte moved its connectors to ELv2 in July 2025, and read the licence for your use.
Unlearn “more connectors means less work”. Re-learn to check who maintains your top ten sources and how quickly they fix breakages.
Unlearn “ingestion needs a platform”. Re-learn that a library like dlt plus your existing scheduler covers many small and medium teams.
Unlearn “schema evolution is always good”. Re-learn to freeze finance tables and alert on change.
Unlearn “state is a detail”. Re-learn where each tool keeps it, how to back it up, and how to reset it safely.
Unlearn “pip install from the Hub is pinned”. Re-learn to pin taps, targets, images and libraries to exact versions.
Unlearn “the extract script is fine because it ran last night”. Re-learn to make every load loud when it fails.

Sources

comments powered by Disqus

Releted Posts

Graph databases in 2026: revisit Neo4j Community, Memgraph and Apache AGE before you model another relationship in SQL joins

Manoj Shukla
October 30, 2026

Most teams do not decide to adopt a graph database. They arrive at it through a query. Somebody asks “which customers are connected to that blocked customer through a shared phone number, bank account or address?

Manoj Shukla
October 29, 2026

Every growing company ends up with a second, quieter product: the screens its own staff use. Support needs to look up an order and resend an invoice.

Manoj Shukla
October 29, 2026

Most teams do not decide to build a data catalog. They drift into needing one. A new analyst asks which of three “orders” tables is the real one.