Showing Post From Data Pipelines
Data transformation in 2026: revisit dbt Core v1, dbt v2 and SQLMesh before you write another SQL model
Almost every reporting database has a folder of SQL that nobody fully owns. It starts with one view for daily sales.
Read moreData ingestion in 2026: revisit Airbyte, Meltano and dlt before you write another extract script
Almost every data team has a folder of extract scripts. One pulls orders from the shop platform every night, one reads the payment gateway’s settlement API, one copies a few tables from the ERP database, and one downloads a CSV that somebody uploads to a shared drive.
Read moreChange data capture in 2026: revisit Debezium, Apache Flink CDC and PeerDB before you stream another table
Picture a SaaS company in Pune that runs billing for FMCG distributors across Maharashtra (an invented example, not a real client).
Read moreData pipeline orchestration in 2026: revisit Apache Airflow, Dagster and Prefect before you schedule another DAG
Picture a two-wheeler loan company in Coimbatore with about ninety branches (an invented example, not a real client). Its data team of five runs about 140 cron jobs on two Linux VMs: shell scripts that pull collections files from a payment gateway SFTP, Python scripts that clean them, and SQL that loads a PostgreSQL reporting database every night.
Read more