In-process analytics in 2026: revisit DuckDB, Polars and chDB before you spin up another warehouse for a small question

Manoj Shukla
October 20, 2026

Think of a knitwear exporter in Tiruppur that makes T-shirts, polos and kids’ wear for buyers in Europe, the US, the Gulf and Japan (an invented example, not a real client). It has knitting and stitching units in Avinashi and Palladam and a dyeing partner in Perundurai. Every night the ERP drops a CSV of order lines: order, style, fabric, pieces, unit FOB price, buyer country and shipping mode. Thirty months of history comes to about twenty million lines. The people who ask questions are two merchandisers, one finance executive and one developer who also keeps the office network running.

The questions are small and frequent: FOB value by buyer country for each month of the last financial year, or which styles moved by air instead of sea. A consultant has proposed a cloud warehouse with a nightly load job. The developer would rather keep a DuckDB file on the office NAS so that everyone can open it from a notebook. The finance executive has started using Polars, because a friend said it is fast. And the group company’s shipment tracker already runs on a ClickHouse server, so someone suggested chDB to “use the same SQL everywhere”. Last month two merchandisers opened the trial DuckDB file on the NAS together and one got a lock error during a buyer call. Then a Polars upgrade changed the row order of a purchase order reconciliation sheet, and a buyer’s team spent a morning matching lines that had not changed.

This post sets DuckDB, Polars and chDB side by side as they stood on 20 October 2026: history, governance and licences, current versions, file and table formats, larger-than-memory work, multi-process access and file locking, persistence formats, extensions and their trust model, language bindings, SQL versus DataFrame APIs, the security record for 2025 and 2026, and the traps that show up in daily work. It is about analytics that fit on one machine. The earlier Revisit.Tech posts on real-time OLAP servers and on lakehouse table formats met DuckDB only in passing; here the engine inside your own process is the subject.

Everything below was checked against primary material: GitHub release and advisory data, PyPI metadata, the projects’ own documentation and blogs, OSV and CVE records, and a drill on one shared box that installed DuckDB 1.5.6, Polars 2.0.0 and chDB 4.4.0 (on chdb-core 26.9.0) with pip and ran the same invented export data through all three. The drill looks at behaviour and only notes single-run timings with a caveat. If no primary source covers a point, the text says “Not documented”. On licences, this is my reading and not legal advice.

The short version

An in-process engine removes the server, not the jobs the server used to do. Someone still has to decide who may write, how much memory a query may take, which files a query may open and which third-party code may load. With DuckDB, Polars and chDB those decisions move into your process and your configuration. If nobody makes them, the defaults make them for you.
On honest questions, the answers agree. The same monthly FOB report over 20 million rows gave identical result hashes in DuckDB SQL, Polars expressions, Polars SQL and chDB. The differences that matter are around the query: type inference, row order, memory and locking.
Files are the safest interface between people. A DuckDB database file allows one read-write process or many read-only ones, never both. chDB allows one process per data directory and one data directory per process. Parquet written once by one job and read by many is boring and works with all three.
“Larger than memory” means different things. With a 1 GB limit, DuckDB sorted all 20 million rows to Parquet by spilling about 0.99 GB to disk. Polars 2.0 did it with a 1 GB out-of-core budget but its process still peaked at about 3.3 GB. chDB refused the same sort at 1 GB and 2 GB limits and completed only without a limit.
Upgrades are where in-process engines bite. Polars 2.0 runs lazy queries on the streaming engine by default, and a left join on a million rows no longer came back in left-hand order. DuckDB 2.0 is tentatively planned for 21 October 2026, with a new default storage format; a file made by the 2.0 alpha could not be opened by 1.5.6.
Extensions and SQL are code. DuckDB checks extension signatures on every load and refused a core extension with one flipped byte. It also reads /etc/passwd by default, as does chDB, and so does Polars SQL through read_csv. Lock file access down before anyone types SQL you did not write.

Where things stand on 20 October 2026

Times below are in IST. Release times come from GitHub published_at; PyPI upload times are within minutes of them. Stars were read on 20 October and only show attention, not quality.

DuckDB

Latest: v1.5.6 (28 September 2026, 7:05 PM), the sixth patch of the 1.5 “Variegata” line that began with v1.5.0 on 9 March 2026. The 1.5.6 announcement says it ships bug fixes, performance work and security patches; the notes include “Harden temp file reads” (pull request #25283).
LTS line: v1.4.5 (17 June 2026, 4:12 PM). The release calendar lists end of life for 1.4 “Andium” on 17 November 2026 and for 1.5 on 1 March 2027.
Next: the calendar shows 2.0.0 on 21 October 2026 and 2.0.1 on 18 November, both marked tentative. Alpha builds are on PyPI as pre-releases; the one I installed reported v2.0.0-alpha49003.
Licence and governance: MIT. The non-profit Stichting DuckDB Foundation in the Netherlands holds the intellectual property and trademarks, and its page says DuckDB, DuckLake and Quack stay MIT “irrevocably”. The core contributors work at DuckLabs (duckdblabs.com now redirects to ducklabs.com), which the DuckDB blog said on 26 August 2026 would join Amazon Web Services as a subsidiary in early September, with no change to roadmap, licensing or governance.
Attention: 42,282 stars.

Polars

Latest Python: 2.0.0 (6 October 2026, 5:22 PM), after two release candidates in September and 1.44.2 on 9 September. Latest Rust crate: 0.55.2 (6 August 2026, 3:18 PM).
Release rhythm and packaging: 22 stable Python releases dated 2026. pip install polars now pulls a separate runtime wheel (polars-runtime-32 here); the docs offer polars[rt64] for frames above 2^32 rows and polars[rtcompat] for CPUs without AVX2.
Licence and governance: MIT, in the pola-rs organisation. A company around Polars was announced on 3 August 2023 by Ritchie Vink and Chiel Peters; on 29 September 2025 it announced an 18 million euro Series A led by Accel. It sells Polars Cloud and a distributed engine; the home page says Polars “is and always will be open source”. The company’s legal name: Not documented on the pages I read.
Attention: 40,126 stars.

chDB

Latest: chdb 4.4.0 (11 September 2026, 2:46 PM), which requires chdb-core>=26.7.0 and adds chdb.durable, a database whose state lives in object storage with a lease that makes single-writer a guarantee. The engine package is chdb-core v26.9.0 (28 September 2026, 2:23 PM), whose notes give a ClickHouse baseline of v26.9.2.8-stable. On this box, SELECT version() returned 26.9.2.1, and the chDB overview page on clickhouse.com still said “ClickHouse v25.8.2.1”.
The split: chDB 4.1.0 (5 March 2026) moved the engine into a new chdb-core repository; the chdb wheel is now pure Python. There were 20 chdb releases dated 2026, three of them betas.
Licence and governance: Apache-2.0 for chdb and chdb-core. chDB was started by Auxten; ClickHouse announced on 6 March 2024 that chDB “is now part of ClickHouse” and that its creator had joined. The README says it is named chDB “for trade mark and other reasons”.
Attention: 2,920 stars for chdb, 23 for chdb-core.

Benchmarks you will be shown

Every figure you will be shown is its publisher’s own run. ClickBench is run by ClickHouse. db-benchmark is maintained by DuckDB Labs as a fork of the old H2O.ai suite (report generated 17 August 2026, 4:40 PM). PDS-H is Polars’ own TPC-H derived suite, last posted on 1 June 2025, and the Polars 2.0 post reports Polars’ own TPC-H and TPC-DS derived runs against DuckDB 1.5.6, a DuckDB 2.0 alpha and DataFusion 54.0.0. chDB’s overview page shows its own DataStore versus pandas timings. None of them is your workload, so I repeat no ranking here.

Timeline

2018: DuckDB started at CWI in Amsterdam by Mark Raasveldt and Hannes Muehleisen; 0.1.0 followed on 27 June 2019.
23 June 2020: first commit of Polars, per the company announcement.
25 February 2023: the chdb repository is created.
3 August 2023: a company around Polars is announced.
31 August 2023: ClickHouse’s blog introduces chDB to its community.
6 March 2024: chDB becomes part of ClickHouse.
3 June 2024: DuckDB 1.0.0. 1 July 2024: Polars 1.0.0 on PyPI.
9 September 2025: four DuckDB npm packages are briefly published with crypto-stealing malware after a phishing attack (CVE-2025-59037).
16 September 2025: DuckDB 1.4.0, the first LTS line.
12 November 2025: DuckDB 1.4.2 fixes weaknesses in database encryption (CVE-2025-64429).
27 January 2026: chDB 4.0.0. 5 March 2026: chDB 4.1.0 splits off chdb-core.
9 March 2026: DuckDB 1.5.0, with the plan for a 2.0 release in the autumn.
April 2026: DuckLake 1.0 specification and implementation.
12 May 2026: DuckDB introduces the Quack remote protocol as a preview.
26 August 2026: DuckLabs announces it will join AWS.
11 and 28 September 2026: chDB 4.4.0; DuckDB 1.5.6 and chdb-core 26.9.0.
6 October 2026: Polars 2.0.0.
21 October 2026 (tentative): DuckDB 2.0.0. 17 November 2026: end of life for DuckDB 1.4 LTS.

First question: does this question need a server?

A server database quietly does four jobs that nobody lists in a requirement document.

It owns the state. One process holds the data and decides the order of writes from many clients.
It owns the memory. It knows every running query and can queue, cancel or limit them as a group.
It owns access. Clients log in, get roles and can only touch what they are granted. Files on the server’s disk are not theirs to read.
It owns upgrades. The server version changes once, in one place, on a planned night.

An in-process engine is a library inside your Python, R, Java or Node process. It gives you a fast columnar SQL or DataFrame engine with no network hop and no service to run. In return, all four jobs come back to you. The writer is whichever process got the lock first. The memory limit is whatever this one process was told. Access is whatever the operating system user can read. The version is whatever pip installed on this laptop last Tuesday.

So the honest promise of an in-process engine is narrower than “a warehouse without the warehouse”:

“One process, with these files, this much memory and this code loaded, will answer this question correctly, and nobody else is writing the same thing at the same time.”

When that sentence fits your question, a server is overhead. When it does not, an in-process engine needs help: a single writer job, a catalog, or a real server.

What each one actually is

DuckDB 1.5.6

DuckDB is an analytical SQL database that runs in-process, with clients for C, the CLI, Java, Go, Node.js, ODBC, Python, R, Rust and WebAssembly at the primary support tier, and ADBC, C# and C++ at the secondary tier. It runs in memory or keeps a single-file database on disk, queries Parquet, CSV and JSON files directly, spills large sorts, joins, aggregations and windows to disk, and uses MVCC with optimistic concurrency between threads of one process.

Most format and connector support lives in extensions: httpfs for HTTP and S3, iceberg, delta, ducklake, spatial and many more. Core extensions are signed by the DuckDB team, community extensions are built and signed by the DuckDB team from public source, and both are checked on every load. DuckDB 1.5 added VARIANT and GEOMETRY types and a reworked CLI. DuckDB 2.0 brings Quack as a stable client-server protocol, a PEG parser, a new default storage format and a reworked C API.

Polars 2.0.0

Polars is a DataFrame library with a query engine written in Rust, used mainly from Python. The heart of it is the lazy API: you describe a query with expressions (scan_parquet, filter, group_by, join, agg), Polars optimises the plan, and collect or sink_* runs it. An eager API and a SQL interface (pl.sql, SQLContext) sit on the same engine. The SQL guide says plainly that the DataFrame interface is primary and new features usually reach the expression API first, although the 2.0 post calls SQL “a first class citizen” from now on.

Two 2.0 changes matter for operations. Lazy queries now run on the streaming engine by default, which does not promise row order for joins, group-bys or unpivots unless you ask. And out-of-core processing is on by default: per the release post, spilling starts at about 80% of RAM with a default disk budget of 64 GB, for sorts, window functions and many expressions, with joins and group-bys still on the roadmap. Polars has no database file of its own; persistence means writing Parquet, IPC, CSV, Delta or Iceberg.

chDB 4.4.0 on chdb-core 26.9.0

chDB is the ClickHouse engine packaged as a library. The overview page lists bindings for Python, Go, Rust, Node.js, Bun and C/C++, and the README’s maturity table adds early or experimental Java, Ruby, .NET, Zig and WebAssembly work. You get ClickHouse SQL, more than 70 input and output formats, table functions such as file(), s3(), url(), deltaLake() and iceberg(), and the ability to query pandas DataFrames and Arrow tables in memory through Python(). A Session with a path keeps databases and MergeTree tables on disk in the same layout as clickhouse-local.

On top sit DataStore, a pandas-compatible API that compiles to SQL, and a beta agent tool with engine-enforced read-only mode. Memory limits and external sort and group-by thresholds are ClickHouse settings, set per query. One rule from the README shapes everything else: “chdb-core allows only one active data path per process”.

Side by side

Where the official material says nothing, I say “Not documented”. Drill results are marked as such and come from one shared VM.

Licence and governance

DuckDB: MIT, IP held by a non-profit foundation; the main developers now work for an AWS subsidiary. MotherDuck is a separate company that contracts with DuckLabs.
Polars: MIT; the core team works for the Polars company, which sells a cloud and distributed product on the same API.
chDB: Apache-2.0, owned and developed by ClickHouse.
In practice: all three are permissive, with no copyleft duty on code that only calls the library. The real question is who funds the roadmap: a foundation with a company behind it, or a company. My reading, not legal advice.

Interface: SQL, DataFrame or both

DuckDB: SQL first, in a friendly dialect (FROM first, GROUP BY ALL), plus a relational Python API. In the drill it queried a Polars DataFrame by variable name and returned a Polars frame with .pl().
Polars: expressions first, with SQL on top; both gave the same answers in the drill.
chDB: ClickHouse SQL first, with DataStore and a DB API 2.0 layer on top. It queried an Arrow table and a pandas DataFrame through Python() in the drill.
Type inference differs. From the same CSV and NDJSON files, DuckDB inferred order_date as DATE, chDB as Nullable(Date) and Polars as String. The Nullable inference then broke a chDB MergeTree ... ORDER BY line_id created from Parquet: “Sorting key contains nullable columns”.

File formats and table formats

DuckDB: Parquet, CSV and JSON built in, others through extensions. Iceberg tables can be read from storage without a catalog, and written through an attached Iceberg REST catalog. Delta is read and written through the delta extension. DuckLake, a lakehouse format with a SQL catalog, reached 1.0 in April 2026.
Polars: read, scan, write and sink for CSV, Parquet, NDJSON and IPC, plus Delta (through the deltalake package) and Iceberg scan and write functions.
chDB: the ClickHouse format list, including Parquet, CSV, JSON, Arrow and ORC. ClickHouse’s documentation says Iceberg writes exist from version 25.7 behind allow_insert_into_iceberg, and Delta Lake writes are Beta and disabled by default. I tested only reading in chDB.
Drill: a Delta table written by Polars in two commits gave identical counts and sums in Polars, DuckDB’s delta_scan and chDB’s deltaLakeLocal. Iceberg was not tested.

Larger than memory

DuckDB: memory_limit defaults to 80% of RAM, and spilling goes to a .tmp folder next to the database, or to temp_directory, capped by max_temp_directory_size (default 90% of free disk). The tuning guide lists gaps: several blocking operators in one query can still run out of memory, and list(), string_agg() and so PIVOT cannot spill.
Polars: out-of-core is on by default in 2.0, starting at about 80% of RAM with a 64 GB disk budget, for the operations listed above. The release notes add POLARS_OOMKILL_THRESHOLD_MB. The environment variable I used to lower the budget, POLARS_OOC_MEMORY_BUDGET_MB, came from the library binary; its documentation: Not documented on the pages I read.
chDB: ClickHouse settings per query: max_memory_usage (0, that is unlimited, by default in this build), max_bytes_before_external_sort and max_bytes_before_external_group_by, with a ratio setting of 0.5 for sorting.
Drill, sort 20 million rows to Parquet: DuckDB with a 1 GB limit finished, with up to 987,103,232 bytes in 7 temp files and a peak RSS of 1,533 MB. Polars with a 1,024 MB budget finished, with up to 2,489,307,072 bytes in 1,314 spill files and a peak RSS of 3,349 MB. chDB with a 1 GB limit, and again with 2 GB, failed with MEMORY_LIMIT_EXCEEDED even with external sort thresholds of 100 MB and 400 MB; without a limit it finished at a peak RSS of 3,210 MB. A group-by over 5 million orders spilled 54 MB and finished in chDB under 1 GB. All three sorted files had 20,000,000 rows in the same order.

Concurrency and multi-process access

DuckDB: one process may open a database file read-write, or many processes read-only; the docs say so plainly, and file locks enforce it. Inside that one process, many threads can write as long as they do not touch the same rows. Writes from many processes need the Quack protocol (beta since 1.5.2) or DuckLake with a PostgreSQL catalog. The tuning guide says fast concurrent small queries are “not a primary design goal”.
Polars: no shared database, so no locks. Parallelism is threads inside one process. The docs warn that Python’s fork start method does not mix with Polars’ threads; use spawn.
chDB: one data directory per process, and a second process on the same directory is refused. A Session without a path is temporary.
Drill: with DuckDB holding the file read-write, a second process failed in both read-write and read-only mode: “Could not set lock on file … Conflicting lock is held in /usr/bin/python3.13 (PID 416275)”. With a read-only holder, a second read-only open worked and read-write failed, with a hint to use read-only. In chDB, a second process on the same path got “Cannot lock file …/status. Another server instance in same directory is already running”, and one process asking for a second path got “EmbeddedServer already initialized with path …, cannot connect with different path”.

Persistence and storage formats

DuckDB: one database file plus a write-ahead log. Versions 1.0 to 1.5 write storage version 64 by default, readable by every 1.x release, and you can opt into newer formats with STORAGE_VERSION. Forward compatibility is only “best effort”, and DuckDB 2.0 moves the default to storage format v2.0.0. Between incompatible versions the docs suggest EXPORT DATABASE with the old binary and IMPORT DATABASE with the new.
Polars: no database format; Parquet and Arrow IPC are the usual choices, plus Delta and Iceberg for tables.
chDB: a directory in the clickhouse-local layout, which ClickHouse says is interchangeable with clickhouse-local. chDB 4.4 adds chdb.durable for state kept in object storage under a lease.
Drill: a file made by DuckDB 1.5.6 reported storage_version v1.0.0+ and opened fine in the 2.0 alpha. A file made by the 2.0 alpha reported v2.0.0+, and 1.5.6 refused it: “Trying to read a database file with version number 69, but we can only read versions between 64 and 68.” On disk with defaults, the 20 million rows took 214 MB as a DuckDB file, 900 MB as a chDB MergeTree with no sort key and Nullable columns, and 2.0 GB as Polars IPC, against 166 MB of zstd Parquet. Untuned defaults, not a compression claim.

Extensions and what they can do

DuckDB: the security page says extensions run “with the same privileges as the user running DuckDB’s (parent) process”. Signed core extensions load by default, signed community extensions load by default (allow_community_extensions = false turns them off), and unsigned ones need allow_unsigned_extensions, which can only be set at start-up. Known core extensions are installed and loaded automatically on first use unless you turn off autoinstall_known_extensions and autoload_known_extensions. DuckDB 2.0 plans organisation-hosted extension repositories with pinned RSA keys.
Polars: expression plugins are compiled shared libraries loaded into the process; a trust or signing model for them: Not documented. LazyFrame.deserialize warns that it uses pickle for Python UDFs and “can execute arbitrary code”.
chDB: no extension store; Python UDFs (scalar, stateless) and, since chdb-core 26.9, Python aggregate functions run inside the engine.
Drill: loading a downloaded, signed httpfs worked. The same file with one byte flipped was refused: “its signature is either missing or invalid”. With community extensions turned off, the signed community h3 was refused with the same wording, which can mislead someone debugging it.

Language bindings and packaging

DuckDB: the Python wheel has no Python dependencies; its native module was 60,489,720 bytes on this box.
Polars: polars plus polars-runtime-32, about 189 MB installed, with optional extras.
chDB: depends on chdb-core, pandas and pyarrow; the chdb folder alone was about 534 MB, and pandas 3.0.6 and pyarrow 26.0.0 added about 72 MB and 167 MB.

Release rhythm and upgrade risk

DuckDB: minor releases can break things (in 1.5, date_trunc on a DATE returns a TIMESTAMP); LTS lines get about a year of community support.
Polars: 2.0 is a long list of breaking changes, from the streaming default to stricter casts, read_csv going through scan_csv and Parquet ENUM read as String.
chDB: package and engine are versioned separately, the engine follows ClickHouse release lines, and the docs can lag behind.

Worked example for the Tiruppur scenario

Remember that the company’s trouble was never speed. It had a shared file that nobody owned, an upgrade nobody tested, and no rule about where questions get answered.

Step 1: write down where the data lives and who writes it. One writer: the nightly ERP export. Everything else reads. That alone rules out a shared read-write DuckDB file on the NAS, because two merchandisers opening it is exactly the case the lock exists to stop. The DuckDB docs also advise care with database files on network or shared file systems.

Step 2: make Parquet the interface. A small job converts the nightly CSV into Parquet, one file per month, with explicit types (dates as DATE, money as integer paise, not floats). In the drill, the same CSV became a DATE in DuckDB, a Nullable(Date) in chDB and a String in Polars, so let one job decide the schema once.

Step 3: give each person an engine, not a server. The merchandisers ask SQL questions in DuckDB from a notebook over the Parquet folder. The finance executive keeps Polars for her reconciliations. Each laptop gets memory_limit and temp_directory on a local SSD, never on the NAS. If someone needs a DuckDB database file, it is a personal copy or a read-only snapshot that the nightly job publishes.

Step 4: pin the versions and test the upgrade. Pin duckdb==1.5.6 and polars==2.0.0 in one requirements file. DuckDB 2.0 is tentatively due on 21 October: do not let a 2.0 client write a database file that a 1.5 user must open, because 1.5.6 refused a 2.0 file in the drill. For the Polars reconciliation sheet, add an explicit sort before export, or maintain_order="left" on the join, and compare row order before and after any upgrade.

Step 5: harden the one script that reads files it did not choose. The buyer portal team wants to let buyers upload a CSV and run a fixed report on it. The DuckDB security page says to treat SQL like Bash or Python, so the report SQL stays fixed, uploads land in one folder, and the connection is locked down before the first query. These are the settings tested in the drill:

import duckdb
con = duckdb.connect(config={
    "memory_limit": "1GB",
    "temp_directory": "/srv/scratch/duckdb_tmp",
    "threads": 4,
})
con.sql("SET allowed_directories = ['/srv/uploads/buyer_csv']")
con.sql("SET enable_external_access = false")
con.sql("SET allow_community_extensions = false")
con.sql("SET autoinstall_known_extensions = false")
con.sql("SET autoload_known_extensions = false")
con.sql("SET lock_configuration = true")
# reads under /srv/uploads/buyer_csv work; /etc/passwd, COPY to /tmp
# and later SET statements are refused; spilling to temp_directory still works

In the drill, a window sort over 20 million rows still spilled about 274 MB under these settings, while /etc/passwd, a COPY to /tmp and a later SET threads were refused. This is defence in depth, not a replacement for the container or low-privilege user that the same page recommends for untrusted SQL.

Step 6: use chDB where ClickHouse SQL is the point. The group’s shipment tracker runs on a ClickHouse server, so the developer can use chDB to prototype ClickHouse queries against the same Parquet without touching production. Keep one data directory per process, test every memory limit, and note that failed exports left 0-byte Parquet files in the drill.

Step 7: decide the line for a server. When several people must write the same tables at once, or buyers may run their own queries, they need a writer service: Quack once it matures, DuckLake with a PostgreSQL catalog, or their existing ClickHouse server. Until then, twenty million lines is a small question.

Security and hygiene

An in-process engine has no login screen. Whatever the process can read, a query can read, and whatever code it loads runs as you.

DuckDB. No repository advisory was published in 2026 as of 20 October. The latest is GHSA-vmp8-hg63-v2hp, CVE-2025-64429 (12 November 2025, 6:32 PM), on the database encryption added in 1.4.0: a possible fallback to a non-cryptographic random generator, key wiping a compiler may remove, a header flag that could downgrade GCM to CTR, and an unchecked OpenSSL rand_bytes() result. Versions 1.4.0 and 1.4.1 were affected; 1.4.2 fixed it. Earlier, GHSA-w2gf-jxc9-pf2q, CVE-2024-41672, showed sniff_csv reading files while enable_external_access was off (fixed in 1.1.0). The supply chain event was GHSA-w62p-hx95-gf2c, CVE-2025-59037 (9 September 2025, 8:09 PM): a maintainer was phished through a fake npm site that relayed the real one, and four packages (duckdb 1.3.3, @duckdb/node-api 1.3.3, @duckdb/node-bindings 1.3.3, @duckdb/duckdb-wasm 1.29.2) went out with code that tampered with cryptocurrency transactions. The team noticed within four hours, had npm delete them and published higher safe versions. PyPI wheels were not part of that advisory.

The Quack preview is worth reading before you use it: by default the server makes a random token at start-up and binds only to localhost; it does not use TLS by default and expects a proxy to terminate TLS for anything non-local; and the default authorisation callback allows every query. That is sensible for a laptop and wrong for a network.

Polars. The repository lists no security advisories, and OSV has no entries for the polars PyPI package or the polars crate. The documented risk is pickle inside LazyFrame.deserialize. In the drill, pl.sql("SELECT COUNT(*) FROM read_csv('/etc/passwd')") worked, so a SQL string from a user is file access, exactly as in the other two.

chDB. The chdb repository lists no security advisories, and OSV has no entries for the chdb or chdb-core PyPI packages. ClickHouse’s own security changelog covers the server; its latest entry is CVE-2025-1385 (library bridge, fixed in v25.1.5.5). Which server fixes reach a chdb-core build you infer from the ClickHouse baseline in its release notes; a chDB statement on this: Not documented. In the drill, file('/etc/passwd', LineAsString) worked by default and under readonly=2 (the mode the 4.2 agent tool uses, which “keeps file()/s3()/url() usable”), while readonly=1 refused it.

Habits that matter more than the version number

Pin the engine, its runtime wheel and its extensions, and read the notes before every minor or major bump.
Before running SQL you did not write, restrict file access and lock the configuration; for real untrusted input, add a container or a low-privilege user.
In DuckDB, turn off community extensions and automatic install where you do not need them; keep unsigned extensions for development only.
Do not deserialise Polars plans or load plugins from sources you do not trust.
Watch the npm, PyPI and GitHub advisories of every engine you embed; they are part of your supply chain.

Decision guide

Lean towards DuckDB if

Your users think in SQL and want to point it at Parquet, CSV, JSON, Delta, Iceberg or DuckLake.
You want a single-file database that can travel, and you can live with one writer at a time.
You want the most hardening switches of the three.
You can plan around 2.0: a new storage format, a new parser and the end of 1.4 LTS in November.

Lean towards Polars if

Your work is pipelines in Python with typed expressions, and SQL is occasional.
You want lazy plans, streaming and spilling with nothing else to run.
You will write explicit sorts or maintain_order wherever row order reaches a person.
You are fine with files as the only persistence.

Lean towards chDB if

You already run ClickHouse and want the same SQL, functions and formats in a script or notebook.
You need ClickHouse-only features such as its table functions or MergeTree tables locally.
You can accept a larger install, the pandas and pyarrow dependency, one data path per process, and memory settings that need testing.

Be careful whichever you pick

Do not trust “larger than memory” until your own biggest query has run under your own limit.
Run your three most common questions on your own data instead of reading a vendor benchmark.
Do not upgrade an engine inside a report pipeline without comparing output, including row order and types.

Tiruppur (invented) pick. Parquet written once a night by one job. DuckDB 1.5.6 for the merchandisers’ SQL, with memory limits and a local temp folder. Polars 2.0.0 for the finance pipeline, with explicit sorts on every exported sheet. chDB only on the developer’s machine for ClickHouse prototypes. A hardened DuckDB connection for the buyer upload report. No warehouse until many people need to write the same tables at once.

Checklist

Each dataset has exactly one writer, and everyone knows which job it is.
Shared data is Parquet (or a table format) with an explicit schema, not a writable database file on a NAS.
Engine versions, runtime wheels and extensions are pinned in one file under version control.
Every long-running process sets a memory limit and a temp directory on local disk.
The largest real query has been run under that limit and its spill size recorded.
Exports that a person reads are sorted explicitly.
No DuckDB 2.0 client writes a file that a 1.x user must open, until everyone has moved.
Any process that runs SQL it did not write has file access restricted and its configuration locked.
Community or unsigned extensions are turned off where they are not needed.
Someone reads the release notes and advisories of all three engines each month.

Common mistakes

Putting one DuckDB file on a network share and asking several people to open it.
Assuming a read-only open will work while another process holds the file read-write.
Opening a second chDB data directory in the same process and expecting it to work.
Trusting that Polars keeps join or group-by order after moving to 2.0.
Leaving dates as strings because Polars inferred them that way from CSV.
Building a chDB MergeTree from inferred Nullable columns and ordering by a nullable key.
Pointing temp_directory at a slow shared disk, or at nothing at all.
Reading a vendor benchmark as a promise about your laptop.
Letting users type SQL into an in-process engine with default file access.
Treating a failed export as harmless when it left an empty file that the next job reads.

Drill on this box, 20 October 2026

This box is one shared Debian 13 VM (8 vCPUs, 15 GiB of RAM) that other jobs also use. Installs ran at about 10:29 to 10:30 AM IST, tests from about 10:30 to 10:47 AM. Every timing below is a single run on a shared machine and is not a speed claim or a ranking.

Install. In a fresh venv with Python 3.13.5, pip install duckdb==1.5.6 polars==2.0.0 chdb==4.4.0 pyarrow deltalake brought in polars-runtime-32 2.0.0, chdb-core 26.9.0, pandas 3.0.6, pyarrow 26.0.0 and deltalake 1.6.6. DuckDB had no dependencies. A second venv held the DuckDB 2.0 pre-release 2.0.0.dev2610192346 for the storage test only.

Data. An invented Tiruppur order book generated with DuckDB: 20,000,000 order lines in 5,000,000 orders from 1 April 2024 to 29 September 2026, written as one zstd Parquet file of 166,423,249 bytes, plus a 2 million line CSV (126,368,886 bytes), a 200,000 line NDJSON file (39,038,319 bytes) and a 2,500 row style list.

Same answers. Q1 was monthly FOB value and line count by buyer country for April 2025 to March 2026 (144 rows). Q2 joined the style list and returned the top 10 category and fabric pairs by pieces with distinct order counts. Hashes of both results were identical across DuckDB SQL, Polars expressions, Polars SQL and chDB. Single runs, Q1 and Q2 in seconds: DuckDB 0.10 and 0.69, Polars 0.22 and 0.86, Polars SQL 0.21 and 0.83, chDB 0.26 and 0.55. Peak process RSS: 1,453 MB, 3,331 MB, 3,407 MB and 1,487 MB.

Inference. Row counts and sums matched across engines for CSV and NDJSON. order_date came back as DATE (DuckDB), String (Polars) and Nullable(Date) (chDB).

Memory. Spill sizes and peaks are in the side-by-side section. Single-run times for the sort: DuckDB 3.42 s, Polars 7.68 s, chDB 5.77 s (without a limit). The failed chDB runs left three 0-byte Parquet files.

Locks and paths. The messages quoted in the concurrency section came from these runs. Some chDB processes also printed Exception in global_context->shutdown(): The ShutdownTablesThreadPool is not initialized at exit, with exit code 0.

Polars 2.0 order. A left join of a 1,000,000 row frame against a reversed one:

default collect (2.0 auto->streaming): left order kept = False first k values [312506, 312513, 312524, 312526, 312533]
maintain_order=left: left order kept = True
engine=in-memory: left order kept = True
group_by output sorted by key = False  first keys [5, 68, 70, 77, 99]

Access and extensions. DuckDB read /etc/passwd by default; enable_external_access = false alone blocked it and the project’s own CSV too; adding allowed_directories for the data folder let paths inside it through. Turning external access or community extensions back on was refused while running. Defaults read back: unsigned extensions off, community extensions, autoinstall and autoload on. DuckDB’s own INSTALL hung on this box’s network path, so I fetched httpfs, delta and h3 for v1.5.6 with curl over HTTPS and loaded them from files.

Formats and interop. The Delta table had 1,000,000 rows plus a 1,000 row append; all three engines returned 1,001,000 rows and the same sum of pieces.

What this box could not test: DuckDB INSTALL from its repositories, Iceberg in any engine, Quack and DuckLake, chDB durable and object storage, cloud storage of any kind, Windows and macOS, the other language bindings, GPU engines, multi-user load, and any fair speed comparison. Nothing outside the drill folder was changed.

Unlearn

Unlearn that every analytical question needs a server and a load job.
Unlearn that “in-process” means “in memory”.
Unlearn that a database file is a safe thing to share on a drive.
Unlearn that a library upgrade cannot change your report.
Unlearn that a library has no attack surface because it has no port.

Re-learn

Re-learn the four jobs a server did: state, memory, access and upgrades. Assign each one.
Re-learn files as the contract between people, with one writer and a fixed schema.
Re-learn memory limits and temp folders, tested with your own largest query.
Re-learn row order as something you ask for, not something you get.
Re-learn extension and file policies as part of the code review.

Closing

In-process engines in 2026 are good enough that the old reflex of loading every question into a warehouse deserves a second look. DuckDB 1.5.6 reads the widest set of file and lakehouse formats from SQL here, with clearly documented locking and hardening switches and 2.0 tentatively due this week. Polars 2.0.0 is the strongest fit for typed pipelines in Python, now streaming and spilling by default, at the price of row order you must request. chDB 4.4.0 brings real ClickHouse into a process for teams who already speak it, with the heaviest install and memory settings you must test. When a question outgrows one process, the post on real-time OLAP servers covers the next step, and the post on lakehouse table formats covers how many engines can share the same files safely. Learn what a server was doing for you. Unlearn the warehouse reflex. Re-learn locks, limits and row order. Practise on your own data. Apply it before you spin up another warehouse for a small question.

Sources

DuckDB

Polars

chDB

Benchmarks (each run by its own publisher)

Security records

Internal

comments powered by Disqus

Releted Posts

Log storage in 2026: revisit Grafana Loki, VictoriaLogs and OpenObserve before you keep another month of logs

Manoj Shukla
October 15, 2026

Picture a platform team in Pune that runs a B2B lending stack for NBFCs across Maharashtra and Gujarat (an invented example, not a real client).

Manoj Shukla
October 12, 2026

Picture a payments and lending analytics platform in Hyderabad (an invented example, not a real client). Card authorisations and EMI schedules still land in PostgreSQL 16.

Manoj Shukla
October 12, 2026

Picture an ecommerce and last-mile logistics platform in Bengaluru (an invented example, not a real client). Order capture and inventory still run on PostgreSQL 16.