Stream processing in 2026: revisit Apache Flink, RisingWave and Arroyo before you write another consumer that keeps state

Most teams that use Kafka have at least one consumer that does more than it should. It started as a simple loop that read events and wrote them somewhere. Then somebody asked for “orders per minute”, so the loop got a dictionary. Then somebody asked for “alert if the same phone number places three cash-on-delivery orders in ten minutes”, so the dictionary got timestamps. Then the pod restarted during a sale, the dictionary was empty again, a rebalance replayed a few thousand events, and the per-minute numbers on the dashboard went up and down like a cricket score in the last over. Nobody could say which number was right.

That consumer is a stream processor. It just does not have the parts a stream processor needs: state that survives a crash, a clear idea of event time and late data, windows that close at the right moment, and output that does not double count after a replay. This post compares three open source engines Indian teams look at in 2026 to do that job properly: Apache Flink, RisingWave and Arroyo. We mention Kafka Streams, Spark Structured Streaming and Bytewax briefly. We look at versions, licences, the programming model (SQL or code), time and watermarks, windows and when results are emitted, joins and state size, checkpoints and recovery, what “exactly-once” really covers, upgrades, operations, footprint and security history. We also say when a plain consumer is still the honest answer.

This post sits next to several earlier ones and tries not to repeat them. Change data capture in 2026 covers getting row changes out of your databases; here we assume the events are already in a topic. Kafka and Redpanda and RabbitMQ, NATS and Kafka queues are about the broker; this post is about what reads from it and keeps state. Schema registries in 2026 matters because every engine here has to decode your events. Real-time OLAP in 2026 is often the better answer for dashboards, and we explain when. Lakehouse table formats is where many stream results finally land. Idempotency keys is the simple idea that rescues most “exactly-once” plans. Durable execution in 2026 is the right tool when the job is a business workflow, not a continuous calculation. Data ingestion in 2026 covers batch pulls. And if you run these engines on Kubernetes, Kubernetes autoscaling in 2026 explains why scaling a stateful job is not the same as scaling a web pod.

We end with a small drill: Arroyo 0.15.0 and RisingWave 3.1.0 running locally on loopback, with a graceful stop, a crash, a changed query, a late event, a window that waits for the watermark and an interval join. We say clearly what we did not run, including Flink.

A scenario to keep us honest

Imagine a leather footwear brand in Kanpur. The company is invented for this post. It makes formal shoes, sandals and safety boots in its own unit, sells through its own online store and three marketplaces, and runs about 40 retail outlets across Uttar Pradesh, Bihar and Madhya Pradesh. Some outlets are in market areas with poor mobile data, so their billing machines store sales and upload them in batches when the connection returns, sometimes an hour late. The engineering team has nine people.

The setup today:

• An order service and an inventory service on PostgreSQL, in a cloud region in Mumbai.
• A three-broker Kafka cluster that already carries orders, payments, pack_scans (from handheld scanners in the warehouse) and pos_sales (from the outlets).
• A Python consumer called live-metrics that reads all four topics, keeps counts in a dictionary, and writes them to Redis every few seconds for a sales dashboard.

During the last festive sale three things went wrong. The live-metrics pod was restarted by a deployment in the middle of the evening peak, its dictionary started from zero, and the dashboard showed a sudden drop that made the owner call the CTO. A consumer group rebalance replayed a few minutes of orders, so some SKUs were counted twice, and the online store briefly showed a popular sandal as out of stock while the warehouse still had it. And a batch of late outlet sales, uploaded after an outage, landed in “the current minute” instead of the minutes they actually happened, so the outlet sales chart had a tall spike at 9:40 PM that never happened in real life.

Now the business wants four things before the next sale:

1. A live, per-SKU view of units sold across all channels in the last few minutes, so the store can stop selling before stock runs out.
2. A cash-on-delivery check: flag when the same phone number places three or more COD orders within ten minutes.
3. An alert when a paid order has not been packed within two hours.
4. Per-minute sales by channel and outlet for the dashboard, with late outlet uploads counted in the right minute.

The CTO has used Flink at a previous job. A senior developer has read that RisingWave lets you do this “with just PostgreSQL SQL”. A younger developer likes Rust and has been reading about Arroyo. We will come back to Kanpur in the worked plan.

The short version

• Apache Flink is the general-purpose engine. Version 2.3.0 came out on 25 June 2026, and 1.20 is the long-term support line. It has the widest set of APIs (SQL, Table API, DataStream in Java and Python), the most connectors and the deepest control over state, time and checkpoints. It also has the most knobs, and you run a JobManager and TaskManagers, usually on Kubernetes with the Flink Kubernetes Operator.
• RisingWave is a streaming database. Version 3.1.0 came out on 21 September 2026. You talk to it with PostgreSQL clients, define sources and materialized views in SQL, and query the results directly. State lives in object storage. The core is Apache-2.0, but several features need a paid licence key, and that list grows over time, so read it before you design around a feature.
• Arroyo is a compact engine written in Rust, SQL-first, that runs as a single binary for small setups. Cloudflare acquired it in April 2025, and it now also powers Cloudflare Pipelines. The last release is 0.15.0 from 1 December 2025. The repository is active, but there has been no release for about eleven months, which matters if you need fixes.
• None of them makes your output exactly-once by itself. Exactly-once inside the engine is about state. End to end, it depends on your sink: transactional writes, or idempotent upserts keyed by something stable like a window and a key.
• If your logic is per-event (validate, enrich, route), a plain consumer with idempotent writes is often still the right answer. Reach for an engine when you need state over time: windows, joins across streams, de-duplication over hours, or “this did not happen within X minutes”.

Where things stand on 30 October 2026

• Apache Flink: 2.3.0 on 25 June 2026 is the current stable release. Recent bugfix releases are 2.2.1 (15 May 2026), 2.1.3 (14 June 2026), 2.0.2 (11 May 2026) and 1.20.5 (3 June 2026). The site lists 1.20 as the LTS line. The project’s policy is that the current and previous minor versions get bugfixes, and a minor that is losing support gets one final bugfix release. Around Flink, the Kubernetes Operator 1.16.0 came out on 15 September 2026, Flink CDC 3.7.0 on 10 October 2026, and Flink Agents 0.4.0 on 30 October 2026.
• RisingWave: 3.1.0 on 21 September 2026 (4:30 PM IST on GitHub), with a 3.0.5 bugfix on 12 October 2026. 3.0.0 came on 11 June 2026. The 3.1.0 notes have several breaking changes, including local filesystem connectors being disabled by default and WebSocket ingestion moving behind a licence.
• Arroyo: 0.15.0 on 1 December 2025 (9:42 PM IST on GitHub), described as the first release since the team joined Cloudflare. Commits continue in 2026, including a port to a newer DataFusion version in September and October, with the latest commit on 29 October 2026. There is no newer release.
• Others: Apache Kafka 4.4.0, which includes Kafka Streams, came out on 22 October 2026. Apache Spark 4.2.0 came out on 14 July 2026 and 4.2.1 on 27 October 2026. Bytewax says in its README that it has been community-maintained since May 2025, because the company is no longer commercially viable; its last release is 0.21.1 from November 2024.
• Security notes this year: Flink had CVE-2026-35194 (CVSS 8.1 at NVD), where users who can submit SQL could run code on TaskManagers through SQL code generation; it is fixed in 1.20.4, 2.0.2, 2.1.2 and 2.2.1. RisingWave had CVE-2026-112262 (CVSS 7.1 by VulnCheck), where low-privileged SQL users could read and write server files through local filesystem connectors before 3.1.0. We did not find published advisories for Arroyo.

The first question: what kind of state is this?

Before you compare engines, sort your needs by what the processor has to remember. That decides more than any feature list.

Group 1: no state across events

Validate an order, add the outlet’s city from a lookup, route it to a topic. Each event is handled alone. A consumer with retries and idempotent writes is enough. Adding an engine here buys you a cluster to run and little else.

Group 2: small state per key, short time

“Count orders per SKU per minute”, “same phone, three COD orders in ten minutes”. State is a counter or a short list per key, and it can be thrown away once the window is over. Every engine here handles this well. The hard part is not the counting. It is deciding what to do with late events and when a window is final.

Group 3: joins across streams over time

“Paid order not packed within two hours” needs orders and pack_scans together. The engine must keep unmatched orders for up to two hours, and must know when two hours have passed in event time, not wall clock time. This is where state size, watermarks and timers start to matter.

Group 4: large or long-lived state

“Running stock per SKU since the start of the season”, “de-duplicate payment callbacks over 24 hours”, “customer lifetime value updated on every order”. State can grow with every key you have ever seen. You need state TTL, a state backend that can spill to disk or object storage, and a clear upgrade story, because this state is expensive to rebuild.

A quick test

Write each requirement as one sentence and underline the time words: “per minute”, “within ten minutes”, “within two hours”, “since the season started”. If none of your sentences has a time word, you probably do not need a stream processor. If most of them have short windows, any of the three will do and the choice is about your team and operations. If you have long-lived state or complex timers, Flink’s depth starts to matter.

What each tool is

Apache Flink

Flink is a distributed stream processor with batch as a special case. A job is a graph of operators. A JobManager coordinates the job and its checkpoints, and TaskManagers run the operators in parallel slots. State is kept per key, in memory (the hashmap backend, which is the default), in RocksDB on local disk, or since 2.0 in the ForSt backend, which is built for disaggregated state on remote storage. The 2.0 release notes describe disaggregated state and an asynchronous execution model as the main architectural change, aimed at cloud setups where local disks are small and recovery should not need to download all state first. The notes include Nexmark figures from the project’s own run; treat them as the project’s numbers, not a promise for your job.

You can write Flink jobs in SQL, in the Table API, or in the DataStream API in Java or Python. SQL is the fastest way to start. The DataStream API is where you get timers and custom state for logic SQL cannot express well.

A Flink SQL version of the Kanpur per-minute sales looks like this:

CREATE TABLE orders (
  order_id   STRING,
  sku        STRING,
  qty        INT,
  channel    STRING,
  pay_mode   STRING,
  phone_hash STRING,
  order_ts   TIMESTAMP_LTZ(3),
  WATERMARK FOR order_ts AS order_ts - INTERVAL '2' MINUTE
) WITH (
  'connector' = 'kafka',
  'topic' = 'orders',
  'properties.bootstrap.servers' = 'kafka-1:9092',
  'properties.group.id' = 'flink-live-sales',
  'scan.startup.mode' = 'group-offsets',
  'format' = 'json'
);

SELECT window_start, window_end, channel, sku, SUM(qty) AS units
FROM TABLE(TUMBLE(TABLE orders, DESCRIPTOR(order_ts), INTERVAL '1' MINUTE))
GROUP BY window_start, window_end, channel, sku;

And the COD check can be an OVER aggregation, which counts per phone over the last ten minutes of event time:

SELECT order_id, phone_hash, order_ts,
       COUNT(*) OVER (
         PARTITION BY phone_hash
         ORDER BY order_ts
         RANGE BETWEEN INTERVAL '10' MINUTE PRECEDING AND CURRENT ROW
       ) AS cod_orders_10m
FROM orders
WHERE pay_mode = 'COD';

A few things in Flink that surprise people, all from the 2.3 documentation:

• Checkpointing is off by default. You turn it on with an interval (execution.checkpointing.interval in config.yaml, or enableCheckpointing in code). Without it, a restart loses state.
• The default checkpoint storage is the JobManager’s heap. For production you set execution.checkpointing.dir to a durable filesystem or object store.
• In Flink SQL, table.exec.state.ttl defaults to 0, which means state is never cleaned up. An unbounded GROUP BY phone_hash will remember every phone number forever unless you set a TTL.
• The Kafka sink’s delivery guarantee in the DataStream API defaults to none, and in the SQL connector to at-least-once. Exactly-once needs checkpointing and a transactional ID prefix, and the Kafka transaction.timeout.ms must be larger than the longest checkpoint plus the longest restart, or the documentation warns that data loss may happen.

Flink 2.0 also removed a lot of old surface: the DataSet API, the Scala APIs, the old SourceFunction and SinkFunction interfaces, and per-job mode. Java 17 is the default, Java 8 is gone. And the 2.0 notes say plainly that state compatibility is not guaranteed between 1.x and 2.x. If you are on 1.x with large state, the move to 2.x is a migration project, not an upgrade. That is one reason the 1.20 LTS line exists.

Flink 2.3.0 has two changes worth knowing even if you never read release notes. First, a critical fix (FLINK-35661) for mini-batch aggregation that could silently drop records in some configurations. Second, Flink SQL now fails at planning time when an upsert sink’s primary key does not match the query’s upsert key, unless you add an explicit ON CONFLICT strategy (DO NOTHING, DO ERROR or DO DEDUPLICATE). The notes explain that the old behaviour kept the full history per key, which could grow state without limit. Some queries that ran on 2.2 will refuse to start on 2.3 until you choose. That is a good change, but plan for it.

RisingWave

RisingWave calls itself a streaming database. You connect with psql or any PostgreSQL driver, create sources and tables that read from Kafka and other systems, and create materialized views. A materialized view is kept up to date continuously as events arrive, and you can query it like a table, or push its changes to a sink. In our drill, SELECT version() returned “PostgreSQL 13.14.0-RisingWave-3.1.0”, which tells you what it emulates on the wire. It is PostgreSQL-compatible for clients, not PostgreSQL inside.

The architecture has four kinds of nodes: frontend (serving), compute (streaming), meta (scheduling, checkpoints, recovery) and compactor. All persistent data, including streaming state, lives in object storage such as S3, in an LSM-tree store called Hummock. Local disk is a cache. For a small setup there is a single_node mode that runs everything in one process with a local directory as the store, which is what we used in the drill.

The same Kanpur per-minute view in RisingWave:

CREATE SOURCE orders_src (
  order_id   varchar,
  sku        varchar,
  qty        int,
  channel    varchar,
  pay_mode   varchar,
  phone_hash varchar,
  order_ts   timestamptz,
  WATERMARK FOR order_ts AS order_ts - INTERVAL '2 minutes'
) WITH (
  connector = 'kafka',
  topic = 'orders',
  properties.bootstrap.server = 'kafka-1:9092',
  scan.startup.mode = 'earliest'
) FORMAT PLAIN ENCODE JSON;

CREATE MATERIALIZED VIEW sales_per_min AS
SELECT window_start, channel, sku, SUM(qty) AS units
FROM TUMBLE(orders_src, order_ts, INTERVAL '1 minute')
GROUP BY window_start, channel, sku;

-- final results only, emitted after the watermark passes the window end
CREATE MATERIALIZED VIEW sales_per_min_final AS
SELECT window_start, channel, sku, SUM(qty) AS units
FROM TUMBLE(orders_src, order_ts, INTERVAL '1 minute')
GROUP BY window_start, channel, sku
EMIT ON WINDOW CLOSE;

Two details from the documentation matter here. By default, RisingWave emits on update: the view changes with every barrier (every second by default), so you see partial windows as they fill. With EMIT ON WINDOW CLOSE, it waits until the watermark passes the end of the window and emits a final result. And events whose timestamp is older than the current watermark are treated as late and are not processed. In our drill, a late row was dropped without any error to the client.

On licensing: the RisingWave documentation says the Community Edition stays under Apache-2.0, and that self-managed premium features are unlocked with a purchased licence key in the same binaries. The premium list includes items like elastic disk cache, time travel queries, secret management, resource groups, automatic schema change for CDC sources, a set of sinks (Snowflake, BigQuery, DynamoDB, OpenSearch and others), the SQL Server CDC source and WebSocket ingestion. There is a free trial of paid features up to 4 RWUs (4 CPU cores and 16 GiB). The vendor says the elastic disk cache can cut S3 costs by up to 90%; that is the vendor’s claim. The point is not that this is wrong. Open core is a normal model. The point is that the line moves: in 3.1.0, WebSocket ingestion moved behind the licence, and the since_timestamp option on a new feature needs a premium licence. Check every feature you plan to rely on against the current list.

Arroyo

Arroyo is a stream processing engine written in Rust on top of Apache Arrow and DataFusion. It is SQL-first: you define sources, sinks and a query, and Arroyo plans a distributed dataflow with windows, joins and state. It also supports user-defined functions in Rust (including async and aggregate UDFs) and in Python. It has a web UI and an API, and a single arroyo binary can run a pipeline locally with arroyo run, with state and checkpoints in a local directory or object storage. For clusters it can schedule workers as processes or on Kubernetes.

Cloudflare’s blog of 10 April 2025 announced that Cloudflare had acquired Arroyo, that Arroyo was open-sourced in summer 2023, and that it is built in Rust with an object-storage-native state backend. Arroyo now runs inside Cloudflare Pipelines. The 0.15.0 announcement says the release brings improvements from running at scale at Cloudflare and adds an Iceberg sink with two-phase commit, and that Cloudflare Pipelines (then in beta) supported stateless Arroyo pipelines, with more stateful features planned.

The repository has both Apache-2.0 and MIT licence files and describes itself as dual-licensed. GitHub detects it as Apache-2.0.

Here is the query we ran in the drill, against Arroyo’s built-in test source:

CREATE TABLE impulse WITH (
  connector = 'impulse',
  event_rate = '20'
);

SELECT tumble(interval '5 seconds') AS window, count(*) AS n,
       min(counter) AS first_counter, max(counter) AS last_counter
FROM impulse
GROUP BY window;

And a Kafka source for Kanpur would look like this. Arroyo’s DDL takes a WATERMARK FOR clause, and its Kafka connector options are documented on the connector page:

CREATE TABLE orders (
  order_id varchar,
  sku varchar,
  qty int,
  channel varchar,
  order_ts timestamp NOT NULL,
  WATERMARK FOR order_ts AS order_ts - INTERVAL '2 minutes'
) WITH (
  connector = 'kafka',
  format = 'json',
  bootstrap_servers = 'kafka-1:9092',
  topic = 'orders',
  type = 'source',
  'source.offset' = 'earliest',
  'source.read_mode' = 'read_committed'
);

Two documented facts to note. The configuration reference lists 0.0.0.0 as the default bind address for the API (port 5115), controller, compiler, admin and node services, and the default checkpoint URL is /tmp/arroyo/checkpoints with a 10 second interval. Bind to a private address and set a durable checkpoint location before anything else. And the Kafka sink supports exactly-once with Kafka transactions and two-phase commit per checkpoint epoch, but the documentation says the system does not currently recover cleanly from failures during the commit phase.

Kafka Streams, Spark Structured Streaming and Bytewax, briefly

• Kafka Streams is a Java library, not a cluster. It runs inside your application, keeps local state stores backed by changelog topics in Kafka, and supports exactly-once processing within Kafka. If your team writes Java, your inputs and outputs are all Kafka topics, and you want no new infrastructure, it is a strong choice. It is tied to Kafka at both ends, and rebalancing a stateful app can take time while state is restored.
• Spark Structured Streaming fits teams that already run Spark for batch. It uses a micro-batch model by default, with watermarks and a checkpoint location. Latency is usually seconds rather than milliseconds, which is fine for many dashboards.
• Bytewax is a Python dataflow library on a Rust core. Its README now says it is community-maintained after the company stopped in May 2025. We would not start a new production system on it without a plan to maintain it yourself.

Side by side

Licence and governance

Flink is an Apache Software Foundation project under Apache-2.0, with many companies contributing and a predictable release and support policy. RisingWave is Apache-2.0 for the Community Edition, developed mainly by one company, RisingWave Labs, with a premium tier unlocked by a licence key. Arroyo is open source under Apache-2.0 and MIT, now owned and developed by Cloudflare, and its roadmap follows what Cloudflare Pipelines needs.

None of these is a reason to reject a tool. They are reasons to ask different questions. For Flink: which version line will we stay on? For RisingWave: which of our features are premium today, and which might be tomorrow? For Arroyo: are we comfortable with a release cadence that depends on one company’s priorities, and can we build from source if we need a fix?

Programming model: SQL or code

All three support SQL. The difference is what happens when SQL is not enough.

• In Flink, you drop down to the Table API or the DataStream API, with keyed state and timers in Java or Python. Process Table Functions (PTFs), added in recent releases and extended in 2.3, let you write stateful, timer-driven logic and call it from SQL.
• In RisingWave, you stay in SQL and add user-defined functions. The model is “everything is a materialized view”, which is very easy to reason about and to query, but there is no low-level operator API.
• In Arroyo, you stay in SQL and add UDFs in Rust (including async UDFs) or Python.

For Kanpur, three of the four needs are natural SQL. The “not packed within two hours” alert is the interesting one. It is an absence: something that did not happen. In Flink you can write it as a keyed process function with a timer per order, which is the classic way and easy to test. In SQL, you can try a LEFT interval join between orders and pack scans and look for rows where the pack side is null after the bound has passed; whether and when unmatched rows are emitted depends on the engine and its watermark handling, so test it with real data before you rely on it. A simpler and often better option is to let the engine maintain “paid orders without a pack scan” as a view or table and run a small scheduled query against it.

Time, watermarks and late data

This is where the Kanpur outlet uploads matter. All three engines use event time with watermarks. A watermark is the engine’s statement “I do not expect events older than this any more”. It is usually defined as the maximum event time seen minus an allowance, like order_ts - INTERVAL '2' MINUTE.

The trade-off is simple and cannot be avoided. A small allowance gives fast results but drops more late events. A large allowance accepts more late events but delays final results and keeps more state.

What happens to late events differs:

• RisingWave’s documentation says late events are not processed. In our drill, a row with an event time behind the watermark was dropped at the table itself, silently.
• In Flink SQL, window aggregations also drop late rows by default. In the DataStream API you can configure allowed lateness and send late events to a side output, which is useful when you want to count them or correct them later.
• Arroyo uses watermarks the same way; we did not test its late-data handling with a Kafka source.

For Kanpur, the right design is not a huge watermark allowance on one stream. It is two paths. Online orders are near real time, so they get a two-minute allowance. Outlet uploads can be an hour late, so treat them as their own stream with their own watermark (or no streaming window at all), and correct the per-minute outlet chart from a batch query on the warehouse later. Mixing both into one watermark either delays the online numbers or drops the outlet sales.

One more trap: a watermark only moves when events arrive. If an outlet’s partition goes quiet, the watermark for that partition stops, and windows waiting on it do not close. Flink has idleness settings for sources and watermark alignment for skewed sources (2.3 added a buffer for alignment, set by pipeline.watermark-alignment.buffer-size). In our RisingWave drill with very sparse test data, the closed-window view lagged one window behind what we expected, until another event arrived. With real traffic this is less visible, but quiet partitions at night are real.

Windows and when results are emitted

Tumbling windows (fixed, non-overlapping), hopping or sliding windows, and session windows are supported in all three in some form. The bigger question is when you see a result:

• Emit on update: the result row for a window changes as events arrive. Good for a live dashboard. The consumer of the output must handle updates (upserts, or retractions if you read a changelog).
• Emit on close: the result appears once, when the watermark passes the window end. Good for alerts, billing and anything you write to an append-only system.

RisingWave makes this choice explicit with EMIT ON WINDOW CLOSE, and in the drill the two views behaved exactly as documented, apart from the lag noted above. Flink’s window table-valued functions in SQL emit final results when the window closes, while a plain GROUP BY on a time column is an updating aggregate. Arroyo’s tumbling and hopping windows emit when the watermark passes; in the drill, each 5 second window appeared once, after it closed.

For the Kanpur dashboard, emit on update into an upsert store is fine. For the COD alert and the packing alert, you want each alert once, so emit on close, plus an idempotent alert key.

Joins and state size

Stream joins are where state grows quietly.

• A regular join between two streams in Flink SQL keeps all rows from both sides forever, unless you set table.exec.state.ttl. With the default of 0, nothing is ever cleaned.
• An interval join (pack_ts BETWEEN order_ts AND order_ts + INTERVAL '2' HOUR) lets the engine drop rows once the time bound has passed in event time. This is almost always what you want for “within X” logic.
• A temporal or lookup join (enrich an order with the outlet table as it was at order time) keeps one version of the dimension per key, not all history.

RisingWave and Arroyo follow the same ideas: interval joins and watermarks let state be cleaned; unbounded joins keep growing. In the drill, RisingWave’s interval join between orders and dispatches returned exactly the two orders dispatched within ten minutes and excluded the one dispatched after about 19 minutes, and the result survived a restart.

A practical habit: for every stateful query, write down what the key is, how many keys you expect, and when a key’s state can be dropped. If you cannot answer the third question, you have a memory leak with a SQL syntax.

Checkpoints and recovery

All three take periodic consistent snapshots of state together with source offsets, so that after a crash they can rewind the source and continue as if nothing happened.

• Flink uses checkpoint barriers flowing through the job. Aligned checkpoints are simpler; unaligned checkpoints help when back pressure makes alignment slow. Checkpoint storage and the state backend are separate choices. Savepoints are checkpoints you trigger yourself, for upgrades and rescaling. 2.3 added the option to take checkpoints while recovering from unaligned checkpoints, off by default.
• RisingWave uses barriers too, on a fixed interval (one second by default), and commits state to its object store. Recovery is managed by the meta node. You do not manage savepoints yourself; the database is the state.
• Arroyo checkpoints every 10 seconds by default to the configured checkpoint URL (a local path or object storage), with a SQLite or PostgreSQL database for pipeline metadata.

The drill showed what checkpoint recovery means in practice. After a graceful stop, Arroyo took a final checkpoint and, on restart, finished the window that was open with exactly the right counters. After a crash with kill -9, it went back to the last periodic checkpoint and recomputed from there. State inside Arroyo was correct, but the output sink, which was plain stdout, saw one window twice with two different counts. That is not a bug in Arroyo. It is what “exactly-once state” means, and the next section is about why that is not the same as exactly-once output.

Exactly-once end to end

Every engine here can give you exactly-once state. Exactly-once results in your database, topic or alert system need one of these:

1. A transactional sink that commits only when a checkpoint completes. Flink’s Kafka sink does this with Kafka transactions, if you set it to exactly-once, give a transactional ID prefix, and size transaction.timeout.ms correctly. Downstream consumers must then read with isolation.level=read_committed, and they will see results only after each checkpoint, so your checkpoint interval becomes part of your latency. Arroyo’s Kafka sink does two-phase commit too, with the documented caveat about failures during the commit phase.
2. An idempotent sink: upsert by a stable key. For a per-minute count, the key is (window_start, channel, sku). If the engine replays and writes the same window again, the row is overwritten, not added. This works with PostgreSQL, Redis, ClickHouse with the right engine, and most key-value stores. It is the simplest robust choice, and it is the same idea as idempotency keys for payment retries.
3. A database that is the sink: in RisingWave, the materialized view is the result. If your consumers can query RisingWave directly, you avoid the sink problem for those consumers. As soon as you push results to another system, you are back to options 1 and 2.

Our drill made one more point visible. After the crash, Arroyo emitted the window starting at 2:27:55 PM IST a second time with 49 events, after it had already emitted it with 100. With an upsert sink keyed by window, the second write would replace the first, and in a real Kafka replay the recomputed value would be the same as the first because the same offsets are read again. With an append-only sink, you get two rows for one window and a dashboard that does not add up. For alerts, use an alert key such as phone_hash + window_end and let the alert system de-duplicate.

State compatibility and upgrades

This is the part teams discover in the second year.

• Flink: savepoints let you stop, change and restart a job with its state, as long as the state is compatible. Changing a SQL query often changes the operator graph and makes the old state unusable. Upgrades across minor versions are usually fine with savepoints; 1.x to 2.x is explicitly not guaranteed. Plan how you would rebuild state from the topic if you had to: how far back does Kafka retain data, and how long would a replay take?
• RisingWave: you upgrade the database, and materialized views keep their state. The release notes are worth reading line by line. 3.0.5, for example, fixed a sink bug where an update that changed a primary key could leave the old row behind downstream, and noted that upgrading does not remove rows already left stale, so you have to reconcile them yourself. Changing a view usually means creating a new one and backfilling.
• Arroyo: in our drill, starting a changed query (a 10 second window instead of 5) on the same state directory was refused with a clear message that the state is for a different pipeline, unless you pass --force. That is the right default. Treat a query change as a new pipeline with its own state, and backfill.

Operations and footprint

• Flink needs Java, a JobManager and TaskManagers, a checkpoint store, and in practice the Kubernetes Operator. It scales to very large jobs. It also asks you to understand memory configuration, parallelism, back pressure, checkpoint duration and state backends. Budget learning time for at least one person.
• RisingWave in production runs as several services with object storage and a metadata store. It can also start as one process. In the drill, the embedded compactor refused to start when we told it the machine had 2 GiB or 4 GiB of memory, and started at 8 GiB; that was a sizing setting, not actual usage, but it tells you the small single-node mode still expects a decent machine. The printed configuration also had telemetry enabled by default, and setting the RW_TELEMETRY_ENABLED=false environment variable did not stop telemetry attempts in our run, so turn it off in the configuration file if your policy needs it.
• Arroyo was the lightest to start: one binary, a local state directory, no Java and no separate services for a local run. A Kubernetes setup has a controller, workers, a database and object storage. It has the smallest community of the three, so you will read more source code when something is unclear.

None of these engines autoscales stateful jobs as easily as a web service. Changing parallelism means redistributing keyed state. Flink’s operator and adaptive scheduler help, RisingWave lets you change the parallelism of running jobs with ALTER statements, and Arroyo’s options are simpler. Do not attach a CPU-based autoscaler to a stateful job and hope.

Security history

• Flink: CVE-2026-35194 this year allowed users with SQL submission rights to run code on TaskManagers through code generation; it is fixed in 1.20.4, 2.0.2, 2.1.2 and 2.2.1, and 2.3.0 came after those. Older CVEs (2020) were about the REST API reading and writing files. The Kubernetes Operator had CVE-2026-40564 (jar URL not validated), fixed in 1.15.0. The lesson: treat “who can submit SQL or upload a jar” as code execution rights, and keep the REST endpoint and web submission (web.submit.enable is true by default) off public networks.
• RisingWave: CVE-2026-112262 allowed low-privileged SQL users with CREATE on a schema to read and write server files through the local filesystem source and sink before 3.1.0. 3.1.0 disables those connectors by default. The default authentication config we saw in the drill trusted local connections and allowed password logins from any address. Set users, passwords and network rules before you expose port 4566 anywhere.
• Arroyo: no published advisories that we found. Its services bind to all interfaces by default according to the configuration reference, and the local run has no login. Keep it on a private network.

When a consumer is still the right answer

A plain Kafka consumer, in the language your team already uses, is often the better choice when:

• Each event is handled alone: validate, enrich from a cache, write to a database, call an API.
• State is tiny and can live in your database with an upsert, for example “last seen status per order”.
• You need side effects such as sending SMS or calling a courier API, with retries and idempotency. That is closer to a workflow; see the durable execution post.
• The time window is long and the result is a report. A real-time OLAP database that ingests the topic and answers GROUP BY minute at query time is simpler than maintaining windows in a stream engine, and it handles late events naturally because it just stores them.

The Kanpur live-metrics consumer was a stream processor pretending to be a consumer. Some of its jobs should move to an engine. Others should move to the database.

A worked plan for Kanpur

1. Split the requirements by state. The per-SKU recent sales view, the COD check and the packing alert need state over time. The per-minute dashboard can mostly come from the real-time OLAP database that already ingests orders and pos_sales, which also handles the late outlet uploads correctly because they are stored with their own event time.
2. Pick one engine for the stateful three. For a nine-person team with strong SQL and no Flink operator experience in-house except the CTO, RisingWave is the most direct fit: the team can query the views with psql, the packing alert becomes “paid orders with no pack scan, older than two hours” as a view, and the COD check is a hopping window or a self join on the phone hash. If the team wants more control over timers and expects to grow into complex logic, Flink SQL on the Kubernetes Operator is the alternative, with the packing alert as a small DataStream job with a timer. Arroyo is worth a pilot for the per-SKU view because it is light, but the gap since the last release makes it a harder choice for the core path today.
3. Define event time and watermarks per stream. Online orders and payments: two minutes. Pack scans: five minutes, since scanners sometimes sync late. Outlet sales: not part of the real-time path at all.
4. Make every sink idempotent. The per-SKU view writes to Redis with key sku:window_start. Alerts carry alert_type + order_id or phone_hash + window_end as the key, and the alert service ignores repeats.
5. Set state limits before go-live. For each query, write the key, the expected number of keys and the cleanup rule. In Flink, set table.exec.state.ttl or use interval joins. In RisingWave, prefer windowed and interval queries over unbounded joins.
6. Set up checkpoints and storage. Object storage in the same Mumbai region, with lifecycle rules for old checkpoints. Check that Kafka retention is longer than the time you would need to rebuild state from scratch, plus a margin.
7. Lock down access. Private network only, no public web UI, separate SQL users for the team and for the dashboard, and a short list of people who can create sources and sinks.
8. Rehearse failure. Kill a worker during a load test and check the dashboard and alerts for duplicates. Change a query and practise the new-pipeline-plus-backfill path. Do this before the sale, not during it.
9. Retire live-metrics in steps. Keep it running in parallel for two weeks, compare numbers per minute, and switch the dashboard only when the differences are explained.

When each is better

• Flink is better when you need the widest connector set, complex event-time logic with timers, very large state, many teams sharing one platform, or Java and Python APIs beyond SQL. It is also the safest choice for hiring and long-term support.
• RisingWave is better when your team thinks in SQL and PostgreSQL, when you want to query stream results directly without another serving database, and when object-storage-backed state and simpler operations matter more than low-level control.
• Arroyo is better for small, SQL-only pipelines where a single binary and low resource use matter, for teams comfortable with Rust who may read or patch the source, and for people already using Cloudflare Pipelines who want to run the same engine elsewhere.

When not to adopt each

• Do not adopt Flink if nobody on the team can own it. A Flink cluster with default memory settings, checkpointing off and state TTL at zero is worse than the consumer it replaces.
• Do not adopt RisingWave if a feature you need is on the premium list and you have not budgeted for a licence, or if you expect it to behave like PostgreSQL in every detail. It speaks the protocol; it is a different database.
• Do not adopt Arroyo for a critical path if you need regular releases and security fixes on a schedule, or if your connectors are not in its list. Watch the release page and the repository activity before deciding.
• Do not adopt any engine for stateless work, or for reports that a real-time OLAP database can answer at query time.

A short history

Flink grew out of the Stratosphere research project in Berlin, entered the Apache incubator in 2014 and became a top-level project soon after. It became the reference design for event time, watermarks and checkpoint barriers in open source stream processing, and 2.0 in March 2025 was its first major version in about nine years, removing old APIs and moving towards disaggregated state. RisingWave Labs opened the RisingWave repository in January 2022, released 1.0 in July 2023 and 2.0 in October 2024, and reached 3.0 in June 2026, steadily adding premium features alongside the Apache-2.0 core. Arroyo was started as a company, open-sourced in 2023, and joined Cloudflare in April 2025. Kafka Streams and Spark Structured Streaming have been around since 2016, and both are still the default in many shops that already run Kafka or Spark.

Decision guide

Start with the state

• No state across events: consumer.
• Short windows per key: any engine, or an OLAP database if the output is a dashboard.
• Joins over time and absence alerts: an engine with interval joins, and Flink if you need timers.
• Large, long-lived state: Flink or RisingWave, with a tested rebuild plan.

Then match the team

• Java or Python developers who like code: Flink DataStream or Kafka Streams.
• SQL and PostgreSQL people: RisingWave, or Flink SQL.
• A small team that wants one binary and is comfortable reading Rust: Arroyo, with eyes open on releases.

Then check the licence and the vendor

• Flink: Apache governance, many vendors.
• RisingWave: read the premium features page for every feature you plan to use, and recheck at each upgrade.
• Arroyo: open source, one owner, release cadence tied to Cloudflare’s work.

Checklist

• Each stateful query has a written key, expected key count and cleanup rule.
• Event time column and watermark allowance are set per stream, and late events are counted somewhere.
• Emit policy is chosen per output: update for dashboards, close for alerts and append-only sinks.
• Checkpointing is on, stored in durable storage, and the interval is chosen with sink latency in mind.
• Every sink is transactional or idempotent, with a stable key such as window plus key.
• Kafka transaction.timeout.ms is sized above checkpoint plus restart time if you use transactional sinks, and downstream readers use read_committed.
• State TTL is set in Flink SQL, or interval joins are used instead of unbounded joins.
• Kafka retention is long enough to rebuild state from scratch.
• Query changes follow a written path: new pipeline, backfill, compare, switch.
• Web UIs and SQL ports are private, with users and roles; SQL submission is treated as code execution rights.
• You are on a supported version line with the 2026 security fixes.
• Telemetry and default bind addresses are reviewed.
• A failure rehearsal (kill a worker, restart, compare outputs) has been done under load.

Common mistakes

• Using processing time for business metrics. The dashboard looks fine until a replay or a late upload moves numbers into the wrong minute.
• Leaving Flink checkpointing off because the job “worked in testing”. It worked until the first restart.
• Assuming exactly-once means no duplicates downstream. It means consistent state. Duplicates come back through non-transactional, non-idempotent sinks.
• Unbounded joins with no TTL. Memory and checkpoint size grow slowly for weeks, then recovery takes an hour.
• One watermark for streams with very different lateness. You either drop the late stream or delay the fast one.
• Changing a query and reusing the state. Either the engine refuses, or worse, it accepts and the results are subtly wrong. Treat it as a new pipeline.
• Exposing the web UI or SQL port. In Flink, web submission is on by default; in Arroyo, services bind to all interfaces by default; in RisingWave, the default auth config we saw allowed password logins from any address.
• Not reading release notes. Flink 2.3 will refuse some upsert queries until you choose a conflict strategy, and RisingWave 3.1 disables local filesystem connectors and moves WebSocket ingestion behind a licence.
• Choosing by benchmark. Vendor benchmarks are run by the vendor on their chosen queries. Your bottleneck is more likely state size, sink speed or a skewed key.

The drill we ran

The drill was small and local and ran on 30 October 2026 between about 2:25 PM and 2:31 PM IST, inside a user folder. We downloaded release tarballs and checked their SHA-256 sums. We did not install system packages, use Docker or touch system directories. Arroyo was bound to 127.0.0.1, and RisingWave ran inside an unprivileged private network namespace with only loopback. These are results from one run on this box, not a benchmark.

1. Arroyo 0.15.0, normal run. We ran arroyo run with a local state directory and the query shown above: a test source producing 20 events per second, counted in 5 second tumbling windows, printed to stdout. Full windows had 100 events each. On a normal stop (SIGTERM), Arroyo printed that it was stopping with a final checkpoint.
2. Arroyo, graceful restart. On restart with the same state directory, the first window printed was the one that had been open at stop, with 60 events and counters continuing exactly from where the previous run ended. No loss, no duplicate. Windows with no data during the downtime simply did not appear.
3. Arroyo, crash. We started it again and killed it with kill -9 after about 27 seconds. On restart, it restored from the last periodic checkpoint. It printed the window starting at 2:27:55 PM IST a second time, now with 49 events, although the crashed run had already printed it with 100, and it produced the counters after the checkpoint again in later windows. The test source uses wall clock time, so the replay was not identical; with Kafka the same offsets would replay. The lesson stands: a non-idempotent sink sees duplicates after a crash.
4. Arroyo, changed query. We changed the window to 10 seconds and reused the state directory. Arroyo refused: the specified state is for a different pipeline, pass --force to continue. We did not force it.
5. Arroyo snag. Passing the state directory as a file:// URL also created an empty directory literally named file: in the working directory. Harmless, but surprising.
6. RisingWave 3.1.0, setup. We ran risingwave single_node with a local store directory. The binary needed the Python 3.12 shared library, which this box did not have, so we fetched a user-level Python 3.12 runtime into the drill folder. The embedded compactor panicked at start with a memory assertion when we set total memory to 2 GiB or 4 GiB, and started at 8 GiB. The printed config showed telemetry enabled; the RW_TELEMETRY_ENABLED=false environment variable did not stop telemetry attempts, which failed only because there was no network.
7. RisingWave, windows. We created an append-only orders table with a watermark of event time minus 5 seconds, and two per-minute views, one emit on update and one EMIT ON WINDOW CLOSE. After four orders, the update view showed partial windows at once, and the close view was empty, as documented. After a later order moved the watermark past the second window, the close view showed only the first window. After a restart and another order, it showed the second window but not the third. With sparse test data, final results lagged one step behind what we expected. We did not investigate further.
8. RisingWave, late event. We inserted an order with an event time two minutes behind the watermark. It was dropped: six inserts, five rows in the table, views unchanged, no error to the client.
9. RisingWave, interval join. A view joining orders with dispatches within ten minutes returned the two orders dispatched after about 2 and 4 minutes, and excluded the one dispatched after about 19 minutes.
10. RisingWave, restart. After a stop and restart on the same store, tables, views and join results were intact, and new inserts flowed through. The store directory was 1.5 MB.
11. A pgrep check at the end showed no drill process running.

What we did not run: Apache Flink (there is no Java on this box, and we did not fit a JDK and a Flink tarball into the time box), Kafka Streams, Spark, Bytewax, any Kafka broker, transactional Kafka sinks, rescaling, version upgrades, large state or any performance test. A useful next step is the same crash-and-replay test with a local Kafka broker, Flink 2.3 with an exactly-once Kafka sink, and a read_committed consumer counting duplicates.

Unlearn, re-learn

• Unlearn “a consumer with a dictionary is a stream processor”. Re-learn that state, event time and recovery are the actual job.
• Unlearn “exactly-once means no duplicates”. Re-learn that it covers engine state, and that sinks need transactions or idempotent keys.
• Unlearn “late data is rare”. Re-learn to measure it per source and design a separate path for sources that are often late.
• Unlearn “the window closes at the end of the minute”. Re-learn that it closes when the watermark says so, and the watermark moves only with data.
• Unlearn “a SQL change is a small change”. Re-learn that it is often a new pipeline with new state and a backfill.
• Unlearn “open source core means every feature is free”. Re-learn to check the premium list and release notes at each upgrade.
• Unlearn “SQL access is read access”. Re-learn that in stream engines it can mean code execution or file access, and to patch and restrict accordingly.

Sources

• Apache Flink: Flink repository, Downloads and update policy, Flink 2.3.0 release announcement, Flink 2.0.0 release announcement, Flink 2.2.0 release announcement, Checkpointing, Configuration, Table configuration, Kafka DataStream connector, Kafka SQL connector, CVE-2026-35194 at NVD, CVE-2026-40564 at NVD, CVE-2020-17518 at NVD, CVE-2020-17519 at NVD
• RisingWave: RisingWave repository, v3.1.0 release, Release notes, Premium features, Architecture, Watermarks, Emit on window close, CVE-2026-112262 at NVD
• Arroyo: Arroyo repository, v0.15.0 release, Announcing Arroyo 0.15.0, Cloudflare acquires Arroyo (Cloudflare blog), Configuration, SQL DDL, Kafka connector, UDFs
• Others: Apache Kafka downloads, Apache Spark news, Bytewax repository
• Related posts on Revisit.Tech: change data capture in 2026, Kafka and Redpanda, RabbitMQ, NATS and Kafka queues, schema registries in 2026, real-time OLAP in 2026, lakehouse table formats in 2026, idempotency keys, durable execution in 2026, data ingestion in 2026, Kubernetes autoscaling in 2026, MQTT brokers in 2026