Kafka and Redpanda: revisit the choice before you treat them as the same bus

Many teams say “we run Kafka” when they mean “our services talk to a Kafka-compatible broker”. That shortcut was useful when one open source broker dominated the protocol. It is less useful now. Apache Kafka and Redpanda both accept Kafka clients, yet they are different codebases, different consensus stacks, different licences and different day-two jobs.

This post covers history in plain words, where the Kafka API match holds, where it does not, what the licences allow for a working team, and a small drill before you move traffic. The facts come from Apache Kafka release notes and documentation and from Redpanda’s own docs and licence text, checked on 5 October 2026. Nothing here is legal advice. Vendor performance claims are labelled as the vendor’s own.

The short version

• Apache Kafka is an Apache Software Foundation project under Apache License 2.0. Brokers, Connect and tools need Java 17 from Kafka 4.0; clients and Streams need Java 11.
• Kafka 4.0.0, announced on 18 March 2025, runs only in KRaft mode. ZooKeeper mode and ZooKeeper-to-KRaft migration were removed. Kafka 3.9 is the last bridge release for clusters that still need that migration.
• Redpanda is a separate product from Redpanda Data, written in C++, packaged as a single binary, and built around a thread-per-core model (Seastar) with Raft for partition replication. It is not a Java fork of Kafka.
• Redpanda documents Kafka API compatibility for clients built for Kafka protocol 0.11 or later, with a short published exception list (for example one SCRAM mechanism per user, and no server-side KIP-890 Transactions V2).
• Redpanda Community Edition uses Business Source License 1.1. You may use it, including in production for your own workloads, but you may not offer it as a commercial streaming or queuing service to third parties. Each BSL release converts to Apache 2.0 four years after its release date.
• Several Redpanda features, including Tiered Storage, RBAC and Whole Cluster Restore, need an Enterprise licence. Kafka’s tiered storage (KIP-405) reached production-ready status in Apache Kafka 3.9 under the Apache 2.0 project.
• Same client libraries do not mean same upgrade path, same admin tooling or same licence risk for a SaaS team.

Two products, one wire protocol

Apache Kafka started at LinkedIn and became an Apache project. Its core idea is a durable, partitioned, replicated commit log that many producers and consumer groups can share. For years the control plane lived in Apache ZooKeeper. That era is over for current major versions: Kafka 3.9.0 (6 November 2024) was announced as the final major 3.x line with ZooKeeper mode, and Kafka 4.0 made KRaft the only mode.

Redpanda takes a different route. Its docs describe an event streaming platform that stores events in topics and speaks the Kafka API to producers and consumers. Internally it uses Raft groups per partition, a controller partition for cluster metadata, and a thread-per-core design so each application thread is pinned to a CPU core. The company versions self-hosted Redpanda Streaming with a year-based scheme; docs checked on 5 October 2026 list the latest Redpanda tag as v26.2.3.

So the useful mental model is not “Redpanda is Kafka with a new coat of paint”. It is “two brokers that can often share clients, with different insides”.

What Kafka API compatible really means

Redpanda’s Kafka Compatibility page says Apache Kafka clients for protocol 0.11 or later work with minimal or no application changes, and that modern clients negotiate protocol versions. It lists validated clients across Java, librdkafka, several Go and Python clients, Rust and Node.js. It also lists exceptions you should read before a migration:

• One SASL/SCRAM mechanism per user (not both SCRAM-SHA-256 and SCRAM-SHA-512 on the same user).
• Redpanda HTTP Proxy does not expose topic and ACL CRUD the way some other REST proxies do.
• Kafka’s request_percentage quota is not supported; Redpanda documents byte-rate and topic-mutation quotas instead.
• Redpanda does not implement the server-side portion of KIP-890 (Transactions Server-Side Defense). Kafka 4.x clients that expect Transactions V2 fall back to the earlier transaction protocol when talking to Redpanda.

That list is short, and it is also the point: “compatible” is a contract with edges. MirrorMaker, Connect plugins, schema registry setups, exactly-once paths and admin automation still need a real test against the broker you will run.

On the Kafka side, 4.0 itself moved the protocol forward. KIP-848’s next-generation consumer rebalance protocol is generally available and enabled by default on the server; consumers opt in with group.protocol=consumer. KIP-890’s second phase completed on Kafka. KIP-932 Queues for Kafka (share groups) is early access. If your plan is “point the same jars at Redpanda”, confirm which of these KIPs your clients actually need.

Licence: Apache 2.0 versus BSL 1.1

Apache Kafka’s licence is Apache 2.0. For most product teams that means a familiar open source baseline.

Redpanda Community Edition uses Business Source License 1.1. The Additional Use Grant in Redpanda’s licenses/bsl.md allows use of the Licensed Work, with one clear limit: you may not use it for a “Streaming or Queuing Service”. That term means a commercial offering that lets third parties (other than your employees and individual contractors) cause topic creation on the Licensed Work. Cloud, hosting and similar providers that would offer Redpanda as part of a paid service to customers are called out. The Change License is Apache 2.0, four years from each release date.

Redpanda’s licensing overview restates the same Community rules and describes Enterprise Edition under the Redpanda Community License, with a trial path and a feature table. Tiered Storage, RBAC, Remote Read Replicas, Whole Cluster Restore, Iceberg Topics and several other items are Enterprise features. After licence expiry, Redpanda documents restricted behaviour per feature (for example, you cannot create or modify topics to enable Tiered Storage) while the cluster keeps running without data loss.

In plain words for an internal platform team: BSL is often fine for running your own bus. It becomes a legal conversation if you sell a managed streaming service built on Community Redpanda. Kafka Apache 2.0 does not carry that competitive-use restriction. Feature licensing is a second axis: long retention on object storage may be free in one stack and an Enterprise flag in the other.

Architecture and day-two operations

Kafka after ZooKeeper

KRaft keeps metadata in a Raft quorum of controllers inside Kafka. Operators no longer run a separate ZooKeeper ensemble on current majors, but they do run controllers, manage metadata.version, and follow the published upgrade rules. Kafka 4.0 also raised language baselines (Java 17 on brokers, Connect and tools) and removed old client protocol API versions (KIP-896), so upgrades are staged: check broker and client floors before you jump.

Tiered storage in Apache Kafka (KIP-405) is production-ready from 3.9, with later improvements for disablement, quotas and offset visibility. Kafka Streams and Connect remain first-class parts of the same project, which matters if your pipeline is not only produce and consume.

Redpanda’s single binary path

Redpanda stresses one binary, no JVM, no ZooKeeper, Raft replication, DMA and XFS-oriented tuning, and automatic hardware tuners. Partitions are Raft groups with a leader and followers; commits wait for a majority when producers use acks=all. Tiered Storage can offload segments to object storage, but as noted above it sits behind Enterprise licensing on Redpanda.

Operationally, fewer moving parts can mean a smaller alert surface. It does not remove the need for disk, network, partition count, consumer lag and ACL discipline. It also does not give you a magic live converter from a Kafka KRaft cluster. Plan for dual-write, MirrorMaker-style replication, or a staged consumer cutover, and practise offset and schema behaviour on a non-production topic first.

Performance claims

Redpanda’s introduction page says the C++ design can deliver greater throughput and up to 10x lower p99 latency than other platforms. That is the vendor’s own claim. Treat it like any vendor benchmark: ask for the method, machine shape, client settings, durability settings (acks, fsync policy), partition count and whether the run matches your workload. Do not paste a marketing multiple into a capacity plan.

A decision guide

Write answers to these in one meeting:

1. Do you need Apache 2.0 only, or is BSL acceptable for internal production use after legal review?
2. Will any product you sell let customers create topics on your brokers? If yes, read the Redpanda BSL Additional Use Grant with counsel.
3. Do you depend on Kafka Streams, Connect, MirrorMaker 2 or KIPs that Redpanda lists as exceptions or does not implement the same way?
4. Do you need Tiered Storage or cluster restore features, and under which licence do they ship in each product?
5. Who on the team can operate a JVM+KRaft control plane versus a C++ single-binary cluster?
6. What is the client language mix, and have you checked Redpanda’s validated client list plus your exact versions?
7. What is the cutover plan for offsets, schemas and exactly-once producers?
8. Who owns upgrades and support: Apache community plus your own on-call, or a vendor relationship?

Practice drill: prove the client path

Unlearn two ideas first: “Kafka-compatible means identical” and “a licence difference never matters for internal use”. Then, in a sandbox:

1. Run a small Apache Kafka 4.x KRaft cluster and a small Redpanda cluster from current stable docs.
2. Point one producer and one consumer (your real client library and version) at Kafka. Produce a keyed topic with a known partition count. Note offsets and payload hashes.
3. Point the same binaries at Redpanda by changing bootstrap servers only. Repeat produce and consume. Confirm ordering per key, compression, and any transactional path you actually use.
4. Exercise admin paths you rely on: topic create, ACL or SASL user create, consumer group describe, and delete. Stop if an exception from Redpanda’s compatibility list blocks you; decide whether that path is required.
5. If you use schema registry or Connect, run one connector and one schema evolve against both brokers.
6. Measure what you care about under your own load script: p99 produce latency, consumer lag recovery after a kill -9 of one broker, and disk growth for a day’s retention. Label the numbers as your run, not a vendor chart.
7. Draft the cutover: freeze producers, drain consumers, or dual-write, and write the rollback that puts bootstrap servers back on Kafka.

The drill is the decision. A wiki table is not.

Sources

• Apache Kafka 4.0.0 Release Announcement (18 March 2025)
• Apache Kafka 3.9.0 Release Announcement (6 November 2024)
• Apache Kafka documentation: KRaft (current docs tree)
• Redpanda docs: Introduction to Redpanda
• Redpanda docs: How Redpanda Works
• Redpanda docs: Kafka Compatibility
• Redpanda docs: Licenses and Enterprise Features
• Redpanda BSL 1.1 text in the redpanda repository