Distributed coordination in 2026: revisit etcd, ZooKeeper and Consul before you write another leader election

Most teams do not decide to build a distributed coordination system. It creeps in. A scheduled job runs on one VM. Someone adds a second VM for safety, and now the job runs twice, so a developer adds a “lock” row in a table. A month later the lock row is stuck because the process holding it crashed, so a timeout is added. Then the timeout is shorter than a slow run, and two copies run again. Meanwhile another team has a Kafka cluster that brought its own ZooKeeper ensemble, the database failover tool wants a “DCS”, and an old Consul cluster from an experiment three years ago is still answering DNS queries for a few services. Nobody owns any of it.

A coordination store is the small, strongly consistent system that a group of processes uses to agree on a few facts: who is the leader right now, who holds this lock, which configuration version is current, which instances of a service are alive. It is not a database for your business data. It stores a few kilobytes to a few gigabytes, it cares far more about being correct than being fast, and it pays for correctness with a quorum of members that must agree on every write.

This post revisits the three open-source names that most teams meet: etcd, the Raft-based key value store under every Kubernetes cluster; Apache ZooKeeper, the older Java coordination service behind Hadoop-era systems and many Kafka clusters; and HashiCorp Consul, the service discovery and service mesh tool that also has a strongly consistent key value store, sessions and locks. We will also look briefly at the options you may already have and may not need to replace: PostgreSQL advisory locks and lease rows, Kubernetes Lease objects, and embeddable Raft libraries. The aim is not to pick a winner. The aim is to help you see what each tool is built for, what it costs to run, and what no tool can do for you, which is make a lock safe on its own.

If you are new to this series, a few earlier posts connect directly. Database failover with Patroni, which needs exactly this kind of store, is covered in PostgreSQL high availability. If what you really want is “run this job exactly once”, read background jobs on PostgreSQL and durable execution before you reach for a lock service. Retries that must not repeat side effects are the subject of idempotency keys. Service discovery overlaps with DNS servers in 2026 and service mesh in 2026. The ZooKeeper story for Kafka users is in Kafka and Redpanda, and the HashiCorp licence story for secrets is in Vault and OpenBao.

A scenario to keep us honest

The company in this section is invented. Any match with a real company is an accident.

Vaigai Rural Credit is a microfinance lender based in Madurai with about 600 staff, most of them loan officers in the field, and a technology team of 18. Its systems grew in layers:

  • A loan servicing application on a small k3s cluster, used by loan officers through a mobile app.
  • A PostgreSQL primary and two replicas on VMs, managed by Patroni, which uses a three-node Consul cluster as its DCS. That Consul cluster is on version 1.16, the last release before the licence change, because nobody wanted to think about the licence.
  • A Kafka cluster on version 3.6 with a three-node ZooKeeper ensemble, used for repayment events and SMS reminders.
  • A daily job that builds the NACH debit file for the sponsor bank: one file per day, listing every EMI to be collected. It runs from cron on two VMs, guarded by a “lock” row in PostgreSQL with a 30 minute timeout.
  • A handful of services that find each other through Consul DNS names, and a few feature toggles stored as Consul keys.

In June, the NACH job ran long because the database was slow after a replica rebuild. At minute 31, the second VM saw an “expired” lock row, took it, and generated a second file. Both files reached the bank’s upload folder. The bank’s own duplicate check caught most of it, but some borrowers saw two debit attempts in their passbook, and the field staff spent a week explaining. The postmortem asked the obvious question: “Should we run a proper lock service?” The team lead, Meenakshi, asked a better one: “Which of our three coordination systems should we keep, and would a proper lock service have stopped this?”

We will come back to Vaigai at the end with a worked plan. The short answer to her second question is no, not by itself. That is the most important idea in this post.

The short version

  • A coordination store gives you agreement, not safety. Leases expire on the server’s clock while your process keeps working. Every lock that protects a side effect needs a fencing token that the protected resource checks, or a design where duplicate work is harmless.
  • etcd is the default choice in 2026 for a general coordination store if you run nothing else. It is small, Apache 2.0, a graduated CNCF project with three supported release lines, and its API (leases, transactions on revisions, watches, lock and election helpers) maps cleanly onto coordination problems.
  • ZooKeeper is still sound, still maintained by the Apache Software Foundation, and still the right answer where the software you run expects it (HBase, Solr, older Kafka, some Hadoop tools) or where your team already writes Java with Apache Curator. Do not start a new ZooKeeper ensemble only for locks in 2026 unless one of those reasons applies. Kafka 4.0 removed ZooKeeper mode in March 2025, so many ensembles exist only for an old Kafka.
  • Consul is a service discovery and service mesh product first, with a good key value store and sessions on the side. It is a fine choice if you need DNS-based discovery across VMs and data centres. Since 1.17 it is under the Business Source License 1.1, owned by IBM, and there is no credible open-source fork. Internal use is allowed under its terms, but decide that knowingly.
  • Before any of these, check what you already have. A Kubernetes cluster already has Lease objects backed by etcd. A PostgreSQL database already has advisory locks and can hold a lease row with an epoch counter. For many teams those are enough.
  • Running a quorum store is an operations job: three or five members, fast disks, snapshots you have restored, compaction for etcd, log purging for ZooKeeper, planned upgrades. Budget for it or do not run one.

Where things stand on 31 October 2026

etcd:

  • The latest releases are v3.7.3, v3.6.16 and v3.5.35, all published on 22 October 2026. Three release lines are supported: 3.7, 3.6 and 3.5. The 3.4 line received its final update on 1 June 2026.
  • etcd v3.7.0 came out on 8 July 2026. It removes all the old experimental flags, the v2 discovery service, the v2 client and v2 requests. It adds a streaming range read (RangeStream, which Kubernetes is preparing to use behind a feature gate), a keys-only range optimisation, prioritisation of lease revokes, and a faster lease keep-alive path. The v2 snapshot files and the --snapshot-count flag still exist in 3.7 and are planned for removal in 3.8. Container images are now multi-architecture only.
  • etcd v3.6.0 (15 May 2025) was the first release with full downgrade support, added Kubernetes-style feature gates and /livez and /readyz endpoints, and moved offline tools such as snapshot restore and defragmentation into etcdutl. The project reported at least 50 percent lower memory use and about 10 percent better throughput, from its own benchmarks. You must be on 3.5.20 or later before moving to 3.6.
  • etcd is now maintained as Kubernetes SIG etcd, and the Raft implementation lives in its own repository, etcd-io/raft, released as v3.7.0 alongside etcd 3.7.
  • Six security advisories were published in 2026, rated medium or low. Most concern the built-in authorisation checks, for example CVE-2026-33413 (bypasses in multiple APIs, March) and CVE-2026-73499 (a watch authorisation bypass, July); another, CVE-2026-73500, covers unbounded goroutines during TLS handshakes. If you rely on etcd’s own users and roles, stay on current patch releases.

Apache ZooKeeper:

  • The download page lists 3.9.6 as current and 3.8.7 as stable, both released on 3 September 2026. The 3.7 line has been end of life since 2 February 2024. The project supports two release lines at a time and retires the older stable line about six months after a new minor version.
  • Release votes for 3.9.7 and 3.8.8 passed on the developer list on 30 October 2026. On the day of writing, the download page still showed 3.9.6, so treat 3.9.7 as “about to be announced” and check before you plan an upgrade. The main branch is working towards 3.10.0.
  • The 3.9.6 and 3.8.7 releases fixed two issues the project rated critical: CVE-2026-79993, where the deleteContainer operation skipped the ACL check and allowed unauthenticated deletion of empty znodes, and CVE-2026-59739, an incomplete earlier fix that left a leak in the SetWatches path on reconnect. Earlier, 3.9.5 fixed a reverse-DNS fallback in hostname verification, and 3.9.4 (2025) a missing permission check on the AdminServer snapshot and restore commands.
  • The project’s security page is blunt: security features are opt-in, ZooKeeper is meant to run on a trusted network, and new znodes default to an open ACL.

HashiCorp Consul:

  • The latest release is Consul 2.1.0. Its changelog is dated 21 October 2026, and the GitHub release went out on 22 October. Consul 2.0.0 came out in May 2026, and the 2.0 line reached 2.0.4 on 10 September with several security fixes.
  • From 2.0, Consul follows IBM’s Version, Modification, Fix numbering: a new version each April, a modification each October, and monthly fixes. Support for the 1.22 line ends on 31 October 2026, which is today. The 1.21 line is a long-term support release until 30 April 2027, and the 2.0 line is supported until 30 April 2028.
  • Consul 2.1.0 moves transparent proxy traffic redirection from iptables to nftables, which is a breaking change that needs the nft tool and Linux 5.2 or later. It also bumps the Raft libraries to hashicorp/raft v1.8.0 and raft-wal v0.5.0. Several of the new features in 2.0 and 2.1, including the AI-related roles in 2.1, are Enterprise only.
  • Consul 2.0.4 fixed a pre-authentication RPC memory exhaustion issue and an intention bypass through regex injection, among others, and made mesh:write mandatory for Lua and Wasm Envoy extensions, a breaking change for some setups.
  • Licence: Consul moved to the Business Source License 1.1 in August 2023, effective from Consul 1.17.0. IBM completed its acquisition of HashiCorp on 27 February 2025 and is now the licensor. The api and sdk Go modules remain under MPL 2.0. I found no maintained open-source fork. A former Consul lead maintainer asked on Hacker News in August 2025 whether there was interest in one and concluded, in his own reply, that there did not seem to be.

Around them:

  • Kubernetes Lease objects (coordination.k8s.io/v1) are the standard leader election mechanism for controllers. Coordinated Leader Election, which adds a LeaseCandidate object so the control plane can choose leaders with version awareness, has been beta since Kubernetes 1.33 and is still off by default.
  • Apache Kafka 4.0.0, released in March 2025, removed ZooKeeper mode completely. Kafka 4.4.0 came out on 22 October 2026. A cluster still on ZooKeeper has to move to KRaft on the 3.x line before it can go to 4.x.
  • PostgreSQL 18 is the current major version. Patroni, at 4.1.5, still supports etcd, ZooKeeper, Consul and the Kubernetes API as its DCS, and asks for three or five DCS nodes.

The first question: do you need a coordination store at all?

Before comparing three quorum systems, be honest about the problem. Most “we need a lock service” conversations are really one of these:

  • “This scheduled job must run once.” That is a job scheduling problem. A job queue on your existing database, with a unique key per run, solves it with less machinery. A durable execution engine solves it with more features.
  • “This side effect must not happen twice.” That is an idempotency problem. A unique constraint, an idempotency key, or a check at the receiver solves it. A lock only reduces the chance of a duplicate; it does not remove it.
  • “Exactly one of these replicas should do the background work.” That is leader election. If you run on Kubernetes, use a Lease object through your framework. If you have one database, a lease row with an epoch works.
  • “Services need to find each other.” That is service discovery. Kubernetes Services, DNS, or your load balancer may already do it.
  • “Several systems must agree on a small piece of state, survive the loss of a machine, and react quickly when it changes.” That is the real coordination store use case, and here etcd, ZooKeeper and Consul earn their keep.

Software you already run may also choose for you: Kubernetes brings etcd, HBase and Solr bring ZooKeeper, Nomad deployments often bring Consul. If one is already run well, the cheapest coordination store is the one you already operate.

What each tool is

etcd

etcd is a key value store with a flat, ordered key space, built on Raft. Every change gets a cluster-wide revision number, and the store keeps old revisions (multi-version concurrency control) until you compact them. The pieces that matter for coordination are:

  • Leases: a lease has a TTL, keys can be attached to it, and when the lease expires or is revoked, all attached keys are deleted together. Clients keep a lease alive with keep-alive requests.
  • Transactions: a “compare, then success ops, else failure ops” request. You can compare a key’s value, its version, its create revision or its mod revision. This is how locks and elections are built safely.
  • Watches: stream every change to a key or prefix from a given revision. Because revisions are kept until compaction, a client that reconnects can resume from where it stopped instead of missing events.
  • Concurrency helpers: the Go client has concurrency.Mutex and concurrency.Election, and etcdctl exposes them as lock and elect.

A small taste with etcdctl (examples, adjust endpoints and keys):

# a lease of 10 seconds, with a key attached
etcdctl lease grant 10
etcdctl put --lease=694d9a2b1c3e5f07 /services/billing/node-a 10.0.4.15:8080

# run a command while holding a lock
etcdctl lock /locks/nach-file -- ./build-nach-file.sh

# campaign for leadership and print the current leader
etcdctl elect /elect/billing-sync pod-a
etcdctl elect -l /elect/billing-sync

# follow configuration changes under a prefix
etcdctl watch --prefix /config/loan-service/

When you hold a lock through the Go client, the lock key’s create revision is your fencing token, and Mutex.IsOwner() gives you a comparison you can add to a transaction so that a write only succeeds while you still own the lock. That is the pattern the Jepsen analysis of etcd recommended, and we come back to it below.

etcd is a single Go binary. Clients talk gRPC on port 2379 by default, with an HTTP gateway for JSON. Peers talk on 2380. Its data is a write-ahead log plus a bbolt database file.

Apache ZooKeeper

ZooKeeper presents a tree of small data nodes called znodes, much like a file system where every path can hold a little data. The pieces that matter are:

  • Sessions: a client connects with a session timeout. If the session expires, the server ends it.
  • Ephemeral znodes: znodes tied to a session. They disappear when the session ends. This is how liveness is expressed: “I am alive” is “my ephemeral node exists”.
  • Sequential znodes: the server appends a monotonically increasing counter to the name. Locks, queues and elections are built from ephemeral sequential nodes: create one, see whether you have the lowest number, and if not, watch the node just before yours.
  • Watches: classic watches fire once and must be set again; persistent and recursive watches were added in later 3.x releases.
  • Versions and zxids: every znode carries version numbers and the transaction id (zxid) that created and last changed it. A znode’s creation zxid can serve as a fencing token.

ZooKeeper does not ship ready-made lock or election APIs on the server. They are “recipes” built in the client. In practice, Java teams use Apache Curator, which implements InterProcessMutex, LeaderLatch, LeaderSelector, service discovery and more. The latest Curator on Maven Central is 5.9.0.

A minimal three-server configuration looks like this (example, not run for this post):

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/lib/zookeeper/data
dataLogDir=/var/lib/zookeeper/txlog
clientPort=2181
autopurge.snapRetainCount=5
autopurge.purgeInterval=24
4lw.commands.whitelist=srvr,ruok,mntr
admin.serverAddress=127.0.0.1
server.1=zk1.internal:2888:3888
server.2=zk2.internal:2888:3888
server.3=zk3.internal:2888:3888

Two lines in that file deserve attention. dataLogDir puts the transaction log on its own device, which the admin guide strongly recommends for stable latency. The autopurge lines turn on automatic cleanup of old snapshots and logs, which is not enabled unless you configure it. The admin.serverAddress line keeps the embedded AdminServer, which listens on all interfaces at port 8080 by default, off the network.

HashiCorp Consul

Consul is several things in one binary: a service catalog with health checks, a DNS interface for that catalog, a key value store, sessions and locks, and a service mesh built on Envoy. Servers (usually three or five per data centre) run Raft through the hashicorp/raft library. Client agents run on every node and use a gossip protocol (Serf) for membership and failure detection.

For coordination, the important pieces are:

  • Sessions: a session ties together a node, a set of health checks, an optional TTL and a lock-delay. If the node fails its gossip checks, a linked health check goes critical, the TTL expires or someone destroys the session, the session is invalidated. The design is openly inspired by Google’s Chubby paper.
  • Lock acquisition on keys: a KV write with ?acquire=<session> succeeds only if no other session holds the key. Each successful acquire increments the key’s LockIndex.
  • Lock-delay: after a session is invalidated, its locks cannot be reacquired for the lock-delay period, 15 seconds by default and up to 60. It gives a possibly still running old holder time to notice and stop. The docs are honest that this is “not a bulletproof method”.
  • Sequencers: the tuple of key, LockIndex and session acts as a sequencer that a downstream system can check, which is the Chubby name for a fencing token.
  • Blocking queries and watches: almost every read endpoint accepts an index parameter and waits for a change, and consul watch wraps this for scripts.

Examples (adjust addresses and names):

# run a command while holding a lock
consul lock -n 1 locks/nach-file ./build-nach-file.sh

# create a session and take a lock by hand
curl -s -X PUT http://127.0.0.1:8500/v1/session/create \
  -d '{"Name":"nach-builder","TTL":"30s","LockDelay":"15s","Behavior":"release"}'
curl -s -X PUT "http://127.0.0.1:8500/v1/kv/locks/nach-file?acquire=<session-id>" -d 'vm-2'

# watch a configuration key
consul watch -type=key -key=config/loan-service/sms-enabled ./reload.sh

# discover a service through DNS
dig @127.0.0.1 -p 8600 billing-api.service.consul SRV

The lighter options you may already have

PostgreSQL advisory locks. pg_try_advisory_lock(key) takes a lock that belongs to the database session; pg_try_advisory_xact_lock(key) takes one that ends with the transaction. They are fast, need no new system, and are released if the connection dies. Their limits: they live on the primary, so they disappear in a failover; session-level locks do not mix well with a pooler in transaction mode, because your session is not really yours; and they carry no fencing token.

A lease row with an epoch. When you need a fencing token in plain SQL, a single row works well:

-- take or renew the lease; returns the epoch only if we won
UPDATE leases
   SET holder = 'vm-2',
       epoch = CASE WHEN holder = 'vm-2' THEN epoch ELSE epoch + 1 END,
       expires_at = now() + interval '60 seconds'
 WHERE name = 'nach-file'
   AND (holder = 'vm-2' OR expires_at < now())
RETURNING epoch;

The epoch increases every time ownership changes. Pass it along with the work and make the final write check it, for example by storing it with the generated file record and rejecting any write with a lower epoch. Because the lease is decided by the database’s own clock (now()), clock skew between your VMs does not matter. This is not a consensus system; it is only as available as your database. For many teams that is exactly the right trade.

Kubernetes Lease objects. Every controller built with client-go or controller-runtime already does leader election through a Lease with a holder identity, a lease duration and a renew time, protected by the API server’s optimistic concurrency. It is the obvious choice for workloads that already run on Kubernetes. Note what the client-go source itself says about its leader election: it “does not guarantee that only one client is acting as a leader (a.k.a. fencing)”. The same rule applies here as everywhere else.

Raft libraries. etcd-io/raft (Apache 2.0, v3.7.0), hashicorp/raft (MPL 2.0, v1.8.0) and Apache Ratis for Java (3.3.1) let you embed consensus in your own service. That suits teams building a database or a single-binary product. For an application team it means owning snapshots, membership changes and compaction yourself, which is what etcd’s maintainers already do for you.

Side by side

Consensus: Raft and ZAB

etcd and Consul use Raft. ZooKeeper uses ZAB, the ZooKeeper Atomic Broadcast protocol. Both are leader-based: one member orders all writes, a write is committed once a majority has it on disk, and a new leader is elected if the old one stops sending heartbeats. Raft was designed to be easy to explain and implement, with terms, a replicated log and randomised election timeouts. ZAB was designed for primary-backup replication of state changes, with epochs, transaction ids (zxids) and a recovery phase in which a new leader brings followers in sync before it accepts new writes.

For a user the differences are small: both need a majority, stall writes during an election and suffer from slow leader disks. The visible differences are in the read model and the API.

Consistency of reads

This is where the three behave differently, and where people get surprised.

  • etcd reads are linearizable by default. A follower that receives a read checks with the leader (ReadIndex) before answering, so you see the latest committed write. You can ask for a serializable read, which any member answers from local state. Serializable reads still work when quorum is lost, which is useful for diagnosis and dangerous for decisions. The Jepsen analysis of etcd 3.4.3 found key value operations to be strict serializable.
  • ZooKeeper writes are linearizable, but reads are served by whichever server the client is connected to, from its local state. The guarantee is sequential consistency per client plus a “single system image”: a client never sees an older view after reconnecting to another server. If you need the latest value, call sync before the read.
  • Consul has three modes for reads. default is strongly consistent in almost all cases, with a small window around leader changes where an old leader may answer. consistent adds a round trip to confirm leadership and has no such window. stale lets any server answer and keeps working without a leader. Consul DNS uses stale by default. Writes are always consistent.

The practical rule: if a read decides whether to do something unsafe, use a linearizable read in etcd, consistent in Consul, or sync then read in ZooKeeper. Better still, make the write itself conditional.

Leases, sessions and locks

  • etcd: leases are server-side TTLs. A lock is a key attached to your lease with the lowest create revision under a prefix. When the lease expires, the key goes, and the next waiter gets the lock. The create revision is a natural fencing token.
  • ZooKeeper: sessions are server-side timeouts. A lock is the lowest ephemeral sequential znode under a parent. When the session expires, the node goes. The creation zxid or the sequence number can serve as a fencing token.
  • Consul: sessions combine health checks, an optional TTL and lock-delay. A lock is a key acquired with a session. LockIndex plus session is the sequencer. Consul’s docs say the TTL is a lower bound: the server will not expire a session early, but may expire it later. In our drill, a 10 second TTL session that was never renewed lasted about 21 seconds.

All three are advisory. None of them stops a process that has lost its lock from continuing to write to your database, your bank’s SFTP folder or your object store.

Fencing: the part every lock needs

Here is the failure that bit Vaigai, in general form. Process A takes a lock with a lease. A pauses: a long garbage collection, a stalled disk, a VM migration, or simply a slow query. The lease expires on the coordination server. Process B takes the lock and starts work. A wakes up, still believing it holds the lock, and finishes its write. Two writers, one lock.

No coordination store can prevent this on its own, because the store cannot reach into process A and stop it. The Jepsen analysis of etcd 3.4.3 put it strongly: etcd locks “are fundamentally unsafe” when used alone, and it observed mutual exclusion violations even in healthy clusters with short TTLs. The fix it recommends is the same one Martin Kleppmann described in 2016 and that Chubby called sequencers:

  • Every time the lock changes hands, the new holder gets a number that is larger than every previous holder’s number.
  • The holder sends that number with every write to the protected resource.
  • The resource rejects any write that carries a smaller number than the largest it has seen.

With etcd, use the lock key’s revision, or put IsOwner() as a condition in the same etcd transaction if the protected state lives in etcd. With ZooKeeper, use the zxid or sequence number of your lock node. With Consul, use the LockIndex. With a lease row, use the epoch. If the protected resource cannot check a token (an SFTP upload to a bank, an SMS gateway, a payment API), you need idempotency instead: a deterministic file name per business date, a unique reference per debit, an idempotency key per request. Then a duplicate becomes a no-op instead of a second debit.

Short TTLs speed up failover and make this failure more frequent; long TTLs do the opposite. Neither removes it.

Watches and configuration

  • etcd watches are streams from a revision. A client can reconnect and resume from the last revision it processed, as long as that revision has not been compacted. If it has, the client gets an error and must reload the full state.
  • ZooKeeper’s classic watches are one-shot triggers that carry no data; you read again and set a new watch. Since 3.9.0 a watch event also carries the zxid that triggered it. Curator’s caches hide most of this.
  • Consul blocking queries return when the index moves past the one you gave. They can wake for unrelated changes, so clients compare values before acting.

For application feature flags, none of these is the best tool; use the coordination store for configuration that infrastructure must agree on, such as which node is primary.

Service discovery

Consul is the only one of the three built for service discovery: a catalog, health checks run by local agents, DNS answers that leave out unhealthy instances, and federation across data centres. ZooKeeper and etcd can store service registrations, and many systems do so through ephemeral nodes or leased keys, but you build the health checking and the client side yourself. On Kubernetes, Services and DNS make all three unnecessary for this purpose. Across a fleet of VMs and several sites, Consul remains the most complete option of the three.

Operations: quorum, disks, compaction and snapshots

Quorum sizing is the same arithmetic for all three. A cluster of n voters needs a majority. Three tolerate one failure; five tolerate two; even numbers add cost without tolerance. etcd suggests five and no more than seven. ZooKeeper’s guide says three is usually enough and five lets you take one down for maintenance and still survive a failure. Consul recommends no more than five voting servers. More members means more acknowledgements per write.

Disk latency is the hidden requirement, because all three fsync the log before acknowledging a write. etcd asks for p99 wal_fsync under 10 ms and p99 backend_commit under 25 ms, and its FAQ says that losing leadership because of a slow disk is intended behaviour. ZooKeeper’s guide strongly recommends a dedicated transaction log device and warns that swapping seriously degrades performance. Never put the coordination store on the same busy disk as your database.

Housekeeping differs:

  • etcd keeps old revisions until compaction. The default quota is 2 GB, with 8 GB as the suggested maximum. At the quota, etcd raises an alarm and refuses writes; the fix is compact, defragment each member one at a time, then disarm the alarm.
  • ZooKeeper keeps all snapshots and transaction logs unless you enable autopurge.
  • Consul compacts its Raft log through snapshots automatically.

Backups:

  • etcd: etcdctl snapshot save from a healthy member, etcdutl snapshot status to check, etcdutl snapshot restore to build a new cluster.
  • ZooKeeper: copy snapshots and logs, or since 3.9.0 use the AdminServer snapshot command, the one that needed a permission fix in 3.9.4. Protect it.
  • Consul: consul snapshot save and restore are in the community edition; the scheduled consul snapshot agent is Enterprise only.

A restore also brings back the locks and leader keys of that moment. Check for them before clients act.

Upgrades

etcd upgrades one minor version and one member at a time, and since 3.6 has a supported downgrade path. Before 3.7, check your startup flags for removed experimental options. ZooKeeper supports rolling upgrades within a line; across lines, read the notes on TLS, authentication and Java versions. Consul upgrades servers first, then clients; the step to 2.1 includes the nftables change. Whatever you run, rehearse on a copy of production data, because coordination stores fail rarely and teams rarely practise recovering them.

Security

  • etcd has built-in users, roles and TLS with client certificates; most 2026 advisories were in that role system. On Kubernetes, only the API server should talk to etcd, over mutual TLS.
  • ZooKeeper’s security is opt-in: open ACLs by default, ACLs not recursive, a trusted network assumed. Use SASL or TLS, set ACLs, limit four-letter words, and bind or disable the AdminServer.
  • Consul ACLs are off until you enable them, as are gossip encryption and TLS. Without ACLs, anyone on the network can write keys, register services and destroy sessions.

Licence and governance

  • etcd: Apache 2.0. Created at CoreOS in 2013, accepted into the CNCF in December 2018, graduated in November 2020, and now maintained as Kubernetes SIG etcd. Its future is tied to Kubernetes, which is a strong guarantee of maintenance.
  • ZooKeeper: Apache 2.0, an Apache Software Foundation top-level project since November 2010, run by its project management committee with public votes for every release.
  • Consul: BUSL 1.1 from 1.17.0, with MPL 2.0 as the change licence after four years. The additional use grant allows production use except for offering a competitive hosted or embedded product. Owned by IBM, with commercial direction set by IBM’s product line. There is no open-source fork comparable to OpenBao for Vault.

For most internal use the licence permits what you do. The real questions are whether your legal team accepts source-available terms, whether you embed Consul in something you sell, and whether you mind new features landing in Enterprise.

Performance

Throughput numbers matter little here. Any of the three, on good disks, handles the write rate of elections, locks and configuration changes many times over. What matters is fsync latency, time to elect a new leader, and how fast clients notice. The etcd project’s 3.6 claims (about 10 percent more throughput, at least half the memory) come from its own runs. We did not benchmark anything. On loopback with default timings, an etcd leader was replaced about 1.4 seconds after being killed; that is a property of a fast local machine, not a promise for your network.

A worked plan for Madurai

Back to Vaigai Rural Credit. Meenakshi’s team lists what really needs agreement: the Patroni leader key (real coordination), Kafka metadata (today in ZooKeeper, replaceable by KRaft), the NACH job (run once per business date, with an external side effect that cannot check a token), discovery for a few VM services, and three feature toggles that change twice a month.

The plan they write down:

  • Fix the NACH job first, with no new system. The file name becomes deterministic per business date and batch, and a nach_runs table gets a unique constraint on the same pair. The job inserts its row before generating anything and uploads only if the insert succeeded; a second upload with the same name replaces rather than adds. The lock row becomes a lease row with an epoch, stored with the run. This alone would have prevented June’s incident.
  • Kafka: migrate to KRaft on the 3.x line, then upgrade to 4.x and retire ZooKeeper. No new use of ZooKeeper meanwhile.
  • Patroni: move the DCS from the old Consul 1.16 cluster to a three-member etcd 3.6 cluster on small SSD-backed VMs, away from the database disks, moving to 3.7 after one more patch release. Test Patroni’s DCS failsafe mode before enabling it.
  • Discovery: VM services go behind the internal load balancer with fixed names in CoreDNS; k3s services already use Kubernetes Services. Toggles move to a config file in Git.
  • Retire Consul once nothing depends on it. Had they needed multi-site discovery for many VM services, they would have upgraded to the 1.21 LTS or 2.x line and accepted the licence knowingly.
  • etcd operations: alerts on wal_fsync and backend_commit p99 and the space alarm, scheduled snapshots copied off the hosts, a quarterly restore test, and one-member-at-a-time defragmentation when the file grows.

The end state is one coordination store, one queue and one database instead of three quorum systems nobody owns, and the job that caused the incident no longer depends on any lock for safety.

When each is better

  • etcd: you want a small, general coordination store with a clean API, revision-based watches that resume after a disconnect, a permissive licence and neutral governance, or your tools (Patroni, the CoreDNS etcd plugin) already support it.
  • ZooKeeper: the software you run requires it (HBase, SolrCloud, older Kafka, some Hadoop components), or your Java team already relies on Apache Curator and a healthy ensemble.
  • Consul: you need health-checked service discovery and DNS across many VMs or data centres, or you already use it for a mesh or with Nomad, and your organisation accepts BUSL 1.1 and IBM as the vendor.
  • PostgreSQL locks or a lease row: one database is already your source of truth, and coordination being unavailable when the database is down is acceptable, which it usually is anyway.
  • Kubernetes Leases: the workload runs on Kubernetes. Use your framework’s leader election and add fencing where it matters.

When not to adopt each

  • Do not adopt etcd as a general key value database. It is built for small values and modest data sizes, with a default quota of 2 GB. Do not share the Kubernetes control plane etcd with your applications.
  • Do not adopt ZooKeeper for a new project that does not need it. You take on a JVM, heap tuning, opt-in security and a shrinking pool of people who know it well, for features etcd also provides.
  • Do not adopt Consul only for a lock or a leader key. You would be running a service discovery and mesh platform, with gossip and agents, to get one feature. Also think twice if you embed infrastructure into a product you sell, because of the licence terms.
  • Do not adopt any of them if the real problem is “run once” or “do not repeat side effects”. Fix those with a job queue and idempotency, as Vaigai did.
  • Do not embed a Raft library unless you are building a data system and have people who will own it for years.

A short history

Google described Chubby, its lock service, in a 2006 paper that introduced sessions, lock-delay and sequencers; later systems borrowed those ideas openly. Yahoo built ZooKeeper as a simpler coordination kernel, moved it to Apache in 2008 under Hadoop, and it became a top-level project in November 2010. The ZAB protocol was described in a 2011 paper.

Raft was published in 2014 by Diego Ongaro and John Ousterhout as a consensus algorithm designed to be understandable. etcd, started at CoreOS in 2013, adopted it, and Kubernetes chose etcd as its store. HashiCorp announced Consul, also on Raft, in April 2014.

Recent years were about consolidation. Kafka replaced ZooKeeper with KRaft and finished the removal in 4.0 in March 2025. etcd became a Kubernetes special interest group and removed the old v2 API across 3.6 and 3.7. HashiCorp moved to BUSL in August 2023, IBM completed its acquisition in February 2025, and Consul 2.0 in May 2026 brought IBM-style version numbers.

Decision guide

You run Kubernetes and need leader election for your own controllers or services

Use Kubernetes Lease objects through your framework. Add fencing for any side effect that matters. Do not add a separate coordination store.

You need a run-once job on VMs and you have PostgreSQL

Use a job queue on PostgreSQL or a lease row with an epoch, plus a unique key per run. Do not add a coordination store for this.

You need a DCS for Patroni or a similar tool and run no other store

Run a three-member etcd cluster on dedicated SSD-backed VMs, on the 3.6 or 3.7 line. Monitor fsync latency and the space quota.

You already run Consul for service discovery

Keep it, upgrade off unsupported lines (1.22 support ends today), enable ACLs and TLS if they are off, and use its KV and sessions for coordination needs that fit. Make a conscious decision on the licence.

You run ZooKeeper only for Kafka

Plan the KRaft migration on Kafka 3.x and retire ZooKeeper after the upgrade to 4.x. Until then, keep ZooKeeper on 3.9.6 or 3.8.7 or later because of the critical fixes.

You run HBase, Solr or another ZooKeeper-native system

Keep ZooKeeper, on a supported line, with ACLs, TLS or SASL, a dedicated transaction log device and autopurge enabled. Use Curator for any new coordination code in Java.

Checklist

  • List every coordination need: leaders, locks, run-once jobs, service discovery, configuration. Write down which system holds each today.
  • For every lock that protects a side effect, write down the fencing token or the idempotency key. If there is neither, fix that first.
  • Use three or five voting members, never an even number, and spread them across failure domains.
  • Put the log on fast, dedicated storage. Alert on etcd wal_fsync and backend_commit p99, or the equivalent latency metrics for ZooKeeper and Consul.
  • For etcd, keep automatic compaction on, watch the database size against the quota, and defragment one member at a time.
  • For ZooKeeper, enable autopurge, set ACLs explicitly, restrict four-letter words and the AdminServer.
  • For Consul, enable ACLs, gossip encryption and TLS, and schedule consul snapshot save yourself if you use the community edition.
  • Take snapshots on a schedule, copy them off the hosts, and restore one into a scratch cluster every quarter.
  • After any restore, check for stale lock and leader keys before letting clients act.
  • Stay on supported lines: etcd 3.5, 3.6 or 3.7; ZooKeeper 3.8 or 3.9; Consul 1.21 LTS or 2.x.
  • Document who owns the cluster. A quorum store without an owner is an outage waiting for a date.

Common mistakes

  • Treating a lock as a guarantee. Without fencing or idempotency, a paused process will one day write after its lease has expired.
  • Using wall-clock timestamps in a lock record written by clients. Let the server decide expiry, as etcd leases, ZooKeeper sessions, Consul sessions and now() in PostgreSQL all do.
  • Running two members “for redundancy”. Two members tolerate zero failures and double the chance of losing quorum.
  • Putting the coordination store on the same disk as a busy database, then wondering why leaders change every afternoon.
  • Forgetting etcd compaction and defragmentation until the space alarm stops all writes.
  • Leaving ZooKeeper’s default open ACLs and AdminServer on all interfaces.
  • Making decisions with stale reads: Consul DNS answers and ZooKeeper reads without sync can be behind.
  • Keeping a ZooKeeper ensemble alive for a Kafka cluster that should have moved to KRaft.
  • Staying on Consul 1.16 or older to avoid the licence question. Those versions no longer get security fixes.

The drill we ran

We ran a small drill on 31 October 2026, about 6:25 to 6:35 AM IST, on one Linux machine, with everything bound to 127.0.0.1 on unusual ports. We downloaded etcd v3.7.3 and Consul 2.1.0 from their official release pages and checked the SHA256 sums (not the GPG signature for Consul). ZooKeeper was not run, because the machine has no Java runtime and we did not install one. Multi-node Consul, TLS, ACLs, disk latency, upgrades and load were not tested. All processes were stopped afterwards and the data removed.

etcd, three members, default timings (100 ms heartbeat, 1 second election timeout):

  • A key on a 10 second lease was gone after 12 seconds.
  • With etcdctl lock, a waiter got the lock about 6 ms after the holder finished. When we killed a holder with SIGKILL (lock TTL 5 seconds), the waiter got it about 4.2 seconds later; in a variant where the holder had started a child process, the child kept running after the lock moved on. The lock moved; the work did not stop. That is the fencing problem in miniature.
  • A holder sent SIGTERM while running a 60 second child kept the lock until the child exited, about 59 seconds.
  • etcdctl elect handed leadership to the waiting candidate when we interrupted the leader.
  • We killed the Raft leader. A new one was elected about 1.4 seconds later; of 60 writes sent with a 2 second timeout, 59 succeeded and one timed out. The election key survived.
  • We then killed the new leader, leaving one member of three. Writes and linearizable reads timed out after 3 seconds; a serializable read still returned the last value. /readyz?verbose showed the linearizable read check failing while /livez stayed healthy. The process holding the election key did not notice anything. After restarting the two members, quorum returned within 3 seconds with the same leader key holder.
  • After compacting at revision 484, reading an older revision returned “required revision has been compacted”. Defragmenting one member shrank its file from about 197 kB to 32 kB; the others stayed at about 200 kB.
  • A snapshot (23 keys, revision 484) restored into a new single-member cluster brought back the old election key and three leases, which vanished a few minutes later when nobody renewed them.

Consul 2.1.0, one dev-mode agent:

  • consul lock handed over about 4 ms after the holder finished.
  • After destroying a session holding a lock with a 15 second lock-delay, no other session could acquire the key for 15 seconds.
  • A 10 second TTL session that was never renewed was invalidated after about 21 seconds, consistent with the docs calling the TTL a lower bound.
  • A service with a passing TTL check appeared in the health API, and a DNS SRV query returned it with the registered port.

The drill shows that the primitives behave as documented and that none of them stops a process that has lost its lock. It says nothing about your disks, network or load.

Unlearn, re-learn

  • Unlearn “we have a lock, so only one process runs”. Re-learn “we have a lock, so usually only one process runs; the resource must reject the other one”.
  • Unlearn “more members means more safety”. Re-learn “three or five, on fast disks, in separate failure domains”.
  • Unlearn “ZooKeeper is what serious distributed systems use”. Re-learn that it is still sound, still maintained, and needed by specific software, and that Kafka no longer needs it.
  • Unlearn “Consul is open source like it used to be”. Re-learn that it is source-available under BUSL 1.1, owned by IBM, and fine for many internal uses if you decide so on purpose.
  • Unlearn “a backup is a file”. Re-learn that a backup is a restore you have performed, and that a restore can bring back old leaders.
  • Unlearn “we need a coordination store”. Re-learn to ask first whether the problem is a job queue, an idempotency key, or a Lease object you already have.

Sources

etcd:

Apache ZooKeeper:

HashiCorp Consul:

Others:

Related posts on Revisit.Tech: PostgreSQL high availability, background jobs on PostgreSQL in 2026, durable execution in 2026, idempotency keys, DNS servers in 2026, service mesh in 2026, Kafka and Redpanda, Vault and OpenBao, lightweight Kubernetes in 2026, Kubernetes backup in 2026, chaos engineering in 2026, SQLite in production in 2026.

comments powered by Disqus

Releted Posts

Kubernetes storage in 2026: revisit Rook-Ceph, Longhorn and OpenEBS before you put another database on a persistent volume

Running stateless services on Kubernetes is routine now. Running state is where the calm ends. The moment a database or a file store asks for a PersistentVolumeClaim, somebody must answer: where do those bytes live, how many copies exist, and what happens when a node dies at 2 AM?

Read more

Kubernetes backup in 2026: revisit Velero, K8up and Kanister before you trust another cluster restore

Most Kubernetes teams can rebuild a Deployment in two minutes. Fewer can say how long it takes to get back the data that Deployment was using.

Read more

Log storage in 2026: revisit Grafana Loki, VictoriaLogs and OpenObserve before you keep another month of logs

Picture a platform team in Pune that runs a B2B lending stack for NBFCs across Maharashtra and Gujarat (an invented example, not a real client).

Read more