Kubernetes pod networking in 2026: revisit Cilium, Calico and Flannel before you pick another CNI plugin

Manoj Shukla
October 25, 2026

Picture a software company in Mangaluru that builds billing and stock software for chemists (an invented example, not a real client). About 1,100 medical shops and 40 small hospital pharmacies across Dakshina Kannada, Udupi, Uttara Kannada and Goa use it for GST billing, batch and expiry tracking, and the Schedule H register. The product runs on a self-managed Kubernetes cluster built with kubeadm on 17 rented bare-metal servers in a colocation facility in Bengaluru: three control plane nodes and fourteen workers, split across two racks, each rack on its own subnet. The platform team has three people. The cluster was set up in 2022 by an engineer who has since left; he followed a tutorial, applied kube-flannel.yml, and it worked from day one.

In September, a hospital group in Udupi sent an auditor before signing a contract. The auditor asked for a shell in a pod in the reports namespace, which runs a third-party PDF library. From there, he connected straight to PostgreSQL in the billing namespace and to Redis in another namespace that held session data. The team was surprised, because Git had 31 NetworkPolicy objects, reviewed and applied by Argo CD, including a default deny for every namespace. kubectl get networkpolicy -A listed all of them. None of them was doing anything. Flannel moves packets between nodes; the flanneld agent does not enforce NetworkPolicy, and Kubernetes itself does not either. The objects were accepted by the API server and then ignored.

This post looks at the three CNI plugins most teams meet first, as of 25 October 2026: Cilium, Calico and Flannel. It covers what each one actually does on a node, how Services are handled (kube-proxy in iptables or nftables mode, or a replacement), where NetworkPolicy is enforced, overlay or native routing, BGP, IP address management, encryption with WireGuard or IPsec, observability, upgrades, and the security advisories of the last year.

The facts come from primary material: project READMEs and documentation, release notes, the Kubernetes documentation, GitHub release and security advisory data, Tigera’s security bulletins and CNCF project pages. Release data was fetched through the GitHub API on 25 October 2026. No hands-on drill was run for this post; see the short note near the end. Earlier Revisit.Tech posts covered service mesh with Istio ambient, Linkerd and Cilium and lightweight Kubernetes with k3s, k0s and Talos Linux. This one stays at layer 3 and 4: pod addresses, routes, Services and policy. It does not repeat the mesh angle.

The short version

A NetworkPolicy is a request, not a control. Kubernetes stores it; your network plugin decides whether anything enforces it. The Kubernetes documentation says plainly that creating a NetworkPolicy without a plugin that implements it has no effect. Test enforcement with a real connection, not with kubectl get.
Flannel 0.28.10 is a simple, reliable pod network and nothing more. It gives each node a subnet and carries traffic with VXLAN, host-gw, WireGuard or a few other backends. flanneld does not enforce policy. Its Helm chart can add the Kubernetes SIG network policy controller (kube-network-policies) with netpol.enabled, or you can pair it with Calico for policy. That is a fine choice for a lab, an edge box or a small k3s cluster; it is a weak default for a multi-tenant product.
Calico 3.33.0 gives you a choice of dataplanes. iptables (still the operator’s default), nftables (GA since 3.31), eBPF (with its own kube-proxy replacement) and VPP. It has strong BGP roots, its own IPAM with /26 blocks per node, WireGuard encryption, staged policies you can test before enforcing, and since 3.30 an open source flow log viewer called Whisker. It is Apache 2.0, maintained by Tigera, which also sells commercial editions.
Cilium 1.20.4 is eBPF from the ground up. Policy is enforced by eBPF programs using workload identities, Services can be handled entirely without kube-proxy, BGP is a built-in control plane, encryption is WireGuard or IPsec, and Hubble shows you every flow and every policy verdict. It is a CNCF Graduated project (since 11 October 2023). The price is a bigger surface to understand, a kernel floor (5.10 or equivalent) and a steady stream of patch releases you must keep up with.
kube-proxy itself has changed. The nftables mode has been stable since Kubernetes 1.33, the IPVS mode is deprecated since 1.35, and the documentation for 1.37 says iptables is still the default but a future release will switch to nftables. Pin the mode in your config so an upgrade does not change it for you.
For the Mangaluru company, the practical answer is Calico on a new cluster, with policies first staged and watched in Whisker, then enforced; nftables for both Calico and kube-proxy; VXLAN only across racks; and WireGuard between nodes. Cilium would be an equally sound choice if the team wants Hubble and kube-proxy replacement from day one and is ready to own a faster-moving component. Keeping Flannel without any policy engine is the one option that does not answer the auditor.

Where things stand on 25 October 2026

All times are Indian Standard Time; GitHub gives release stamps in UTC, and I converted them.

Cilium

Latest stable: 1.20.4, released with 1.19.10 and 1.18.16 on 23 October 2026 at 8:21 PM. It is an out-of-band security patch that moves to Go 1.26.10 so that the compromised eMudhra intermediate CA certificates are no longer trusted (the release notes cite CVE-2026-111943 and the Go issue). It also fixes a 1.20.3 regression where Gateway API BackendTLSPolicy handling could cause high CPU use and stop traffic.
1.20.3, 1.19.9 and 1.18.15 came out on 15 October 2026. Among the fixes: conntrack garbage collection deleting live entries, which could cascade into NAT mapping deletion, and a bug where Hubble flows could show the wrong policy as the one that allowed or denied a flow.
1.20.0 (29 July 2026, 8:30 PM): support for the upstream Kubernetes ClusterNetworkPolicy with Admin and Baseline tiers, an automatic netkit mode (bpf.datapathMode=auto, falling back to veth; veth stays the default), online migration from cluster-pool to multi-pool IPAM, Gateway API 1.6.1, GoBGP 4.6.1 under the BGP control plane, and a cilium-cni binary that shrank from roughly 77 MB to 16 MB. The release notes ask for action on upgrade if you use legacy mutual authentication, Kafka-aware policies, the cilium.io/v2alpha1 CiliumNodeConfig API or a custom CNI configuration.
Next: 1.21.0-pre.3 (3 October 2026).
Supported lines: the October patches went to 1.20, 1.19 and 1.18. The 1.17 line last got a release on 17 July 2026 (1.17.18).
Kubernetes versions: the 1.20 documentation lists 1.33, 1.34, 1.35 and 1.36 as end-to-end tested. Kubernetes 1.37 is not on that list yet.
Kernel: Linux 5.10 or later, or an equivalent such as 4.18 on RHEL 8.10. Some features need newer kernels; the documentation keeps a list.
Licence and governance: user space is Apache 2.0; the BPF code is dual licensed GPL 2.0 and BSD 2-Clause. CNCF Incubating from 13 October 2021, Graduated on 11 October 2023. Isovalent, the company that started Cilium, is now part of Cisco and sells an enterprise edition.

Calico

Latest: 3.33.0 (2 October 2026, 4:52 AM; the release notes are dated 1 October). Headline items: netkit pod interfaces on Linux 6.7 and later, nftables flowtable offload (off by default), L2 reachability where Felix answers ARP for pod and LoadBalancer IPs on the node subnet, Felix (not BIRD) programming IPIP cluster routes by default, a single quay.io/calico/calico image that replaces the separate images for calicoctl, CNI, kube-controllers, Typha and others, and calico-node helper services merged into one process to save about 200 MB of memory per node.
3.33.0 also tightened requirements: the eBPF dataplane now needs kernel 5.10 or later with BTF/CO-RE, FIPS image variants were removed, and BIRD programming of IPIP routes is deprecated, with removal planned for 3.35.
3.32.0 (1 May 2026, 3:23 AM): ClusterNetworkPolicy support mapped into new kube-admin and kube-baseline tiers, removal of the older AdminNetworkPolicy and BaselineAdminNetworkPolicy (you must rewrite them before upgrading), Maglev load balancing in the eBPF dataplane, KubeVirt live migration, and a tech preview that serves the projectcalico.org/v3 API as plain CRDs without the Calico API server.
3.31.0 (22 October 2025) made the nftables dataplane GA and simplified enabling eBPF on kubeadm-style clusters.
Patch lines: 3.32.2 (30 August 2026) and 3.31.7 (21 August 2026).
Kubernetes versions: Calico 3.33 is tested against 1.35, 1.36 and 1.37.
Licence and governance: Apache 2.0. Calico is not a CNCF project; it is run by Tigera, which sells Calico Enterprise and Calico Cloud on top of the open source core.

Flannel

Latest: 0.28.10 (8 October 2026, 8:45 PM): Go 1.26, Kubernetes client libraries 0.34.10, the default CNI config version moved to 1.0.0, a VXLAN fix to recover from a closed netlink subscription, and many new end-to-end tests (dual-stack, resilience, Service and CoreDNS checks).
0.28.9 (7 August 2026) added readiness (/readyz) and liveness (/healthz) probes for flanneld and clarified the network policy options in the README.
0.28.2 (27 March 2026, 11:11 PM) fixed GHSA-vchx-5pr6-ffx2 in the experimental Extension backend; more in the security section.
nftables: Flannel has had an nftables implementation for its own masquerade and forwarding rules since 0.25.0 (April 2024).
Licence and history: Apache 2.0. Flannel was started by CoreOS in 2014; the old coreos/flannel repository now redirects to flannel-io/flannel. It is not a CNCF project. k3s still uses Flannel as its default pod network, and the Flannel README points to k3s as the easiest way to get it.

Kubernetes and kube-proxy

Kubernetes: 1.37.1 is the latest patch (24 September 2026, 12:43 AM). 1.37.0 came out on 26 August 2026.
kube-proxy modes on Linux: iptables, ipvs and nftables. The documentation says iptables is still the default in 1.37 and that a future version will change the default to nftables, and it advises setting the mode explicitly.
nftables mode: stable since 1.33, needs kernel 5.13 or later.
IPVS mode: deprecated since 1.35; the documentation says it will be disabled by default from 1.40 (re-enable with the KubeProxyIPVS feature gate) and removed in 1.43. nftables is the recommended replacement.
Cluster-wide policy: the SIG Network network-policy-api project released v0.2.0 (21 April 2026), which merged AdminNetworkPolicy and BaselineAdminNetworkPolicy into one ClusterNetworkPolicy CRD at v1alpha2, with a tier field. Both Cilium 1.20 and Calico 3.32 and later implement it. It is still alpha; treat it as such.

First question: what do you need the pod network to do?

Before you compare features, write down the jobs. For the Mangaluru company, the list looks like this:

Connectivity. Every pod gets an address and can reach every other pod and Service, across both racks, with predictable MTU.
Isolation. Namespaces for different products and customers cannot reach each other unless a policy says so. The reports namespace, which runs third-party code, can reach only the services it needs. This is the auditor’s question.
Visibility. When something is blocked or slow, someone can see the flow, its source and destination, and the policy that decided it, without running tcpdump on four nodes.
Services at scale. About 600 Services today. kube-proxy in iptables mode copes, but each sync is slower than it should be.
Encryption on the wire. The colocation provider’s switches are shared infrastructure. Customer data between nodes should be encrypted.
Routing. The provider can peer BGP with the rack routers if asked. Nobody on the team has run BGP before.
Operations. Three people. Upgrades twice a year. No one wants a component that needs weekly care.

Two things fall out of this list. First, item 2 rules out a pod network without a policy engine, and item 3 rules out “add a policy engine and hope”. Second, items 4 to 6 are nice to have, not urgent; the team should not take on BGP and kube-proxy replacement in the same weekend as fixing isolation.

What each one actually is

What a CNI plugin does, in one paragraph

When the kubelet starts a pod, the container runtime creates a network namespace and calls the CNI plugin named in /etc/cni/net.d/ with an ADD command. The plugin creates an interface for the pod (usually a veth pair; netkit on newer kernels), asks an IPAM plugin for an address, sets routes, and returns the result. That is the whole CNI specification. Everything else people mean by “the CNI” is extra: an agent on each node that programs routes to other nodes, a way to carry pod traffic between nodes (overlay or native routing), a policy engine, sometimes a Service load balancer, and sometimes encryption and flow logs. Cilium, Calico and Flannel differ mostly in that extra part.

Flannel

Flannel runs one small agent, flanneld, on each node. It leases a subnet for the node out of a cluster-wide range, stores the lease in the Kubernetes API (the “kube subnet manager”) or in etcd, and sets up a backend to carry traffic between nodes. The pod interface itself is created by the standard bridge CNI plugin with host-local IPAM, which Flannel configures for you. The recommended backend is VXLAN (UDP port 8472 on Linux by default), with an option for direct routes when nodes share a subnet. host-gw writes plain routes to other nodes and needs layer 2 adjacency. WireGuard encrypts between nodes. IPsec (with strongSwan), IPIP and some cloud route backends are listed as experimental. The backend should not be changed at runtime.

What Flannel does not do is the important part. It does not enforce NetworkPolicy, it does not replace kube-proxy, it has no flow logs, and it does not speak BGP. Its README now lists two ways to add policy: the chart’s netpol.enabled option, which runs the Kubernetes SIG kube-network-policies controller next to Flannel (available since 0.25.5), or using Calico for policy (the old “Canal” combination) or Cilium in chaining mode.

Calico

Calico has a few parts. Felix is the agent on each node; it programs the dataplane: routes, policy rules and, in eBPF mode, Service load balancing. BIRD is the BGP daemon that exchanges routes with other nodes and, if you want, with your routers; confd generates its configuration. Typha fans out datastore updates so that hundreds of Felix instances do not all watch the API server directly. The Tigera operator installs and upgrades everything from an Installation resource. calico-kube-controllers cleans up and syncs state. Since 3.30, Goldmane aggregates flow logs and Whisker shows them in a web console.

Calico’s IPAM hands out addresses in blocks, by default 64 addresses (/26 for IPv4, /122 for IPv6), and a node takes more blocks as it needs them. Between nodes, traffic is either routed natively (with BGP distributing routes), or encapsulated with IPIP or VXLAN, either always or only when crossing a subnet boundary (CrossSubnet). Calico’s policy model includes the standard Kubernetes NetworkPolicy, its own namespaced and global policies with explicit order, deny rules and tiers, staged policies, and now ClusterNetworkPolicy.

The distinctive thing is the choice of dataplane: iptables, nftables, eBPF or VPP on Linux, plus Windows. In iptables and nftables modes, Calico enforces policy with netfilter rules and IP sets, and kube-proxy stays in charge of Services. In eBPF mode, Calico attaches programs to interfaces and replaces kube-proxy itself.

Cilium

Cilium runs an agent on each node and an operator per cluster. The agent compiles and loads eBPF programs that sit on every pod interface and on the host’s interfaces. Instead of matching policy on IP addresses directly, Cilium gives each set of pods with the same security-relevant labels a numeric identity, keeps a map from IPs to identities, and checks policy by identity in the kernel. In overlay mode, the source identity travels in the tunnel header.

By default Cilium runs in encapsulation mode (VXLAN on 8472/UDP, or Geneve on 6081/UDP), which needs nothing from the network. Native routing sends pod traffic unencapsulated and expects the network or BGP to know the pod CIDRs. IPAM has several modes: cluster-scope (the default), Kubernetes host-scope, multi-pool, and cloud modes such as AWS ENI, Azure and GKE. The documentation warns not to change IPAM mode on a live cluster except by a documented path; the only online path is cluster-pool to multi-pool.

Cilium can take over Service handling entirely (kube-proxy replacement, using socket-level load balancing and eBPF NAT), implement L3/L4 and L7 policies (the L7 part goes through an Envoy proxy), run a BGP control plane to advertise pod and Service routes, encrypt with WireGuard or IPsec, connect clusters (Cluster Mesh), and expose every flow through Hubble. Note that the Helm chart’s default is kubeProxyReplacement=false; you turn it on deliberately.

Side by side

Licence, governance and backing

Cilium: Apache 2.0 for user space, GPL 2.0 or BSD 2-Clause for BPF code. CNCF Graduated. Large contributor base; Isovalent (Cisco) is the main commercial backer.
Calico: Apache 2.0. Maintained by Tigera; not part of a foundation. Commercial editions add features on top.
Flannel: Apache 2.0. Community maintained under flannel-io, originally from CoreOS. No foundation, no commercial edition. Releases are regular, roughly monthly in 2026.

Dataplane

Flannel: Linux bridge plus kernel routing and the chosen backend; iptables or nftables only for its own masquerade and forwarding rules.
Calico: iptables (operator default), nftables (GA), eBPF, or VPP. You choose per cluster through linuxDataplane in the Installation resource.
Cilium: eBPF only. Since 1.20, netkit can be selected automatically on supported kernels; veth stays the default.

Services and kube-proxy

Flannel: needs kube-proxy. Works with its iptables or nftables mode.
Calico (iptables or nftables): needs kube-proxy. For nftables mode, kube-proxy must also run in nftables mode; Felix’s nftablesMode defaults to Auto, which turns the nftables dataplane on when it detects kube-proxy in nftables mode.
Calico (eBPF): replaces kube-proxy. On kubeadm-style clusters installed with the operator, setting linuxDataplane: BPF with bpfNetworkBootstrap and kubeProxyManagement enabled lets the operator configure the API server address and disable kube-proxy for you. Maglev and direct server return are available.
Cilium: can run beside kube-proxy or replace it. With kubeProxyReplacement=true, you must give it the API server address and port, and the agent refuses to start if the kernel lacks what it needs. Maglev, DSR and topology hints (PreferSameZone, PreferSameNode since 1.20) are available.

NetworkPolicy and beyond

Flannel: none by itself. Add kube-network-policies through the chart, or Calico, or Cilium chaining.
Calico: full Kubernetes NetworkPolicy; Calico policies with order, explicit deny, global policies, host endpoint protection and tiers; staged policies to preview the effect; ClusterNetworkPolicy in the kube-admin and kube-baseline tiers. Application layer policy exists but is off by default (and had a 2026 advisory, below).
Cilium: full Kubernetes NetworkPolicy; CiliumNetworkPolicy and CiliumClusterwideNetworkPolicy with identity-based rules, entity selectors (for example world, host, kube-apiserver, and since 1.20 cluster-mesh), DNS-aware egress (toFQDNs), L7 rules for HTTP and others, host firewall; ClusterNetworkPolicy since 1.20. A policy audit mode lets you see what would be denied.

Routing between nodes

Flannel: VXLAN (with optional direct routing on the same subnet) or host-gw. No BGP.
Calico: native routing with BGP (a full node-to-node mesh by default, or route reflectors, or peering with top-of-rack routers), or IPIP or VXLAN overlays, each with a CrossSubnet option. 3.33 adds L2 reachability by answering ARP on the node subnet.
Cilium: VXLAN or Geneve overlay by default, or native routing. The BGP control plane advertises pod CIDRs and Service IPs to routers. The documentation is clear that it does not program the datapath, so do not use it for reachability inside the cluster.

IP address management

Flannel: one subnet per node from the cluster range (the manifest default is 10.244.0.0/16), allocated by host-local. Simple, but the per-node range is fixed in size.
Calico: blocks of /26 by default, taken on demand, multiple pools, pools per node selector or namespace, and IP reservations. You can change block size, but it is an ordered procedure.
Cilium: cluster-pool by default, multi-pool for several ranges, Kubernetes host-scope, and cloud-native modes (AWS ENI, Azure, GKE). Changing mode later is only supported from cluster-pool to multi-pool.

Encryption

Flannel: WireGuard backend (kernel 5.6 or later, or the out-of-tree module), or the experimental IPsec backend with a pre-shared key.
Calico: WireGuard for IPv4 and IPv6, enabled separately with wireguardEnabled and wireguardEnabledV6 in FelixConfiguration. Same-node pod traffic is not encrypted, and the docs list some platform limits; read them for your environment.
Cilium: WireGuard or IPsec. WireGuard by default encrypts traffic between Cilium-managed pods; node-to-node encryption is a separate beta option. Cluster Mesh with WireGuard needs every cluster to have it on.

Observability

Flannel: nothing built in beyond logs, the new health probes, and what the kernel tells you.
Calico: Prometheus metrics from Felix and Typha; flow logs through Goldmane and the Whisker console, including which policy allowed or denied a flow and how a staged policy would have acted.
Cilium: Hubble. The hubble CLI is inside every agent pod for node-level flows; Hubble Relay gives a cluster-wide (or Cluster Mesh wide) view, and Hubble UI draws a service map. Flows carry identities, verdicts and, increasingly, the policy responsible.

Footprint and requirements

Flannel: one small DaemonSet. Almost any Linux kernel.
Calico: calico-node DaemonSet, Typha, kube-controllers, the operator, and optionally the API server, Goldmane and Whisker. eBPF mode needs kernel 5.10 or later with BTF/CO-RE; nftables mode needs 5.13 or later with nft 1.0.1 or later.
Cilium: agent DaemonSet, operator, optionally Envoy, Hubble Relay and UI. Kernel 5.10 or later or equivalent; more features on newer kernels.

How a packet actually moves

It helps to follow one request: a pod in reports on worker 3 (rack A) calls the billing-api Service, which has a backend on worker 11 (rack B).

1. Leaving the pod. The packet leaves the pod through its veth (or netkit) interface into the host network namespace. This is the first enforcement point. With Calico in iptables or nftables mode, the packet passes through Felix’s chains for that interface, which check egress policy using IP sets. With Calico eBPF or Cilium, an eBPF program attached to the interface makes the decision. With Flannel alone, nothing checks anything.

2. Service translation. The destination is a Service IP, which no interface owns. Something must rewrite it to a backend pod IP. With kube-proxy in iptables mode, the packet walks a chain of rules per Service and per endpoint; in nftables mode, it uses verdict maps, which the Kubernetes documentation describes as faster to update and, at very large scale, faster per packet. With Cilium’s kube-proxy replacement, most of this happens even earlier: socket-level load balancing picks a backend when the application calls connect(), so the packet carries the pod IP from the start. Calico’s eBPF mode does a similar connect-time translation.

3. Crossing the racks. Worker 11 is on another subnet. In an overlay, the packet is wrapped in VXLAN (about 50 bytes of extra headers), sent to worker 11’s node IP, and unwrapped there. This is why pod MTU must be lower than node MTU, and why a mismatch shows up as “small requests work, large responses hang”. With native routing, the packet goes out unchanged and the rack router must know that worker 11 owns that pod range, which is what BGP peering with the router provides. With CrossSubnet (Calico) or Flannel’s DirectRouting, traffic inside a rack is routed natively and only cross-rack traffic is wrapped.

4. Arriving. On worker 11, ingress policy for the backend pod is checked, again by netfilter rules or eBPF depending on the dataplane, and the packet is delivered into the pod.

5. Encryption, if enabled. With WireGuard, the node-to-node leg is sent through a WireGuard interface, encrypted with keys each agent publishes for its node. With IPsec in Cilium, the kernel’s xfrm layer does the work with a shared key you rotate.

Two lessons come out of this walk. First, policy enforcement happens at the pod’s interface on the node, so “which component programs that interface” is the whole question; a NetworkPolicy object is only input. Second, iptables versus nftables versus eBPF matters most for how fast rules update and how they scale, not for whether a single request is fast. At 600 Services, the Mangaluru team will notice update latency and debugging comfort long before they notice per-packet cost.

Worked example for the Mangaluru scenario

The names, ranges and hosts below are invented. Treat the snippets as sketches to adapt, not tested files; field names change between versions, so check them against the documentation for the exact version you install.

Step 1: find out what is really running

Before choosing anything, confirm the facts on the current cluster.

# which CNI config does the kubelet use?
ls /etc/cni/net.d/ && cat /etc/cni/net.d/10-flannel.conflist

# which kube-proxy mode is set?
kubectl -n kube-system get configmap kube-proxy -o yaml | grep -E '^\s*mode:'
# on a node, ask kube-proxy directly
curl -s http://127.0.0.1:10249/proxyMode

# what policies exist, and does anything enforce them?
kubectl get networkpolicy -A | wc -l
kubectl get pods -A -o wide | grep -Ei 'calico|cilium|network-polic|kube-router'

An empty mode: means kube-proxy uses the default, which today is iptables. If the last command returns nothing, no policy engine is running.

Step 2: prove the gap with one test

Run a throwaway pod in reports and try the database. This is the test that should fail after the fix.

kubectl -n reports run nettest --rm -it --restart=Never \
  --image=registry.example.in/tools/netshoot:2026.10 -- \
  sh -c 'nc -zv -w 3 postgres.billing.svc.cluster.local 5432'

Today it connects. Write this command, and two or three like it, into a small script in Git. It becomes the acceptance test for every later step.

Step 3: decide between a stopgap and the real fix

There is a fast stopgap: upgrade Flannel to 0.28.10 using the Helm chart with netpol.enabled=true, which runs kube-network-policies beside flanneld. That makes the existing 31 policies start to bite, all at once, which is exactly the risk: some of them were written two years ago and never tested. If the team takes this route, they must first read every policy and run the acceptance tests in a staging namespace.

The team decides on the real fix instead: a new cluster on Kubernetes 1.36 with Calico 3.33, built beside the old one, with namespaces moved over two weekends. An in-place migration is possible (Calico documents a Flannel migration, and Cilium documents a node-by-node migration with dual overlays), but with only fourteen workers and a load balancer in front, a parallel cluster gives a clean rollback: point DNS back.

Step 4: kube-proxy in nftables mode, pinned

On the new cluster, the kubeadm configuration sets the kube-proxy mode explicitly, so that a future Kubernetes default change cannot surprise anyone.

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: nftables
nodePortAddresses: ["primary"]

Two behaviour changes to know from the Kubernetes documentation: in nftables mode, NodePorts listen only on the node’s primary addresses by default, and kube-proxy does not add accept rules for NodePorts to a local firewall, so the host firewall must allow the NodePort range itself. Ubuntu 24.04’s kernel is well above the 5.13 minimum.

Step 5: install Calico with the operator

helm repo add projectcalico https://docs.tigera.io/calico/charts
helm install calico projectcalico/tigera-operator --version v3.33.0 \
  --namespace tigera-operator --create-namespace

Then the Installation resource:

apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  calicoNetwork:
    linuxDataplane: Nftables
    ipPools:
      - name: pods-v4
        cidr: 10.48.0.0/16
        blockSize: 26
        encapsulation: VXLANCrossSubnet
        natOutgoing: Enabled
        nodeSelector: all()

VXLANCrossSubnet means pods talk natively inside a rack and through VXLAN only between racks. The pod range must not overlap the Service range, the node subnets, or the office VPN ranges; check all three before you type it. Pin the chart version in Git and let Argo CD apply it, like every other component.

Step 6: stage policies, watch, then enforce

Instead of enforcing the 31 old policies blind, the team rewrites them as staged policies first. Calico’s StagedKubernetesNetworkPolicy takes the same spec as a normal NetworkPolicy.

apiVersion: projectcalico.org/v3
kind: StagedKubernetesNetworkPolicy
metadata:
  name: billing-postgres-ingress
  namespace: billing
spec:
  podSelector:
    matchLabels:
      app: postgres
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: billing-api
      ports:
        - protocol: TCP
          port: 5432

Then they open Whisker and watch real traffic for a week:

kubectl port-forward -n calico-system service/whisker 8443:8443

Whisker shows each flow with the policy that allowed or denied it, and how the staged policies would have acted. Anything that would be denied and should not be (a backup job, a metrics scraper, a migration job) gets an explicit rule. When the week is clean, each staged policy is applied as a normal NetworkPolicy, namespace by namespace, starting with reports.

A default deny for each namespace comes last, and it must allow DNS, or every pod in the namespace will fail in a way that looks like an application bug:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-with-dns
  namespace: reports
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Now the acceptance test from Step 2 must fail, and the same test from a billing-api pod must pass. Both go into CI against the cluster.

Step 7: WireGuard between nodes

After policies are stable, encryption is one change:

kubectl patch felixconfiguration default --type=merge \
  -p '{"spec":{"wireguardEnabled":true}}'

The nodes need the WireGuard kernel module (built into Linux 5.6 and later), and UDP 51820 must be open between nodes. WireGuard adds its own header, so recheck MTU after enabling it. Calico’s documentation says same-node pod traffic is not encrypted; for this threat model (shared switches), that is acceptable. Write that decision down. (Encrypting traffic between nodes is a different job from giving engineers a way into those nodes; for that, see the mesh VPN post.)

Step 8: BGP later, not now

Once the team is comfortable, they can ask the provider for BGP peering with the two rack routers, switch the pool to native routing, and drop VXLAN between racks. A sketch:

apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  asNumber: 64512
  nodeToNodeMeshEnabled: false
---
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: rack-a-router
spec:
  peerIP: 192.0.2.1
  asNumber: 64500
  nodeSelector: rack == 'a'

Turning off the node-to-node mesh is only safe when the routers really carry the routes between racks. This is a change to do with the provider’s network engineer on a call, in a maintenance window, with the VXLAN pool ready to switch back.

If the team had chosen Cilium instead

The equivalent first install, with kube-proxy replacement, WireGuard and Hubble, looks like this. Install it on a cluster where kubeadm skipped the kube-proxy add-on.

helm install cilium cilium/cilium --version 1.20.4 \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=10.20.0.10 \
  --set k8sServicePort=6443 \
  --set encryption.enabled=true \
  --set encryption.type=wireguard \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set policyAuditMode=true

policyAuditMode=true plays the role of Calico’s staged policies: policies are evaluated and verdicts appear in Hubble, but nothing is dropped. Then check the status and watch flows:

kubectl -n kube-system exec ds/cilium -- cilium-dbg status | grep KubeProxyReplacement
hubble observe --namespace reports --verdict AUDIT

Switch audit mode off only after the flows look right. The same acceptance tests apply.

What changes for the auditor

On the new cluster, the auditor’s pod in reports can reach the PDF storage bucket and nothing else. The team can show the policy in Git, the failing connection test in CI, and the denied flow in Whisker. That is evidence, not a list of YAML files.

Security and hygiene

The CNI agent is privileged, so treat it like the kernel

Every CNI agent runs with host networking and wide privileges on every node, and in eBPF designs it loads programs into the kernel. A bug in it is a node-level bug, and a policy bug is a silent hole. Keep it on a supported line, read its advisories, and limit who can change its configuration, its CRDs and the node annotations it reads.

Notable advisories

Cilium published several advisories in the past year. The ones most relevant to pod networking:

GHSA-3fcv-jvfp-m4q9 (CVE-2026-49445), critical, 9.2 (1 June 2026): when L7 features are enabled, the Envoy instance created a world-accessible admin socket on nodes, so a local attacker could read TLS secrets or disrupt traffic. Fixed in 1.19.2, 1.18.8 and 1.17.14. No workaround.
GHSA-hxv8-4j4r-cqgv (CVE-2026-33726), medium, 5.4 (24 March 2026): ingress NetworkPolicy was not enforced for same-node traffic to L7 Services when per-endpoint routing was on and BPF host routing off, which is common with AWS ENI IPAM. Fixed in 1.19.2, 1.18.8 and 1.17.14.
GHSA-fm8w-2m5w-9j7r (CVE-2026-56743), medium, 5.4 (6 July 2026): a plain Kubernetes NetworkPolicy with only an ipBlock peer, on a cluster with a custom clusterName, also allowed traffic from every pod in the same namespace. Affects 1.19.0 to 1.19.4; fixed in 1.19.5. A good example of why the acceptance test should try the “should fail” path.
GHSA-vh48-r624-p8v7 (CVE-2026-104149), high, 7.2 (12 August 2026): ingress host and L7 policies bypassed when attaching to a VLAN interface and its parent. The advisory lists fixes in 1.18.9 and 1.17.15, and no fix for 1.16.
GHSA-33qq-jq9c-6gcc (CVE-2026-83620), high, 7.6, and GHSA-xqhm-7xhv-6ppj (CVE-2026-77531), high, 7.3 (both 12 August 2026): identity spoofing in mutual authentication through certificate substitution, and a secret sync name collision that allowed a cross-namespace L7 policy bypass. Both fixed in 1.19.6, 1.18.12 and 1.17.18.
GHSA-gj49-89wh-h4gj (CVE-2026-41520), high, 7.9 (22 April 2026): sensitive information in the cilium-bugtool debug archive. Fixed in 1.19.3, 1.18.9 and 1.17.15. Think before you attach a bugtool archive to a public issue.
GHSA-5r23-prx4-mqg3 (CVE-2026-26963), medium, 6.1 (19 February 2026): host firewall policies did not apply to traffic from pods on other nodes when native routing, WireGuard and node encryption (beta) were all on. Fixed in 1.18.6.

None of these advisories lists a 1.20 version as affected. The 1.20.4 release itself is a security update for the Go toolchain and its CA trust, so run 1.20.4 (or 1.19.10 or 1.18.16 on older lines).

Calico publishes security bulletins on Tigera’s site rather than as GitHub advisories. Six were published in 2026:

TTA-2026-001 (CVE-2026-41184), medium (27 May 2026): the install-cni container logged the live ServiceAccount token in Canal and Flannel-with-Calico style installs, readable by anyone with pods/log in that namespace. Fixed in 3.31.6 and 3.32.0.
TTA-2026-002 (CVE-2026-41185), medium, and TTA-2026-003 (CVE-2026-6720), high, 7.2 (both 27 May 2026): token disclosure in Azure IPAM CNI logs, and calicoctl printing full cluster credentials to stderr at info or debug log level. Fixed in 3.32.0 and later.
TTA-2026-004 (CVE-2026-41186), medium (30 July 2026): when the debug server is enabled (off by default), kube-controllers and Goldmane exposed Go pprof on all interfaces without authentication. Fixed in 3.31.6 and 3.32.1.
TTA-2026-005 (CVE-2026-6540), high, 7.9 (30 July 2026): application layer policy (off by default) did not normalise HTTP paths, so prefix rules could be bypassed with ../ or encoded slashes. Fixed in 3.31.6 and 3.32.0.
TTA-2026-006 (CVE-2026-41187), medium (30 July 2026): users with deletecollection on tier-scoped policy resources could delete policies in tiers they had no rights on. Fixed in 3.31.6 and 3.32.1.

The 3.33.0 notes also include fixes worth knowing even though they are not bulletins: a window in nftables mode where a failure to program the dataplane could leave Calico’s chains missing, so traffic went unfiltered until Felix recovered, and the removal of tokens, kubeconfig contents and keys from log output.

Flannel had one advisory: GHSA-vchx-5pr6-ffx2 (CVE-2026-32241), high, 7.5 (27 March 2026). The experimental Extension backend passed data from a node annotation straight into shell commands, so anyone who could set node annotations could run commands as root on every Flannel node. Only the Extension backend is affected; VXLAN and WireGuard are not. Fixed in 0.28.2.

Habits that matter more than the version number

Test enforcement, not existence. Keep a small set of “this must fail” and “this must pass” connection tests per namespace, and run them after every CNI or Kubernetes upgrade.
Allow DNS explicitly in every default deny, and decide whether egress to the internet is allowed at all.
Guard the inputs. Node annotations, CNI ConfigMaps, the operator’s Installation, Cilium’s Helm values and the policy CRDs are all security-relevant. Restrict who can write them, and who can read pods/log in the CNI’s namespace. An admission policy that blocks changes to these objects outside the GitOps service account helps here; the admission policy post shows how.
Keep debug features off in production: pprof ports, bugtool archives, verbose CLI logging.
Know which policy features you actually use. Several 2026 advisories hit L7 policy, mutual authentication or application layer policy, features many clusters never turn on. If you do not use them, keep them off.
Encryption is not isolation. WireGuard stops someone on the switch; it does nothing about a pod that is allowed to reach the database.

Operational costs to budget for

Upgrades. Cilium’s guide says the only tested upgrade and rollback path is between consecutive minor releases, first to the latest patch of your current line, with a required pre-flight step that pre-pulls images. Plan one minor at a time, and read the upgrade notes for each. Calico upgrades go through the operator and Helm chart; 3.32 required rewriting AdminNetworkPolicy resources before upgrading, and 3.33 changes image names. Flannel upgrades are usually a manifest or chart bump, but the backend must not change at runtime.
Kubernetes compatibility. Check the CNI’s tested Kubernetes versions before each Kubernetes upgrade. Today Cilium 1.20 lists up to 1.36 and Calico 3.33 lists up to 1.37. Upgrade the CNI first when the matrix requires it.
Kernel management. eBPF dataplanes tie features to kernel versions. A mixed fleet (older nodes from a previous purchase, newer ones from this year) can end up with different behaviour per node. Keep kernels uniform.
MTU and firewalls. Overlays, WireGuard and IPsec each reduce MTU, and each needs ports open between nodes (VXLAN 8472/UDP by default, Geneve 6081/UDP, WireGuard 51820/UDP, BGP 179/TCP). Most “the network is flaky” tickets after a CNI change are one of these two.
IP planning. Pod ranges, block sizes and IPAM modes are hard to change later. Pick ranges that will not collide with the office, the VPN or a future second cluster. If you will ever join clusters with Cilium Cluster Mesh or route between them, give each cluster its own range from the start.
Observability storage. Hubble and Whisker are cheap to turn on; keeping flows for a long time is not. Decide how long you need flows (a week for debugging is common) and where long-term copies go. The log storage post covers the storage side.
People. Calico with BGP and Cilium with kube-proxy replacement both need at least one person on the team who can read routes, conntrack tables and eBPF map dumps under pressure. Budget learning time before the incident, not during it.

Decision guide

Keep or choose Flannel if

The cluster is a lab, a CI runner pool, a single-tenant edge site or a small k3s install where every workload trusts every other.
Or you add a policy engine on purpose (the chart’s kube-network-policies option, or Calico for policy) and you test it.
You value a small, stable component over features, and you do not need BGP, flow logs or kube-proxy replacement.

Lean towards Calico if

You want to choose the dataplane: start with nftables (matching kube-proxy), move to eBPF later when you are ready.
You run on bare metal or in a data centre and want BGP with your routers, or you already think in routes and AS numbers.
You want policy order, explicit deny, tiers and staged policies, and a simple flow log console in the open source edition.
You run Windows nodes, OpenStack or KubeVirt VMs alongside pods.

Lean towards Cilium if

You want kube-proxy replacement, eBPF policy and identity-based rules from the start, on kernels you control.
Flow-level visibility matters a lot to your team, and Hubble’s service map and verdicts would change how you debug.
You want DNS-aware egress policies, L7 policy, Cluster Mesh across clusters, or a CNI that also serves Gateway API.
You can keep up with a fast release stream: three supported minor lines, patch releases every few weeks, and a strict one-minor-at-a-time upgrade path.

On a managed cloud cluster

Check what the provider ships and supports first. Several managed offerings use Cilium or Calico underneath, often with their own defaults and their own support boundary. Replacing the provider’s CNI usually means you own the network support yourself.

Checklist

You know which CNI plugin, version and backend each cluster runs, and which component, if any, enforces NetworkPolicy.
A connection test proves that a denied flow is denied and an allowed flow works, and it runs after every CNI or Kubernetes upgrade.
Every namespace has a default deny that still allows DNS, and the reports or third-party namespaces are locked down first.
kube-proxy’s mode is set explicitly, or kube-proxy is replaced on purpose and the replacement is verified on each node.
Pod, Service and node ranges do not overlap with each other or with office, VPN and other cluster ranges.
MTU is set for the overlay and encryption in use, and tested with a large response, not only a ping.
Node-to-node encryption is on where the wire is shared, and the team knows which traffic it does not cover.
The CNI is on a supported line and patch, and someone reads its advisories or bulletins.
CNI configuration, CRDs and node annotations are writable only by the platform team, and debug features are off.
Flow visibility (Hubble or Whisker) is available, and someone has used it once before an incident.
The upgrade path for the next minor version is written down, including the Kubernetes compatibility matrix.

Common mistakes

Applying NetworkPolicy objects on a cluster whose CNI ignores them, and treating kubectl get networkpolicy as proof.
Turning on a policy engine for the first time and enforcing two-year-old, untested policies all at once.
Writing a default deny without a DNS allowance, then debugging “timeouts” in every application.
Leaving kube-proxy’s mode unset and letting a future Kubernetes default change it during an upgrade.
Choosing Cilium’s kube-proxy replacement but leaving kube-proxy running too, or removing kube-proxy without telling the CNI the API server address.
Picking a pod range that overlaps the office network or a VPN, found only when someone connects from home.
Forgetting the MTU cost of VXLAN plus WireGuard, so large responses stall while health checks pass.
Switching off a BGP node mesh before the routers really carry the routes.
Skipping minor versions in a Cilium upgrade, or upgrading Calico to 3.32 with AdminNetworkPolicy objects still in place.
Running the Flannel Extension backend, or any experimental backend, in production because a blog post showed it.

Drill note

No hands-on drill was run for this post. Nothing was installed, and no cluster was built or changed on the box. Every statement here comes from the projects’ documentation, release notes, security advisories and bulletins listed in the sources, checked on 25 October 2026, and the snippets are sketches based on that documentation. Before you change a production CNI, run your own pilot: build a small cluster with your real kernel and node layout, install the plugin, run the connection tests, enforce one namespace, enable encryption, check MTU with a large download, and practise one minor upgrade and one rollback.

Unlearn

Unlearn that “the cluster has NetworkPolicy” means the cluster enforces it. The API accepts objects that nobody implements.
Unlearn that the CNI is a one-time install. It is a privileged, kernel-facing component with its own advisories and upgrade rules.
Unlearn that eBPF automatically means faster for your workload. It mostly means different update costs, different debugging tools and a kernel floor.
Unlearn IPVS as the “fast” kube-proxy mode. It is deprecated; nftables is the planned replacement.

Re-learn

Re-learn the path of a packet: pod interface, Service translation, node-to-node transport, the remote pod interface. Each step has one owner.
Re-learn overlay versus native routing, and what BGP really adds: routers that know where pods live.
Re-learn MTU arithmetic for each header you add.
Re-learn NetworkPolicy semantics: policies are additive allows, a pod is isolated only for the directions a policy selects, and deny rules, order and tiers are extensions from Calico, Cilium and the newer ClusterNetworkPolicy API.

Closing

The Mangaluru incident was not a Flannel bug. Flannel did exactly what it says: it connected the pods. The gap was an assumption that a policy in Git was a policy on the wire. Flannel 0.28.10 remains a good, small pod network for places where isolation is not the job, or where you add a policy engine deliberately. Calico 3.33.0 gives the widest choice of dataplanes, mature BGP, staged policies and a flow viewer, with Tigera behind it and a commercial upsell beside it. Cilium 1.20.4 goes furthest with eBPF: kube-proxy replacement, identity-based policy, Hubble and Cluster Mesh, at the cost of a faster release pace and more to learn. For the Mangaluru company, the sensible path is a new cluster with kube-proxy pinned to nftables, Calico in nftables mode with VXLAN only across racks, policies staged and watched in Whisker before they are enforced, WireGuard between nodes, a connection test in CI, and BGP as a later, separate project. Then the next auditor’s shell in reports gets a timeout, and the team can show why.

Sources

Cilium

Calico

Flannel

Kubernetes

Security records

Internal

comments powered by Disqus

Releted Posts

Service mesh in 2026: revisit Istio ambient, Linkerd and Cilium before you add sidecars

Manoj Shukla
October 8, 2026

A logistics company in Bengaluru runs a production Kubernetes cluster with 14 nodes and about 420 pods. Three requests landed in one sprint.

Manoj Shukla
October 25, 2026

Picture a software company in Rajkot that builds production planning and job work software for foundries, forging units and engineering workshops across Saurashtra (an invented example, not a real client).

Manoj Shukla
October 25, 2026

Take a pump manufacturer in Coimbatore that makes agricultural and domestic pumps at two plants, one near the city and a smaller one at Kaniyur (an invented example, not a real client).