Chaos engineering in 2026: revisit Chaos Mesh, LitmusChaos and Krkn before you break another thing in production on purpose

Manoj Shukla
October 29, 2026

Most teams first meet chaos engineering as a slogan: break things on purpose. Somebody installs a chaos tool, deletes a few pods in staging, sees them come back and declares the platform resilient. Months later a real node dies during a busy hour, the database failover is slower than expected, a retry storm fills the queues, and on-call learns about a dependency that was never on any diagram.

The tool was not the problem in that story. What was missing was a question worth asking, a number for “healthy”, a small blast radius and a habit of writing down what was learnt. This post compares the three open-source chaos tools that Kubernetes teams most often shortlist in 2026: Chaos Mesh, LitmusChaos and Krkn. It also says a word about Chaos Toolkit, Pumba and Toxiproxy, about managed services such as AWS Fault Injection Service and Azure Chaos Studio, and about Gremlin as the well-known commercial option.

Chaos experiments only make sense on top of the rest of your reliability work, so a few earlier posts are useful background. PostgreSQL high availability with Patroni and CloudNativePG explains the failover you will most likely want to test first. Load testing in 2026 matters because a fault under zero traffic proves very little. Uptime monitoring in 2026 covers the outside-in checks that should tell you, during an experiment, that users are hurting. Service mesh in 2026 is relevant because retries, timeouts and circuit breaking are often exactly what a chaos experiment tests. Kubernetes backup in 2026 is the safety net you want in place before any experiment touches stateful workloads, and runtime security in 2026 is worth reading because a chaos tool is, from a security point of view, a privileged agent that changes your system on request.

We end with a small drill that ran a Chaos Toolkit experiment against a tiny loopback web server and checked Chaos Mesh and Litmus YAML against the projects’ own CRD schemas, without any cluster, and a plain list of what we did not run.

A scenario to keep us honest

Imagine a company in Coimbatore that builds software for the textile trade in western Tamil Nadu: production planning for spinning mills, order and shipment tracking for garment exporters, export document preparation, and payment settlement for small job-work units. The company is invented for this post. It has about 60 engineers, a platform team of five, and two people who share on-call duty for the database with their regular work.

The setup today:

One production Kubernetes cluster in a cloud region in Mumbai, with three node pools across three availability zones, and one staging cluster that is smaller but built from the same Helm charts.
About 35 services, mostly Go and Java, with an NGINX ingress controller and Linkerd for mutual TLS, retries and timeouts between services.
PostgreSQL run by CloudNativePG with one primary and two replicas for the orders and settlements databases, Redis for sessions and rate limits, and a managed Kafka service for shipment events.
Prometheus, Alertmanager, Grafana and Loki, and a Blackbox exporter checking the public pages from outside.
Velero backups every night, with one restore test done last year.

Four things push the team to revisit resilience testing. First, during the festival-season export rush a worker node in one zone was replaced by the cloud provider, and the settlements service was down for 40 minutes because both of its replicas were on that node and its PodDisruptionBudget allowed zero available pods. Second, a large European buyer asked, in a supplier audit, for evidence that the company tests failover and recovery, not just that it has backups. Third, a database failover in a staging test took more than four minutes because one service cached its connection pool and never reconnected. Fourth, an engineer once tried to “test resilience” by deleting pods in production with kubectl, and took out a singleton cron controller that nobody remembered.

Engineers have proposed Chaos Mesh (a friend’s company uses its dashboard), LitmusChaos (it has many ready faults and a hub) and Krkn (the platform team wants to test node and zone loss, not only pods). We will come back to Coimbatore in the worked plan.

The short version

Chaos Mesh is a Kubernetes-native chaos platform. Faults are custom resources (PodChaos, NetworkChaos, IOChaos, StressChaos, TimeChaos, HTTPChaos and more), injected by a privileged Chaos Daemon on every node, with a dashboard, schedules, workflows and an HTTP StatusCheck that can abort a workflow. It is a CNCF incubating project under the Apache License 2.0. The latest release is 2.8.7, on 28 October 2026.
LitmusChaos is a chaos platform with a control plane (ChaosCenter web portal, GraphQL server, authentication server and MongoDB) and an execution plane in each target cluster (a subscriber, Argo Workflows, the chaos operator and fault jobs). It has a large catalogue of ready faults, a ChaosHub and the richest resilience probes. It is a CNCF incubating project under the Apache License 2.0, and Harness sells a commercial version. The latest release is 3.33.0, on 15 October 2026.
Krkn is a Python chaos and resiliency testing tool that runs outside the cluster and talks to the Kubernetes API and cloud APIs. It leans towards platform-level failures (nodes, zones, etcd, whole-cluster shutdown, storage) and judges runs with recovery checks, Prometheus SLO profiles and a resiliency score. It came from Red Hat’s OpenShift performance and scale team, is a CNCF sandbox project under the Apache License 2.0, and its latest release is 5.2.12, on 3 October 2026.
Chaos Toolkit is a small, declarative experiment runner in Python, good for experiments that span systems outside Kubernetes. Pumba targets Docker containers and Toxiproxy injects network trouble in tests. AWS Fault Injection Service, Azure Chaos Studio and Gremlin are the managed and commercial contrasts.
The tool matters less than four decisions: what you want to learn, how you measure steady state, how small the blast radius is and who can stop the experiment. A chaos tool without these is just a well-documented way to cause an outage.

Where things stand on 29 October 2026

Chaos Mesh. The latest release is 2.8.7, published on GitHub on 28 October 2026 at 9:40 AM IST. It is a patch release with fixes for JVMChaos, StressChaos and remote clusters, scheduling options for workflow task pods, a move to the Moby SDK for the Docker runtime client (Docker Engine 19.03 or later), and security updates for containerd and dashboard dependencies. Before that, 2.8.6 came on 21 October and 2.8.5 on 16 October 2026; the 2.8.5 notes fixed a dashboard bug where authentication Secret references were overwritten by inline credentials and removed TLS contents from daemon logs. The 2.8.5 notes also say, openly, that the maintainers used AI tools heavily with manual review to cope with a small maintainer team. Chaos Mesh was accepted into the CNCF on 14 July 2020 and moved to incubating on 16 February 2022.

LitmusChaos. The latest release is 3.33.0, published on 15 October 2026 at 3:28 PM IST. It is mostly a hardening release: the infrastructure access key now uses a secure random source, project invitation and statistics calls got authorisation fixes, a custom TLS certificate setting moved into a Secret, and cron schedule changes now reach the subscriber. Litmus releases roughly once a month. The chaos operator and the litmus-go fault library were tagged 3.33.0 on 27 October 2026. Litmus was accepted into the CNCF on 25 June 2020 and moved to incubating on 11 January 2022.

Krkn. The latest release is 5.2.12, published on 3 October 2026 at 2:24 AM IST, with a report format change and fixes for pod kill counts and Prometheus range queries. Releases came every few days in August and September 2026. The companion CLI, krknctl, is at 0.14.2-beta from 8 September 2026. Krkn was accepted into the CNCF sandbox on 19 December 2023.

Chaos Toolkit and the rest. Chaos Toolkit 1.21.4 was released on 5 October 2026 and needs Python 3.12 or later, with chaostoolkit-lib 1.45.1 from 28 September 2026. Its Kubernetes extension, chaostoolkit-kubernetes, has not had a release since 0.39.0 in May 2024, which matters if you planned to drive Kubernetes faults through it. Pumba 1.2.1 came out on 23 August 2026, and Toxiproxy 2.13.0 on 14 October 2026, its first release since March 2025.

The first question: what do you want to learn?

Before any tool, write one sentence that starts with “We believe that…” and ends with something you can measure. Chaos engineering is an experiment, and an experiment needs a hypothesis. “Let us see what happens if we kill pods” is not a hypothesis. “We believe that if one settlements pod is deleted during normal traffic, the p95 latency of the settlement API stays under 800 ms and the error rate stays under 0.5 per cent” is.

Steady state comes first. The steady state is the normal, measurable behaviour of the system from the user’s point of view: requests served per second, error rate, latency percentiles, orders completed per minute, lag on a Kafka consumer. Pick two or three such signals before you pick a fault. If you cannot measure them today, the first chaos project is actually a monitoring project. All three tools can check signals during a run, but none of them can invent a good signal for you.

Then decide which kind of question it is. Most useful experiments fall into a few groups:

Recovery: does the system come back by itself after a pod, node or zone is lost, and how long does it take?
Degradation: when a dependency becomes slow rather than dead, do timeouts, retries and circuit breakers protect the user, or do they make it worse?
Capacity under failure: with one zone gone, can the remaining nodes carry peak traffic?
Operations: do the alerts fire, does the runbook work, does the on-call person know what to do?
Data: after a database failover, is anything lost or duplicated, and do the applications reconnect?

The last two are often the most valuable and the least tool-dependent.

Game days or continuous chaos? A game day is a planned session where a group of people runs a few experiments together, watches dashboards and writes down what happens. Continuous chaos means experiments run on a schedule or in a pipeline without a human watching every run. Start with game days. They teach you which signals matter, they test people and runbooks as well as software, and the right people are already in the room when something surprising happens. Move an experiment to a schedule only after it has passed by hand a few times, has an automatic abort and has an owner who reads the results.

And decide what you will not test yet. Write down the experiments that are out of scope for now: anything on the primary database in production, anything that crosses into a payment partner’s systems, any experiment during month-end settlement.

What each tool is

Chaos Mesh

Chaos Mesh started inside PingCAP as part of the testing platform for the TiDB distributed database, and was open-sourced on 31 December 2019. Each fault type is a custom resource. The Chaos Controller Manager watches those resources, validates them through admission webhooks and drives the injection. The work on a node is done by the Chaos Daemon, a DaemonSet that enters the namespaces of the target pod, for example to add traffic control rules or intercept file operations. The Chaos Dashboard is a web UI and HTTP API on top, with login based on Kubernetes service account tokens.

The fault catalogue is broad: pod kill, pod failure and container kill; network delay, loss, duplication, corruption, partition and bandwidth limits; DNS errors; HTTP request and response changes; file system I/O delays and errors; CPU and memory stress; clock skew; JVM faults; kernel faults; block device faults; and cloud faults that stop or restart instances through AWS, Azure or GCP APIs. It can also reach physical machines through chaosd and other clusters through the RemoteCluster resource.

A minimal network delay towards a database looks like this:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: ledger-db-delay
  namespace: payments-staging
spec:
  action: delay
  mode: fixed-percent
  value: "50"
  selector:
    namespaces:
      - payments-staging
    labelSelectors:
      app: ledger-api
  direction: to
  target:
    mode: all
    selector:
      namespaces:
        - payments-staging
      labelSelectors:
        app: ledger-postgres
  delay:
    latency: "200ms"
    jitter: "50ms"
  duration: "5m"

The duration field matters. Without it, the fault stays until someone pauses or deletes the object. Scheduling is done with a separate Schedule object that takes a cron expression, and multi-step experiments with a Workflow object, which can run faults in series or in parallel and include HTTP StatusCheck steps.

LitmusChaos

LitmusChaos began in 2017 and became a CNCF project in 2020. The company that created it, ChaosNative, was acquired by Harness in March 2022, and Harness now sponsors the project and sells Harness Chaos Engineering built on it. Since version 3.0 in October 2023 the vocabulary changed: a single fault is a “chaos fault”, a sequence of faults is a “chaos experiment”, and the agent in a target cluster is a “chaos infrastructure”.

Litmus is split into two planes. The control plane is ChaosCenter: a React web portal, a GraphQL backend server, an authentication server and a MongoDB database that stores users, projects, experiments, templates, ChaosHub connections and GitOps settings. The execution plane runs in each target cluster: a subscriber that talks to the control plane, an Argo Workflows controller that runs the experiment steps, the chaos operator that watches ChaosEngine resources, a runner pod per engine and one job pod per fault. An optional chaos exporter publishes results to Prometheus, and an optional event tracker can trigger experiments when tracked resources change, if GitOps is enabled.

The fault library is the largest of the three for application-level faults. The litmus-go repository covers pod delete, container kill, pod stress, pod network faults, DNS and HTTP faults, node drain, taint, restart and stress, kubelet service kill and disk fill, plus AWS (EC2, EBS, RDS), GCP, Azure, VMware, Spring Boot and Kafka faults.

The heart of a run is the ChaosEngine, which ties a target to one or more faults and carries probes:

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: ledger-pod-delete
  namespace: payments-staging
spec:
  engineState: active
  appinfo:
    appns: payments-staging
    applabel: app=ledger-api
    appkind: deployment
  chaosServiceAccount: pod-delete-sa
  jobCleanUpPolicy: delete
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"
            - name: PODS_AFFECTED_PERC
              value: "34"
        probe:
          - name: ledger-health
            type: httpProbe
            httpProbe/inputs:
              url: http://ledger-api.payments-staging.svc:8080/healthz
              method:
                get:
                  criteria: ==
                  responseCode: "200"
            mode: Continuous
            runProperties:
              probeTimeout: 2s
              interval: 2s
              retry: 2
              probePollingInterval: 2s
              stopOnFailure: true

Setting engineState to stop ends a running engine. Most teams build experiments in ChaosCenter and keep YAML like this for review.

Krkn

Krkn, earlier spelt Kraken, came from Red Hat’s performance and scalability team, who introduced it in a Red Hat blog post in October 2020. Their aim was to check not only that components recover after a failure, but that performance does not degrade during and after it. It moved from the redhat-chaos GitHub organisation to krkn-chaos and joined the CNCF sandbox in December 2023.

The big difference from the other two is where it runs. Krkn is a Python program that runs outside the cluster, on a laptop, a CI runner or as a container through krkn-hub or krknctl, and uses a kubeconfig and cloud credentials to cause failures. Some scenarios start helper pods, but there is no permanent controller or daemon to install. A Krkn Operator now exists for teams that prefer a resident component, and Cerberus is a companion service that watches cluster health and publishes a go or no-go signal.

The scenario list reflects its platform focus: pod and container failures, node failures through many cloud and virtualisation APIs, zone outages, whole-cluster shutdown, node resource hogs, network chaos, DNS outages, etcd split brain, service disruption (which deletes all objects in a namespace), SYN floods, HTTP load, PVC disk fill, storage I/O throttling, time skew and KubeVirt VM outages.

A scenario is a small YAML file referenced from the main config:

- id: kill-pods
  config:
    namespace_pattern: ^payments-staging$
    label_selector: app=ledger-api
    kill: 1
    krkn_pod_recovery_time: 120

The pass or fail decision is the interesting part. Krkn checks that the expected pods or nodes came back within the recovery time, can consume the Cerberus signal, can evaluate a profile of Prometheus expressions during or after the run and exit with a non-zero code depending on severity, can fail the run if critical alerts are firing afterwards, and can calculate a resiliency score from 0 to 100 per cent based on weighted SLOs.

A word on Chaos Toolkit, Pumba, Toxiproxy and the managed services

Chaos Toolkit is an open-source experiment runner, started in 2017. An experiment is a JSON or YAML file with a steady-state hypothesis (probes with tolerances), a method (actions and probes) and rollbacks, and the toolkit runs it from the command line and writes a journal. Extensions add drivers for clouds and other systems. It suits experiments that cross system boundaries and teams that want experiments as plain files without a controller in the cluster. Our drill uses it.

Pumba injects faults into Docker containers (kill, stop, pause, network delay and loss through netem), which is handy on a single host or in Docker Compose test setups. Toxiproxy is a TCP proxy from Shopify that adds latency, bandwidth limits, timeouts and connection resets between your application and a dependency.

AWS Fault Injection Service and Azure Chaos Studio are managed fault injection services. They are worth a serious look if almost everything you run is on one cloud, because they can act on the cloud’s own resources with the cloud’s own permission model and audit trail. Check their current fault lists, pricing and Kubernetes support in their own documentation; we did not test them. Gremlin is a commercial chaos platform with agents and a hosted control plane, for teams that want vendor support, a polished UI and reporting more than they want to run the tool themselves.

Side by side

Licence and governance

Chaos Mesh: Apache License 2.0, CNCF incubating since February 2022. Started at PingCAP. The release notes in October 2026 describe a small maintainer team.
LitmusChaos: Apache License 2.0, CNCF incubating since January 2022. Harness is the main sponsor and sells Harness Chaos Engineering, which includes its own SaaS control plane. The open-source project releases about once a month.
Krkn: Apache License 2.0, CNCF sandbox since December 2023. Most maintainers are from Red Hat, and the project is used to test OpenShift releases. Sandbox means earlier in the CNCF process, so plan for more change in components and config.
Chaos Toolkit, Pumba: Apache License 2.0. Toxiproxy: MIT.

Architecture and moving parts

Chaos Mesh: controller manager (three replicas by default in the Helm chart, with leader election), Chaos Daemon on every node, dashboard, a DNS server for DNSChaos, and many CRDs. Everything lives in the cluster.
LitmusChaos: a control plane with four main services and MongoDB (which can serve several clusters), and in each target cluster a subscriber, an Argo Workflows controller, the chaos operator, optional exporter and event tracker, plus short-lived runner and fault pods. It has the most moving parts.
Krkn: a client-side program plus optional Cerberus and optional operator. Helper pods appear only during a scenario. It has the fewest resident parts, but it needs network access to the API server, Prometheus and cloud APIs from wherever it runs.

Fault coverage

Chaos Mesh is strongest at fine-grained in-pod faults: network shaping between selected pods, HTTP and DNS faults, file system I/O errors, clock skew, JVM and kernel faults. It also has cloud and physical machine faults.
LitmusChaos has the widest ready-made catalogue across pods, nodes, cloud resources and some applications, and a ChaosHub to share faults. Its HTTP and DNS faults overlap with Chaos Mesh.
Krkn is strongest at platform-level failures: nodes through cloud APIs, zones, etcd, control plane components, whole-cluster shutdown, storage fill and throttling, plus load generation. Its in-pod fault set is smaller.

Safety and blast radius

Chaos Mesh: selectors by namespace, labels, fields, annotations, pod phase and node, plus modes (one, all, fixed, fixed-percent, random-max-percent) to limit how many targets are hit. The FilterNamespace feature, off by default, restricts injection to namespaces annotated with chaos-mesh.org/inject=enabled, which is the closest thing to a namespace allowlist. It can be installed namespace-scoped instead of cluster-scoped. Pausing an experiment with an annotation or deleting it restores the fault; duration restores it automatically. In a workflow, a StatusCheck with abortWithStatusCheck stops the workflow when the HTTP check fails a set number of times. The docs warn that if the controller cannot reach the daemon, injected network faults cannot be recovered, so a partition that cuts that path is a real risk.
LitmusChaos: targets by namespace, labels and kind, with percentages of pods affected. Chaos infrastructure can be installed in namespace mode, so it can only act within one namespace. Probes with stopOnFailure stop a fault when the steady-state check fails, and engineState stop ends a run. ChaosCenter projects have roles, and only owners can schedule experiments.
Krkn: namespace and name patterns as regular expressions, label selectors, kill counts and exclude labels to protect critical pods. Its RBAC documentation separates a namespace-scoped role for pod, container, PVC and application scenarios from a cluster-wide role for node, network, hog, time, zone and shutdown scenarios. The default config has auto_rollback turned on, and Krkn can be paused or stopped through a signal on a port.
In all three, the most important control is boring: start in staging, hit one pod, set a duration, have a person watching and a way to stop it.

Observability and steady-state checks

Chaos Mesh: events and status on each chaos object, dashboard history, controller metrics, and an HTTP-only StatusCheck in workflows that can abort but not pause.
LitmusChaos: the richest built-in checks. httpProbe, cmdProbe, k8sProbe and promProbe can run at the start, at the end, at both edges, continuously or only during the fault. The ChaosResult resource records a verdict and a probe success percentage, ChaosCenter shows a resilience score, and the chaos exporter publishes metrics to Prometheus.
Krkn: built-in recovery checks for pods and nodes, Prometheus SLO profiles with severity, critical alert checks, URL health checks, Kubernetes object state checks, Cerberus, a resiliency score, and telemetry and metrics that can be stored in Elasticsearch or OpenSearch.

Security of the chaos tool itself

Chaos Mesh: the Chaos Daemon runs as a privileged DaemonSet by default; with privileged off, the chart still adds SYS_ADMIN, NET_ADMIN, SYS_PTRACE and other capabilities, and it mounts the container runtime socket from the host. Anyone who can create its resources can disrupt workloads in that scope. The dashboard uses Kubernetes tokens and its security mode is on by default; keep it on and do not expose the dashboard publicly. The Helm chart also enables pprof profiling by default, which is worth reviewing. The gRPC link between controller and daemon uses mutual TLS by default.
LitmusChaos: the control plane is a web application with its own users, passwords and a MongoDB database, so treat it like any internal admin portal. The documented default login is admin with password litmus; change it at first login and put the portal behind your single sign-on or VPN. The basic install allows all origins, while the HTTPS install applies CORS rules. In the execution plane, network faults start helper pods that, in the litmus-go source we read, use the host PID namespace, privileged mode and NET_ADMIN and SYS_ADMIN capabilities. The 3.33.0 authorisation fixes are a reminder that the portal itself can have bugs.
Krkn: no resident privileged daemon, but the person or CI job running it holds a powerful kubeconfig, and some scenarios need cluster-wide rights or, on OpenShift, the privileged security context constraint. Its status and signal server listens on 0.0.0.0 port 8081 by default, so on a shared CI runner bind it to localhost or a private interface. Telemetry is enabled in the default config; it prints a summary at the end and uploads only if you configure a telemetry endpoint, so check that setting. Cloud node scenarios need credentials that can stop instances; keep them narrow.
For any of them: install in a dedicated namespace, limit who can create chaos resources with Kubernetes RBAC, alert on chaos resources created outside the agreed hours, keep the tool’s images pinned and scanned, and treat its service accounts as high-value credentials. Tell your security team before the first game day, because runtime security tools will see chaos daemons doing what attackers do.

Operations and upgrades

Chaos Mesh: Helm chart and CRDs; upgrade CRDs before the chart, as with most operators. The spec of a running experiment cannot be updated, so you delete and recreate. If recovery is blocked, the docs describe a forced clean-up, which can leave faults behind on nodes, so check afterwards. Patch releases came three times in October 2026, so plan to read release notes often.
LitmusChaos: two things to upgrade, the control plane and each chaos infrastructure, and their versions should match. MongoDB needs backups like any database. Monthly releases mean frequent small upgrades. On 29 October 2026 the ChaosCenter Helm chart in the litmus-helm repository’s main branch still showed version 3.30.0 while the app was at 3.33.0, so check the chart and image versions you are actually installing.
Krkn: upgrade is pulling a new version or container image, which is the easiest of the three. Config keys and scenario formats have changed across versions, so keep your scenario files in a repository and run them in CI after every upgrade.

Resource footprint

None of the three projects publishes a single sizing table that we could confirm, so treat these as relative, not measured.

Chaos Mesh: one daemon pod per node plus controller replicas and the dashboard.
LitmusChaos: the heaviest at rest because of the control plane and MongoDB. The installation guide recommends a 20 GB persistent volume for the portal’s data (1 GB is enough to try it). The execution plane adds a few small pods per cluster.
Krkn: close to nothing resident in the cluster; resource use is during the run, in helper pods and in whatever runs Krkn itself.

A worked plan for Coimbatore

Step 1: write the questions and the “not yet” list. The team writes five hypotheses, one per line, each with a measurable steady state. For example: losing one settlements pod keeps p95 under 800 ms; losing one worker node in a zone causes no user-visible errors for more than 30 seconds; a 200 ms delay between the ledger service and PostgreSQL keeps the error rate under 0.5 per cent; a CloudNativePG primary failover in staging is followed by all services reconnecting within 60 seconds; losing one zone in staging under peak-like load keeps the order API available. The “not yet” list: anything in production for the first two months, the settlements database primary in production, the payment gateway integration, and month-end weeks.

Step 2: fix what is already known. The festival-season outage already told them something: the settlements service needs replicas spread across nodes and zones, and a PodDisruptionBudget that keeps at least one pod. They fix topology spread constraints and PDBs for the 10 most important services before running any tool.

Step 3: make steady state visible. They build one Grafana dashboard for experiments: request rate, error rate and p95 latency for the order and settlement APIs, Kafka consumer lag for shipments, CloudNativePG replication lag, and the Blackbox exporter results for the public pages. They agree that any experiment stops if the error rate goes above 2 per cent for one minute. A k6 load test in staging provides weekday-like traffic.

Step 4: a two-week trial with two paths. Path A installs Chaos Mesh in staging with FilterNamespace on and the dashboard reachable only through port forwarding. Path B installs Litmus ChaosCenter with a chaos infrastructure in namespace mode, and changes the admin password on day one. Both run the same three experiments: one pod deleted, a 200 ms delay to PostgreSQL and a CPU stress on one pod. Separately, the platform team runs Krkn from a CI runner for node and zone scenarios, with only the role Krkn documents and cloud credentials limited to stopping staging instances.

Step 5: the choice. In this invented story the team picks Chaos Mesh for application experiments, because the network and HTTP faults between specific services were exactly what they needed and the CRDs fitted their GitOps flow, and Krkn for platform experiments, because node and zone loss through the cloud API with Prometheus SLO checks matched their release gate. Litmus was close and its probes were the best of the three, but they did not want to run and secure another web portal with its own MongoDB.

Step 6: the first game day. A two-hour session on a Tuesday afternoon in staging. Roles: a lead, an observer who writes everything down, the service owner and one person whose only job is to stop the experiment. They run one pod delete, then the PostgreSQL delay. The delay finds the real problem: a 30 second database timeout plus Linkerd retries turns a 200 ms delay into long queues and a rising error rate after about three minutes. The StatusCheck aborts the workflow. They fix the timeout and retry budget and repeat the test a week later.

Step 7: the database failover. Using CloudNativePG’s own switchover and a Chaos Mesh pod kill on the primary in staging, they test the failover hypothesis. The service with the cached connection pool does not reconnect, as suspected; they fix the pool settings and add a readiness check on the database connection.

Step 8: zones and nodes with Krkn. In staging, under the k6 load, Krkn stops one worker node through the cloud API, waits for recovery and checks a Prometheus SLO profile for error rate and latency. Then it simulates a zone outage. The zone test finds the ingress controller’s replicas in only two zones. They fix it and add the node scenario to the pipeline before each platform upgrade.

Step 9: guard rails before production. Before any production experiment: FilterNamespace on with only two annotated namespaces, Chaos Mesh resources creatable only by a “chaos-operators” group, an alert when any chaos resource is created outside the agreed window, every experiment with a duration and a StatusCheck, experiments reviewed as pull requests, and a short note to customer support before each production run.

Step 10: the first production experiment and the habit. After two months, they run one pod delete on the order API in production on a weekday morning, with one pod out of six, a two-minute duration and the whole team watching. It passes. From then on: one game day a month, scheduling only for experiments with an automatic abort and an owner, and a simple log of date, hypothesis, result and fix. That log is the audit evidence.

When each is better

Choose Chaos Mesh when your platform is Kubernetes, your team is comfortable with CRDs and GitOps, and you need precise faults between specific services: network delay and partition, HTTP and DNS faults, I/O errors, clock skew or JVM faults.
Choose LitmusChaos when you want a portal with users, projects, a hub of ready faults, rich probes and resilience scores, possibly across many clusters from one control plane, or when several teams who are not all Kubernetes experts will run experiments.
Choose Krkn when your main questions are about the platform: nodes, zones, etcd, control plane, storage and whole-cluster recovery, especially on OpenShift, and when you want a pass or fail result based on Prometheus SLOs in a CI pipeline with nothing resident in the cluster.
Choose Chaos Toolkit when experiments span systems beyond Kubernetes and you want a small, file-based runner; Toxiproxy when you want to prove client timeouts in integration tests; and a managed cloud service when nearly everything runs on one cloud and you want its permissions and audit trail.

When not to adopt chaos tooling

When you have no monitoring of user-facing behaviour. An experiment you cannot observe teaches nothing.
When known problems are still open. If incident reviews already list single replicas, missing PDBs or untested backups, fix those first. Chaos engineering is for finding unknowns.
When you have no on-call or no way to roll back quickly. Do not inject faults into a system that nobody can repair at that hour.
When the system is a single server or a monolith with no redundancy. A planned restart test and a restore test will teach you more than a chaos platform.
When trust is low. One badly run experiment that hurts customers can set reliability work back a year; earn trust with staging results first.
When a simpler tool covers the need: kubectl drain for a node, CloudNativePG’s switchover command for a failover test, k6 for load, Toxiproxy in tests.

A short history

2011: Netflix wrote publicly about Chaos Monkey, which randomly terminated instances in production. It was open-sourced in 2012.
15 March 2017: the LitmusChaos repository was created on GitHub.
31 December 2019: PingCAP open-sourced Chaos Mesh.
14 January 2020: Litmus 1.0.0 was released.
25 June 2020: Litmus was accepted into the CNCF sandbox; Chaos Mesh followed on 14 July 2020.
8 October 2020: Red Hat introduced Kraken (now Krkn) in a blog post.
23 July 2021: Chaos Mesh 2.0.0 was released, with Workflow and Schedule.
11 January 2022: Litmus moved to CNCF incubating; Chaos Mesh followed on 16 February 2022.
22 March 2022: Harness acquired ChaosNative, the original creator of LitmusChaos.
4 October 2023: Litmus 3.0.0 was released, with new names for faults, experiments and infrastructure.
19 December 2023: Krkn was accepted into the CNCF sandbox.
1 October 2025: Chaos Mesh 2.8.0 was released.
October 2026: Litmus 3.33.0 (15 October), Chaos Mesh 2.8.7 (28 October), Krkn 5.2.12 (3 October) and Chaos Toolkit 1.21.4 (5 October).

Decision guide

Do you have clear user-facing signals and an on-call team? If not, stop and build those first.
Is your main question about a single service and its dependencies inside Kubernetes? Start with Chaos Mesh, or Litmus if you want a portal and probes.
Is your main question about nodes, zones, the control plane or whole-cluster recovery? Start with Krkn, or your cloud’s managed service if you are on one cloud.
Do several teams, not all Kubernetes experts, need to run experiments with roles and reports? Litmus is the most natural fit.
Do you want nothing resident in the cluster and a pass or fail in CI? Krkn, or Chaos Toolkit for experiments outside Kubernetes.
Is your security team uncomfortable with a privileged DaemonSet? Prefer Krkn or Litmus in namespace mode, and use Chaos Mesh only where you accept that risk.
Do you want vendor support? Compare Harness Chaos Engineering, Gremlin or your cloud’s service against the open-source option using the same three experiments.

Checklist

One written hypothesis per experiment, with a measurable steady state.
A dashboard with the steady-state signals and experiment markers.
Realistic traffic during the experiment.
Blast radius set explicitly: namespace, labels, number or percentage of targets.
A duration on every fault.
An automatic abort (StatusCheck, probe with stopOnFailure, or SLO check) and a person who can stop it by hand.
Namespace allowlisting (Chaos Mesh FilterNamespace or Litmus namespace mode) and narrow RBAC for who can create chaos resources.
The chaos tool’s dashboard or portal behind single sign-on or a VPN, default passwords changed.
Backups verified before experiments that touch stateful workloads.
A written record after each run: what happened, what was learnt, what was fixed.
A rule for moving experiments from game days to schedules, and an owner for each scheduled experiment.

Common mistakes

Starting with production. Staging under load finds most early problems safely.
Testing only pod deletion. Kubernetes is very good at restarting pods; the interesting failures are slowness, partial network loss, zone loss and database failover.
No duration on a fault, so it stays in place after the engineer goes home.
Trusting the tool’s “passed” without checking user-facing signals.
Leaving the chaos dashboard exposed, with default credentials or with authentication turned off “for the trial”.
Running a network partition that cuts the chaos controller from its daemon, so the fault cannot be removed automatically.
Not telling the security team before a privileged pod starts changing network rules.
Not writing down results, so the same weakness is found again six months later.
Scheduling dozens of experiments that nobody reads, until real failures are ignored because “it is probably chaos”.
Forgetting rollbacks or clean-up. Our own drill below shows how easily this happens.

The drill we ran

We ran a small drill on a shared Linux machine with no cluster, no Docker, no system services and nothing beyond the loopback interface.

A tiny Python web server on 127.0.0.1 with a /health endpoint and two endpoints to switch an artificial delay on and off. We started and stopped it ourselves.
We installed Chaos Toolkit 1.21.4 (with chaostoolkit-lib 1.45.1) in a Python virtual environment and wrote an experiment: steady state “health answers 200 within 1 second”, method “switch on a 2 second delay”, rollback “switch the delay off”.
First run: the steady state passed before the fault, the delay was injected, the probe after the method timed out, and the run ended as deviated with exit code 1. That was the expected weakness. The surprise was that the rollback did not run. Chaos Toolkit’s default rollback strategy, as its own help text says, does not play rollbacks when the hypothesis fails. The server was left with the 2 second delay.
Second run, with a harmless 0.3 second delay: it failed immediately, because the steady state before the method already failed. The leftover fault from the first run had poisoned the next experiment. This is a small version of a real production mistake.
Third run, after restarting the server, with the rollback strategy set to always: the same deviation, but the rollback ran and the health endpoint was back to no delay. Fourth run, with the 0.3 second delay: the hypothesis held, the run completed with exit code 0 and the rollback ran.
We then checked the YAML examples in this post. With PyYAML and the jsonschema package, we validated the Chaos Mesh NetworkChaos and Workflow examples against the CRD schemas from the Chaos Mesh 2.8.7 Helm chart, and the Litmus ChaosEngine example against the CRD from chaos-operator 3.33.0. All three were valid. A copy of the NetworkChaos example with two typos (latncy for latency and fixed-percnt for fixed-percent) was rejected with “latency is a required property” and a list of the allowed modes.
A small guard script checked the same files for safety rules: namespace in an allowlist, no mode all, durations, a StatusCheck with abort, probes in Litmus engines and anchored Krkn namespace patterns. A PodChaos that killed all pods with a label in any namespace passed the CRD schema but failed the guard on three rules. Schema validity is not safety.

What we did not run: any Kubernetes cluster, Chaos Mesh controller or daemon, Litmus ChaosCenter or MongoDB, Krkn itself, krknctl or Cerberus, any real fault injection with tc, netem or stress tools, any cloud fault, any dashboard or authentication test, any upgrade, AWS Fault Injection Service, Azure Chaos Studio, Gremlin, Pumba or Toxiproxy, and no benchmarks of any kind.

Unlearn, re-learn

Unlearn “chaos engineering means breaking things”. Re-learn that it means running experiments with a hypothesis, a measured steady state and a controlled blast radius.
Unlearn “pods came back, so we are resilient”. Re-learn to test slowness, partial failure, zones and data, under real traffic.
Unlearn “more experiments on a schedule is better”. Re-learn that one well-run game day with written results beats fifty unread scheduled runs.
Unlearn “the chaos tool is just a test tool”. Re-learn that it is a privileged system that can change production, and secure it like one.
Unlearn “the tool will clean up after itself”. Re-learn to set durations, check rollback behaviour and verify that the system is back to normal after every run.
Unlearn “we need a chaos platform to start”. Re-learn that kubectl drain, a database switchover, a load test and a notebook can be your first chaos programme.

Sources

comments powered by Disqus

Releted Posts

Container engines in 2026: revisit Docker Engine, Podman and containerd before you install another container daemon

Manoj Shukla
October 28, 2026

On most teams, installing a container engine is the first thing done on a new machine and the last thing anybody thinks about.

Manoj Shukla
October 20, 2026

Think of a bicycle and auto parts manufacturer in Ludhiana that sells through about 400 dealers across Punjab, Haryana, Rajasthan and the hill states (an invented example, not a real client).

Manoj Shukla
October 20, 2026

Picture a small software company in Bhubaneswar that builds claims and pre-authorisation software for hospitals that deal with health insurance TPAs (an invented example, not a real client).