Kubernetes backup in 2026: revisit Velero, K8up and Kanister before you trust another cluster restore

Most Kubernetes teams can rebuild a Deployment in two minutes. Fewer can say how long it takes to get back the data that Deployment was using. The manifests live in Git, the images live in a registry, and so the cluster feels safe. Then a namespace is deleted by mistake, the PersistentVolumeClaims go with it, the storage class reclaim policy says Delete, and the team discovers that Git never had the data in the first place.

This post looks at three open-source tools for backing up and restoring what runs inside a Kubernetes cluster: Velero, K8up and Kanister. All three are still actively maintained on 26 October 2026, so no swap was needed. They solve overlapping but different problems. Velero backs up Kubernetes objects and volume data together. K8up backs up volume files and command output with restic, namespace by namespace. Kanister runs application-specific backup and restore workflows, such as a database dump, that you describe once and reuse.

We have written before about restic, BorgBackup and Kopia on servers and about pgBackRest, WAL-G and Barman for PostgreSQL. This one stays at the cluster level: what to capture, where it should go, and how to prove a restore works.

A scenario to keep us honest

Take an invented Dehradun company that builds school management software for private schools in Uttarakhand: admissions, fee collection, attendance, exam marks and report cards. About 30 engineers. Everything runs on one self-managed Kubernetes cluster on rented virtual machines, with a CSI storage driver that supports volume snapshots. Argo CD deploys every application from Git.

The data side looks like this:

• PostgreSQL for the main application, run by the CloudNativePG operator, which already ships its own base backups and WAL to object storage.
• An older attendance module on a MariaDB StatefulSet, with a nightly mysqldump cron job that copies the file to a separate virtual machine.
• A PersistentVolumeClaim holding uploaded documents: student photos, scanned transfer certificates and generated report card PDFs.
• A handful of Secrets created by hand during setup: payment gateway keys, the SMS provider token and the private key of the Sealed Secrets controller.

During the term-end results week, an engineer ran a cleanup script against the wrong kube context and deleted the exams namespace in production. Argo CD recreated the Deployments and Services within minutes. That was the easy part. The uploads volume was gone, because its PersistentVolume had a Delete reclaim policy. The MariaDB data was gone, and the latest dump on the VM was 20 hours old. Two hand-made Secrets were not in Git, so the payment and SMS services could not start. Getting back to normal took most of a working day, and some report cards had to be regenerated from marks re-entered by teachers.

Nothing here was exotic. The team had GitOps and had a database dump. What they did not have was a cluster backup that knew which objects and which volumes belonged together, and a restore they had practised.

The short version

• Git is not a backup of your cluster. GitOps restores the desired state of objects you put in Git. It does not restore volume data, hand-made Secrets, objects created by operators at runtime, or the private key that decrypts your sealed secrets.
• Cluster objects and volume data are two separate problems. Objects are small JSON documents read from the API server. Volume data can be hundreds of gigabytes and has to be read from storage. Good tools treat them separately and then tie them together at restore time.
• Velero (now a CNCF Sandbox project, Apache 2.0) backs up both. For volumes it can take CSI snapshots, move snapshot data to object storage with its built-in Kopia data mover, or read files from the mounted volume. It has backup and restore hooks for application consistency.
• K8up (CNCF Sandbox, Apache 2.0) is narrower and simpler. It backs up the files of every PVC in a namespace with restic, plus the output of any command you annotate on a pod, such as pg_dump. It does not back up Kubernetes objects.
• Kanister (CNCF Sandbox, Apache 2.0) is a workflow engine for data operations. You write a Blueprint that says how to back up and restore a particular application, and run it with an ActionSet. It is the most flexible and needs the most writing.
• Databases deserve their own tool first. If an operator or a database-native tool already does consistent backups with point-in-time recovery, let it own the database. Use the cluster tool for everything around it.
• A backup you have never restored is a hope. Schedule restore drills into a separate namespace or cluster, and time them.

For the Dehradun team, a sensible answer is Velero for objects and the uploads volume, with CSI snapshot data movement to an object store outside the cluster, a pre-backup hook for MariaDB, CloudNativePG left in charge of PostgreSQL, and a monthly restore drill.

Where things stand on 26 October 2026

Versions and dates below are from GitHub release pages, the GitHub API and the projects’ own documentation, checked on 26 October 2026. Times are in IST.

• Velero v1.18.4, released 28 September 2026 at 11:50 AM. It is a small patch release that fixes a backup queue that could get stuck. The previous patch, v1.18.3 (21 September 2026 at 1:44 PM), fixed a case where a file system backup timeout could lose PodVolumeBackup metadata and leave all file system backup volumes unrestorable. The 1.18 line started with v1.18.0 on 6 March 2026 at 12:03 PM, which added concurrent backups, cache volumes for data mover pods during restore, reporting of incremental size, and glob patterns for namespace filters. Its release notes list Kopia 0.22.3 as the bundled dependency. The README’s compatibility table says Velero 1.18 is expected to work with Kubernetes 1.18 and later and was tested on 1.33.7, 1.34.1 and 1.35.0. Licence: Apache 2.0.
• Velero AWS plugin (also used for S3-compatible stores) v1.14.4, released 28 September 2026 at 1:03 PM. Its README maps plugin v1.14.x to Velero v1.18.x.
• K8up v2.16.1, released 22 October 2026 at 8:13 PM, with the Helm chart k8up-4.10.1 at 8:36 PM the same day. It updates the bundled restic module to v0.19.1 and several security-relevant Go dependencies. The previous minor release, v2.16.0 (17 July 2026 at 5:26 PM), added a k8up_schedule_last_job_succeeded metric and fixed relaxed scheduling for mounted RWO volumes. The documentation says K8up v2 officially supports Kubernetes 1.31 or later and is built and tested on Linux x86-64, with arm64 images on a best-effort basis. Licence: Apache 2.0.
• Kanister 0.120.0, released 24 October 2026 at 12:52 AM. It adds Azure Workload Identity support for moving backup data to Azure Blob Storage and removes leftover code for the old repository server. The previous release, 0.119.0, came on 13 May 2026, so releases are roughly every few months rather than every few weeks. Example Blueprints moved to a separate kanisterio/blueprints repository in 0.118.0. The project still uses 0.x numbering and v1alpha1 API groups. Licence: Apache 2.0.
• Underlying engines, for context: Kopia v0.23.1 (16 June 2026 at 6:57 AM) and restic v0.19.1 (5 July 2026 at 1:43 PM).

On governance, the biggest change this year is Velero’s. Velero, formerly Heptio Ark, lived for years in the vmware-tanzu GitHub organisation. In February 2026 an application was filed for the CNCF Sandbox. The application names Broadcom as the sponsoring organisation and lists maintainers from Broadcom, Red Hat and Microsoft. The CNCF accepted Velero on 11 March 2026, and in April 2026 the repositories moved from the vmware-tanzu GitHub organisation to a new, neutral velero-io organisation. Old links redirect, but update your Helm values, plugin image references and bookmarks. K8up was accepted to the CNCF Sandbox on 16 November 2021 and its listed maintainers are from VSHN, where it started. Kanister was accepted on 20 September 2023; its maintainers are listed in the repository’s MAINTAINERS file.

All three are alive and maintained. None of them is abandoned code you need to plan an exit from.

The first question: what are you actually trying to get back

Before comparing tools, write down what a restore must bring back. For most clusters it is four different things.

• Kubernetes objects. Deployments, Services, ConfigMaps, Secrets, Ingresses, custom resources, RBAC. Small, read from the API server, and mostly in Git if you use GitOps. The important exceptions are anything created by hand and anything created by controllers at runtime.
• Volume data. The contents of PersistentVolumes. Large, and only reachable through the storage system or a pod that mounts the volume.
• Application state that needs care. Databases, queues and search indexes, where copying files from under a running process may give you something that does not start.
• The cluster itself. The control plane and its etcd store. On a managed Kubernetes service the provider owns this. On a self-managed cluster you do.

Velero covers the first two and has hooks for the third. K8up covers the second and the third. Kanister is mainly about the third. None of them replaces an etcd snapshot for the fourth, which we come back to later.

What each tool is

Velero

Velero runs as a server in the cluster plus a command-line client. Every operation, whether a backup, a schedule or a restore, is a custom resource that Velero’s controllers act on. A backup queries the API server for the selected resources, writes them as a tarball to object storage, and handles volumes in one of three ways:

• CSI snapshot. Velero asks the CSI driver for a VolumeSnapshot. Fast, and the snapshot stays on the storage system. Velero’s own docs warn that not every CSI driver keeps snapshots durable, so the snapshot may live alongside the original volume and be lost with it.
• CSI snapshot data movement. Velero takes the CSI snapshot, then a data mover pod reads it and uploads the data to the backup storage location with Kopia. After the upload, Velero deletes the snapshot. Your volume data now lives in object storage, away from the storage system. You turn this on per backup with --snapshot-move-data.
• File system backup. The node agent reads files from the mounted volume and uploads them with Kopia. This works for volumes that have no snapshot support, such as NFS or local volumes, but the docs say plainly that it reads from the live file system, so the data is not captured at one point in time.

Since 1.14 the CSI plugin is built into Velero itself. The restic path is being retired: from 1.17, new backups cannot use restic, though you can still restore old restic backups; the docs say that from 1.19, restores from restic backups will be disabled too. If you have old restic-based Velero backups, restore-test them before you upgrade past 1.18.

K8up

K8up is a Kubernetes operator built on restic. You put a Schedule and a credentials Secret in a namespace, and K8up backs up that namespace on the schedule. Its documentation describes three methods:

• PVC backup. For every PVC in the namespace, K8up runs a job that mounts the volume and backs up its files with restic. For ReadWriteOnce volumes, the job has to run on the node where the volume is already mounted, and K8up handles that placement.
• Backup command. An annotation such as k8up.io/backupcommand on a pod tells K8up to run a command inside that container and save whatever it writes to standard output. A pg_dump or mysqldump gives you a consistent database dump this way.
• PreBackupPod. A pod definition that K8up starts for each backup run, runs the backup command in, and removes afterwards. Useful when the dump tool is not in the application image, or when the database is outside the cluster.

Schedules can also run check and prune jobs on the restic repository, and archive snapshots to a second bucket. Its docs are direct about the limit of the plain PVC method: it does not work well for files held open for a long time, like database files.

Kanister

Kanister is a framework rather than a ready-made backup product. It has three custom resources:

• A Blueprint describes actions, such as backup, restore and delete, for a kind of application. Each action is a list of phases, and each phase calls a Kanister function: run a command in a pod, start a temporary pod, scale a workload, take a CSI snapshot, copy a volume’s data to object storage, and so on.
• An ActionSet asks the controller to run an action from a Blueprint against a specific object, such as a StatefulSet, now.
• A Profile holds the object storage location and credentials.

The separate blueprints repository has ready examples for MySQL, PostgreSQL, MongoDB, Cassandra, Elasticsearch, Redis, etcd on OpenShift, and managed databases such as AWS RDS. The MySQL example streams mysqldump --single-transaction through gzip into object storage using Kanister’s kando tool, and the restore action streams it back into mysql. Kanister does not back up Kubernetes objects. It is there for the data that needs application knowledge.

Side by side, without a table

What gets backed up

• Velero: Kubernetes objects from the API server, plus volume data through snapshots, snapshot data movement or file system backup. Filters by namespace, resource type and label, with glob patterns for namespaces since 1.18.
• K8up: the files of PVCs in a namespace, plus standard output of backup commands. No Kubernetes objects.
• Kanister: whatever your Blueprint does. Typically a database dump or volume copy. No Kubernetes objects unless you script it.

Where data goes

• Velero: object storage through a BackupStorageLocation. Volume data goes into a Kopia repository under that location, one repository per namespace.
• K8up: a restic repository, most often S3-compatible object storage, defined per Backup or Schedule.
• Kanister: object storage described by a Profile; S3-compatible, Google Cloud Storage and Azure are the documented location types.

If you run your own S3-compatible store for this, our post on MinIO and its alternatives covers the choices. Whatever you pick, keep it outside the cluster you are protecting.

Point-in-time consistency

• Velero: a CSI snapshot is crash-consistent for one volume, like pulling the power. Data movement keeps that property because it reads from the snapshot. File system backup reads a live file system, so it is less consistent. The docs also note that object backups are not strictly atomic: objects changed while the backup runs may be missed.
• K8up: PVC file backups read the live file system. Backup commands give you whatever consistency the command gives, which for pg_dump is a consistent snapshot of one database.
• Kanister: whatever the functions in the Blueprint give you. A dump-based Blueprint is consistent by the database’s rules.

Application-consistent hooks

• Velero: pre and post backup hooks as pod annotations or in the Backup spec. Commands run inside a container of the pod, are not run in a shell unless you call one, default to a 30 second timeout and default to failing the backup on error. Since 1.15, hooks run around an “ItemBlock”, the group of items backed up together. Restore hooks can add an init container to a restored pod or run a command in it after restore.
• K8up: the backup command annotation and PreBackupPods. The command’s output is the backup. There is no separate “freeze, snapshot, thaw” step because there is no snapshot.
• Kanister: the whole tool is a hook system. A Blueprint can scale a workload down, run a dump, take a CSI snapshot and scale it back up, with a DeferPhase that runs even if earlier phases fail.

Scheduling and retention

• Velero: Schedule resources with cron syntax, and a time zone can be set with CRON_TZ=Asia/Kolkata at the start of the expression. Each backup has a TTL, 30 days by default, after which the backup, its files and its snapshots are garbage-collected; the collector runs every hour by default.
• K8up: schedules with cron syntax plus K8up-only entries such as @daily-random, which picks a stable random time per schedule to spread load. The docs say schedules use the operator’s local time zone, so in a typical container that means UTC. Retention is set on the prune job, for example keep the last 5 and 14 daily snapshots.
• Kanister: an ActionSet runs an action now. The docs I read do not describe a built-in scheduler, so teams usually create ActionSets from a CronJob, a CI pipeline or another controller. Completed ActionSets have to be cleaned up by the user.

Restore behaviour

• Velero: restores objects in a fixed priority order (CRDs, namespaces, storage classes, snapshot classes and so on). By default it does not touch objects that already exist; --existing-resource-policy update tries to update them, and even then PV data is not overwritten. It can remap namespaces in a restore, and the backup storage location can be switched to read-only during a restore so nothing is deleted by accident.
• K8up: a Restore object restores a restic snapshot to a new PVC or to an S3 bucket. Backups made from standard output cannot be restored to a PVC by K8up; the docs point you to restic dump or mount for those, after which you load the dump yourself.
• Kanister: a restore ActionSet runs the Blueprint’s restore action, using the artifacts recorded by the backup ActionSet.

Security of the backup data

• Velero: Kopia repositories are encrypted, but the docs state a limitation clearly: Velero uses one static, common encryption key for all repositories it creates, so anyone with access to the bucket can decrypt the data. The repository password lives in a velero-repo-credentials Secret and can be changed only before the first backup to that repository. Treat bucket access as access to all your data.
• K8up: restic repositories are encrypted with the password you supply through repoPasswordSecretRef. Lose the password and the backups are unreadable.
• Kanister: functions such as CopyVolumeData accept an encryption key; dump-based Blueprints depend on what the Blueprint does.

Privileges

• Velero: the node agent needs access to pod volumes on the host for file system backup, and data mover pods run as root to restore metadata. The Velero server needs wide read access to the API for backups, and wide write access for restores.
• K8up: backup jobs mount PVCs in your namespaces; PodConfig lets you apply a restricted pod security context where the storage allows it.
• Kanister: the docs say the default installation can only access resources in its own namespace, so it may not be able to back up applications elsewhere until you grant more access. That is a safer default, and one more thing to set up.

How a Kubernetes backup and restore actually works

It helps to see one full cycle, because most failures come from a step people did not know existed.

1. Select. The tool decides what is in scope: which namespaces, which resource types, which volumes.
2. Quiesce, if needed. A hook flushes or dumps the database so the next step captures something usable.
3. Capture objects. The tool lists the selected objects from the API server and writes them to object storage. With Velero this is the backup tarball.
4. Capture volume data. Either the storage system takes a snapshot, or a data mover reads a snapshot or a mounted volume and uploads the data to a deduplicated, encrypted repository.
5. Release. Post hooks unfreeze the application. Temporary snapshots, pods and PVCs are cleaned up.
6. Maintain. Repository maintenance and pruning remove expired data. Velero runs Kopia maintenance jobs every hour by default; K8up runs the prune and check jobs you schedule.
7. Restore. Objects are recreated in a sensible order, new PVCs are provisioned, data is written into them before the application pods start, and restore hooks run.

Step 7 is where the surprises are. API versions must exist on the target cluster. Storage classes and CSI driver names must match or be mapped. Webhooks and operators in the target cluster may react to restored objects before the data is in place.

A worked plan for the Dehradun team

Step 1: write down what belongs to whom

• PostgreSQL: owned by CloudNativePG’s own backups, with WAL archiving and point-in-time recovery. The database runs in its own db namespace, which the nightly schedule below does not include. A separate Velero schedule can keep a copy of the Cluster custom resource and its Secrets for reference, with --snapshot-volumes=false, but the database itself is restored with CloudNativePG’s own procedure. Our post on Patroni and CloudNativePG covers how that operator works.
• MariaDB attendance database: a dump taken by a pre-backup hook and captured with its volume.
• Uploads volume: CSI snapshot with data movement to object storage.
• All objects in the exams, portal and fees namespaces, including hand-made Secrets: Velero.
• The Sealed Secrets private key: exported once, stored offline in the company’s secret store, and also included in a Velero backup of its namespace.

Step 2: install Velero with an off-cluster bucket

The bucket sits with a different provider or region from the cluster. Velero needs delete rights on it to expire old backups, so add protection outside Velero: bucket versioning, or a scheduled copy to a second bucket whose credentials the cluster never sees. Install with the node agent, because the built-in data mover runs there:

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.14.4 \
  --bucket ddn-school-k8s-backup \
  --secret-file ./credentials-velero \
  --backup-location-config region=ap-south-1,s3ForcePathStyle="true",s3Url=https://objects.example.in \
  --use-node-agent \
  --features=EnableCSI

Before the first backup, replace the default repository password in the velero-repo-credentials Secret. Then label one VolumeSnapshotClass per CSI driver with velero.io/csi-volumesnapshot-class: "true" so Velero knows which one to use.

Step 3: make the MariaDB backup consistent

Put the hook in the StatefulSet’s pod template in Git, not on a running pod with kubectl annotate, so it survives restarts and Argo CD does not remove it:

template:
  metadata:
    annotations:
      pre.hook.backup.velero.io/container: mariadb
      pre.hook.backup.velero.io/command: '["/bin/sh", "-c", "mariadb-dump --single-transaction --all-databases -uroot -p\"$MARIADB_ROOT_PASSWORD\" > /var/lib/mysql/backup/attendance.sql"]'
      pre.hook.backup.velero.io/timeout: 10m

The dump lands on the same volume, so the snapshot taken next contains both the data files and a dump that is consistent by MariaDB’s own rules. On restore, if the data files do not start cleanly, the dump is there. Raise the timeout from the 30 second default; a dump of a real database will not finish in 30 seconds.

Step 4: schedule it

velero schedule create school-nightly \
  --schedule="CRON_TZ=Asia/Kolkata 30 1 * * *" \
  --include-namespaces exams,portal,fees,sealed-secrets \
  --snapshot-move-data \
  --ttl 336h0m0s

That runs at 1:30 AM IST every day and keeps each backup for 14 days. Add a weekly schedule with a longer TTL if the school calendar needs it, for example to keep the backup from the night before results are published. Watch the backup phase: Completed is what you want; PartiallyFailed means some items or volumes did not make it.

Step 5: practise the restore every month

Use a separate drill cluster if you have one; a separate namespace on the same cluster is the next best thing. Switch the backup location to read-only first, so a mistake during the drill cannot delete backups:

kubectl -n velero patch backupstoragelocation default \
  --type merge -p '{"spec":{"accessMode":"ReadOnly"}}'

velero restore create drill-exams-nov \
  --from-backup school-nightly-20261101200012 \
  --include-namespaces exams \
  --namespace-mappings exams:exams-drill

velero restore describe drill-exams-nov --details

Then check what matters to the school, not just what Kubernetes reports: does the report card service start, do the latest uploads open, does the attendance database have yesterday’s rows, does the fee service reach the payment gateway with the restored Secret. Write down how long each step took. That number is your real recovery time. Switch the location back to ReadWrite afterwards.

Step 6: what K8up or Kanister would look like instead

If the team did not need object backups, because everything is in Git and Secrets come from an external store, K8up would be a smaller tool to run. One Schedule per namespace, the uploads PVC backed up as files, and the MariaDB dump captured through an annotation:

apiVersion: k8up.io/v1
kind: Schedule
metadata:
  name: exams-backup
  namespace: exams
spec:
  backend:
    repoPasswordSecretRef:
      name: k8up-repo
      key: password
    s3:
      endpoint: https://objects.example.in
      bucket: ddn-exams-restic
      accessKeyIDSecretRef:
        name: k8up-s3
        key: username
      secretAccessKeySecretRef:
        name: k8up-s3
        key: password
  backup:
    schedule: '@daily-random'
  check:
    schedule: '@weekly-random'
  prune:
    schedule: '@weekly-random'
    retention:
      keepLast: 5
      keepDaily: 14

If the team had several databases with different backup procedures, Kanister would let a database owner write one Blueprint per engine and reuse it. Running the published MySQL example against the attendance StatefulSet would look like this:

kanctl create actionset --action backup --namespace kanister \
  --blueprint mysql-blueprint \
  --statefulset exams/attendance-db \
  --profile kanister/school-s3-profile

Both are reasonable. For this team, Velero wins because the incident was as much about missing objects and Secrets as about missing data.

etcd snapshots, and why they are a different tool

On a self-managed cluster, everything the API server knows is stored in etcd. The Kubernetes documentation recommends backing up etcd regularly, encrypting the snapshot files because they contain sensitive data, and gives etcdctl snapshot save for taking a snapshot and etcdutl snapshot restore for restoring it. It also warns that restoring etcd means stopping all API servers first, restoring every etcd member, and then restarting the API servers and other control plane components.

That makes an etcd snapshot the right tool for one job: getting back a control plane you have lost entirely. It is the wrong tool for “restore the exams namespace as it was last night”, because it rolls back the whole cluster state, contains no volume data, and only works on the same cluster setup. Keep both: etcd snapshots for the control plane, if you run it, and Velero or similar for namespaces and data. On a managed service, the provider handles etcd and you cannot take these snapshots at all, which makes object-level backup more important, not less.

Why GitOps does not cover this

Argo CD and Flux are excellent at bringing the cluster back to what Git says. Our post on Argo CD and Flux covers them. But look at what Git usually does not hold:

• Data in PersistentVolumes.
• Secrets created by hand, or values that only exist in the cluster.
• The private key of a Sealed Secrets controller. Without it, every sealed secret in Git is unreadable in a new cluster.
• Objects created at runtime by operators, such as certificates, generated passwords and database users.
• Anything changed with kubectl edit and never committed.

GitOps plus a backup tool is the right pair. During a restore, decide in advance who wins: let Velero restore data and the objects Git does not hold, and let the GitOps controller own the rest. If both try to create the same Deployment at the same time, you will spend the drill debugging the tools instead of the restore.

Operational costs people forget

• Storage and egress. Snapshot data moved to object storage costs money every month, and reading it back during a restore can cost egress. Deduplication and incremental backups help; Velero 1.18 now reports the incremental size per volume, which makes this easier to watch.
• Node resources. Data mover pods and K8up backup jobs read whole volumes and compress and encrypt the data. Set resource requests and run backups outside busy hours. Velero 1.18 added cache volumes for data mover pods during restore because ephemeral disk on nodes ran out.
• Repository maintenance. Kopia and restic repositories need regular maintenance to reclaim space. Make sure those jobs succeed and are not starved of memory.
• Upgrades. Velero ties plugin versions to server versions, and the restic retirement means old backups may need a plan before you upgrade. Read the upgrade notes for each minor release.
• Snapshot leftovers. Failed backups can leave VolumeSnapshots, temporary PVCs or pods behind. Check for them every week.
• People time. Someone must read the nightly backup status, run the monthly drill and fix what the drill finds. No tool does that part.

When something else is right

• The database has a better native tool. For PostgreSQL, a WAL-based tool or an operator’s built-in backup gives point-in-time recovery, which no volume snapshot can. Use the cluster tool around it, not instead of it.
• Your storage platform already does it. Some storage systems replicate volumes and snapshots to another site. That can cover volume data, but you still need object backups.
• You want a product with support and a UI. Commercial Kubernetes backup products exist, and several of them are built on these same projects. That is a buying decision; compare what you would pay against what a missed restore costs.
• The cluster is disposable and stateless. If every bit of state lives in managed databases and object storage outside the cluster, a cluster backup adds little. Back up those services instead, and keep the GitOps repository and secret store safe.

Decision guide

• You need objects and volumes backed up together, and cross-namespace or cross-cluster restore: Velero.
• You only need volume files and database dumps per namespace, want restic, and like one simple Schedule per team: K8up.
• You have several databases with specific backup procedures and want them written once as reusable workflows: Kanister, often together with Velero for the objects.
• Your CSI driver supports snapshots: prefer CSI snapshot data movement over file system backup in Velero.
• Your volumes have no snapshot support, such as NFS: file system backup in Velero, or K8up.
• Self-managed control plane: add etcd snapshots, encrypted and stored off the nodes.

A short checklist

• A written list of what must come back after a restore: objects, volumes, databases, Secrets, keys.
• Backups stored outside the cluster and outside the storage system they protect.
• A second copy, or bucket versioning, that the cluster’s credentials cannot delete, since the backup tool itself needs delete rights to expire old backups.
• Database backups owned by the database tool or operator; hooks or dumps for anything else stateful.
• Schedules with an explicit time zone, and retention that matches your business calendar.
• Alerts on failed and partially failed backups, and on backups that did not run at all.
• The Sealed Secrets key, or your secret store’s recovery material, stored separately.
• A monthly restore drill into a separate namespace or cluster, with recovery time written down.
• Old restic-based Velero backups tested or migrated before upgrading past 1.18.

Common mistakes

• Treating a CSI snapshot as a backup. If the snapshot lives on the same storage system as the volume, losing the storage loses both. Move the data out.
• Copying database files from under a running database. File system backup of a busy database volume can give you files that do not start. Use a dump, a hook or the database’s own tool.
• Annotating running pods by hand. The annotation disappears when the pod is replaced. Put hooks in the pod template in Git.
• Leaving hook timeouts at 30 seconds. Real dumps take longer, and with the default Fail behaviour the backup fails.
• Never looking at PartiallyFailed. A backup that is partly failed every night is a backup that does not cover what you think it covers.
• Restoring over the live namespace during a drill. Use namespace mapping or a separate cluster, and set the backup location to read-only first.
• Forgetting the reclaim policy. The Kubernetes docs say dynamically provisioned volumes inherit the reclaim policy of their StorageClass, which defaults to Delete. Deleting a namespace can then delete the data. For critical volumes, consider a StorageClass with Retain, alongside backups, not instead of them.

About a hands-on drill

No hands-on drill was run for this post. The commands and manifests above are written from each project’s documentation and published examples, and are meant as starting points. Try them on a test cluster with test data before relying on them in production.

Unlearn, re-learn

Unlearn the idea that GitOps means the cluster is backed up. Git brings back the shape of the cluster, not the data inside it or the secrets that were never committed. Unlearn, too, the idea that a snapshot sitting next to the volume is safe. Re-learn backup as three separate questions: which objects, which data, and which application rules for consistency. Then pick tools for each, keep the copies away from the cluster, and restore them on a calendar. The restore drill is the only part that proves any of the rest.

Sources

• Velero repository, README with compatibility matrix and releases
• Velero 1.18 docs: how Velero works, file system backup, CSI snapshot data movement, CSI support, backup hooks, restore hooks, restore reference, backup reference, repository maintenance
• Velero plugin for AWS
• CNCF: Velero Sandbox application, Velero onboarding issue, project pages for Velero, K8up and Kanister
• K8up repository, releases, OWNERS and governance
• K8up 2.16 docs: backup methods, backup pod scheduling, application-aware backups, PreBackupPods, schedules, schedule specification, restore, system requirements
• Kanister repository, releases and MAINTAINERS
• Kanister docs: architecture, functions, tooling, installation
• Kanister blueprints repository, including the MySQL blueprint
• Kopia releases and restic releases
• Kubernetes docs: volume snapshots and operating etcd clusters for Kubernetes