Backup and Restore an etcd Cluster on Ubuntu

If you followed our guide on setting up a 3-node etcd cluster on Ubuntu, you already have a working cluster storing your Kubernetes objects, service discovery data, or whatever critical configuration you decided to trust it with. What that guide does not cover is the question every etcd operator eventually has to answer under pressure: what happens when a node dies, a disk gets corrupted, or someone runs a bad etcdctl del against a prefix they should not have touched?

etcd does not have a recycle bin. A deleted key is gone, and if the cluster itself is unhealthy, there is no dashboard to click “undo” on. The only real safety net is a snapshot you took before the incident and a tested procedure for restoring it. This tutorial walks through exactly that: taking consistent snapshots with etcdctl snapshot save, verifying they are actually usable, restoring a snapshot to rebuild a failed member or an entire cluster, automating the whole thing with a systemd timer, and finally running a disaster recovery drill so you know the procedure works before you need it in production.

This guide is for sysadmins and DevOps engineers who already run etcd, whether standalone or as the backing store for Kubernetes, and want a backup strategy they can actually rely on. You should be comfortable with the Linux command line, have sudo access on your etcd nodes, and ideally have already worked through cluster setup, since this tutorial assumes a running 3-node cluster.

Conceptual Overview

Before touching commands, it helps to understand what a “backup” means in etcd terms, because it is not a file copy.

etcd stores all of its data in a single embedded key-value store backed by a file called db, usually under /var/lib/etcd/member/snap/db. You could, in theory, copy that file while etcd is running, but there is no guarantee the copy is internally consistent: etcd might be mid-write, and a raw file copy can capture a half-written state that will not open correctly later.

A snapshot solves this by asking etcd itself, through its API, for a point-in-time, transactionally consistent copy of the entire keyspace. The etcdctl snapshot save command does exactly this: it talks to a running etcd member over the client API and streams back a .db file that etcd guarantees is valid and complete, no matter what else is happening on the cluster at that moment.

A restore, on the other hand, does not “load” a snapshot into a running cluster the way you might restore a database dump into a live database. Instead, etcdctl snapshot restore creates a brand new, standalone data directory from the snapshot file, with a fresh cluster identity. You then point a new (or freshly wiped) etcd member at that data directory and start it up. This distinction matters: you cannot restore a snapshot “on top of” a running cluster member. You always restore into a new data directory and then bring etcd up against it.

Because of that, disaster recovery with etcd usually falls into two buckets: single member recovery, where one node in a healthy cluster dies and you rebuild it by re-joining an empty member (etcd’s normal replication handles catching it up, no snapshot needed), and full cluster recovery, where you have lost quorum (more than half the members) or the data itself is corrupted, and you need to bootstrap a brand new cluster from your latest snapshot.

Prerequisites

Before starting, make sure you have:

• A running etcd cluster on Ubuntu 22.04 or 24.04, ideally 3 nodes, set up following our etcd cluster guide (or the TLS-secured version, the steps below apply to both, just add your --cacert, --cert, and --key flags to the commands if TLS client authentication is enabled)
• sudo access on every etcd node
• The etcdctl binary available on each node (it ships alongside etcd if you installed from the official release tarball)
• At least 1 GB of free disk space on each node for snapshot storage
• Basic familiarity with systemd service management

For this tutorial, we will use a 3-node cluster with the following layout:

Adjust the IPs and hostnames to match your own environment.

Step-by-Step Hands-On Tutorial

Step 1: Set Environment Variables for etcdctl

etcd ships two API versions, and modern etcd defaults to API v3. Set this explicitly so every command below behaves consistently:

export ETCDCTL_API=3
export ENDPOINTS=10.20.0.11:2379,10.20.0.12:2379,10.20.0.13:2379

If your cluster uses TLS client authentication, also export the certificate paths so you do not have to repeat them on every command:

export ETCDCTL_CACERT=/etc/etcd/ssl/ca.pem
export ETCDCTL_CERT=/etc/etcd/ssl/etcd-client.pem
export ETCDCTL_KEY=/etc/etcd/ssl/etcd-client-key.pem

Confirm the cluster is healthy before you rely on it for a snapshot. A snapshot taken from an unhealthy member can still succeed, but you want to know you are starting from a good baseline:

etcdctl --endpoints=$ENDPOINTS endpoint health

You should see healthy printed for all three endpoints. If one node reports unhealthy, fix that first (see Troubleshooting below) before you build a backup workflow around it.

Step 2: Take a Manual Snapshot

Run this on any single healthy member, snapshotting from that member’s local endpoint is enough since all members hold the full dataset:

sudo mkdir -p /var/backups/etcd
sudo etcdctl --endpoints=https://10.20.0.11:2379 \
  snapshot save /var/backups/etcd/etcd-snapshot-$(date +%Y%m%d-%H%M%S).db

You will see output like this:

{"level":"info","ts":"2026-08-08T09:14:02.101Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/backups/etcd/etcd-snapshot-20260808-091402.db.part"}
{"level":"info","ts":"2026-08-08T09:14:02.145Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://10.20.0.11:2379"}
{"level":"info","ts":"2026-08-08T09:14:02.398Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://10.20.0.11:2379","size":"3.1 MB","took":"296.821ms"}
Snapshot saved at /var/backups/etcd/etcd-snapshot-20260808-091402.db

That .db file is your entire cluster’s keyspace, self-contained and portable. Copy it off the node immediately, onto another server, an object storage bucket, or wherever your backup retention policy points, since a backup that lives only on the machine it might fail alongside is not really a backup.

Step 3: Verify the Snapshot

Never trust a backup you have not test-read. etcdctl can inspect a snapshot’s internal status without restoring it, which is a fast sanity check to run right after every backup:

etcdctl --write-out=table snapshot status /var/backups/etcd/etcd-snapshot-20260808-091402.db

Output looks like this:

+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 4a1f8c2e |    58213 |        412 |     3.1 MB |
+----------+----------+------------+------------+

If this command errors out or reports zero keys when you expect data, the snapshot is not usable and you need to investigate before you rely on it. A REVISION number that keeps climbing between snapshots is a good sign your cluster is actively being written to and your backups are capturing real state.

Step 4: Automate Snapshots with a systemd Timer

Manual backups get forgotten. Wire this into a systemd timer so it runs unattended. First, create a small backup script:

sudo tee /usr/local/bin/etcd-backup.sh > /dev/null <<'EOF'
#!/bin/bash
set -euo pipefail

BACKUP_DIR=/var/backups/etcd
RETENTION_DAYS=7
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
SNAPSHOT_FILE="${BACKUP_DIR}/etcd-snapshot-${TIMESTAMP}.db"

export ETCDCTL_API=3
mkdir -p "${BACKUP_DIR}"

etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/etcd/ssl/ca.pem \
  --cert=/etc/etcd/ssl/etcd-client.pem \
  --key=/etc/etcd/ssl/etcd-client-key.pem \
  snapshot save "${SNAPSHOT_FILE}"

etcdctl --write-out=table snapshot status "${SNAPSHOT_FILE}"

find "${BACKUP_DIR}" -name 'etcd-snapshot-*.db' -mtime +${RETENTION_DAYS} -delete
EOF

sudo chmod +x /usr/local/bin/etcd-backup.sh

Drop the --cacert, --cert, and --key lines if your cluster does not use TLS client authentication. Next, create the systemd service and timer:

sudo tee /etc/systemd/system/etcd-backup.service > /dev/null <<'EOF'
[Unit]
Description=etcd snapshot backup

[Service]
Type=oneshot
ExecStart=/usr/local/bin/etcd-backup.sh
EOF

sudo tee /etc/systemd/system/etcd-backup.timer > /dev/null <<'EOF'
[Unit]
Description=Run etcd snapshot backup every 6 hours

[Timer]
OnCalendar=*-*-* 00/6:00:00
Persistent=true

[Install]
WantedBy=timers.target
EOF

Persistent=true matters here: if the node was down or rebooting when the timer was scheduled to fire, systemd will run it as soon as the system is back up instead of silently skipping that backup window. Enable and start it:

sudo systemctl daemon-reload
sudo systemctl enable --now etcd-backup.timer
sudo systemctl list-timers etcd-backup.timer

You should see the next scheduled run time in the output. Run sudo journalctl -u etcd-backup.service after the first execution to confirm it completed cleanly.

Step 5: Restore a Single Failed Member (No Snapshot Needed)

If your cluster still has quorum (at least 2 out of 3 members alive and talking to each other) and you just lost one node’s disk, you do not need a snapshot at all. Remove the dead member from the cluster, wipe its data directory, and re-add it as a fresh member. etcd’s own replication will stream the current state to it automatically.

On a surviving node, find the dead member’s ID and remove it:

etcdctl --endpoints=$ENDPOINTS member list
etcdctl --endpoints=$ENDPOINTS member remove 8211f1d0f64f3269

On the failed node, wipe its old data directory and re-add it:

sudo systemctl stop etcd
sudo rm -rf /var/lib/etcd/member
etcdctl --endpoints=$ENDPOINTS member add etcd-03 --peer-urls=https://10.20.0.13:2380
sudo systemctl start etcd

Watch etcdctl endpoint health on all three endpoints until the rejoined node reports healthy. This is the fast path, and it is why quorum matters so much: as long as you have not lost a majority of members, you rarely need to touch a snapshot at all.

Step 6: Full Cluster Restore from Snapshot

This is the scenario a snapshot exists for: you have lost quorum, or the data itself is suspect, and you need to bootstrap an entirely new cluster from your last known good backup. Run this on every node that will be part of the new cluster, using the same snapshot file copied to each one.

Stop etcd everywhere first:

sudo systemctl stop etcd

On each node, restore the snapshot into a fresh data directory, giving each node its own name, initial cluster string, and this node’s own peer URL:

sudo etcdctl snapshot restore /var/backups/etcd/etcd-snapshot-20260808-091402.db \
  --name etcd-01 \
  --initial-cluster etcd-01=https://10.20.0.11:2380,etcd-02=https://10.20.0.12:2380,etcd-03=https://10.20.0.13:2380 \
  --initial-cluster-token etcd-cluster-restored \
  --initial-advertise-peer-urls https://10.20.0.11:2380 \
  --data-dir /var/lib/etcd-restored

Adjust --name and --initial-advertise-peer-urls for each node to match that node’s own identity, but keep --initial-cluster and --initial-cluster-token identical across all three, this is what tells them they belong to the same restored cluster.

Once restored on every node, swap the data directory and start etcd:

sudo systemctl stop etcd
sudo mv /var/lib/etcd /var/lib/etcd.old
sudo mv /var/lib/etcd-restored /var/lib/etcd
sudo chown -R etcd:etcd /var/lib/etcd
sudo systemctl start etcd

Do this on all three nodes within a short window of each other, then check cluster health:

etcdctl --endpoints=$ENDPOINTS endpoint health
etcdctl --endpoints=$ENDPOINTS member list

Once all members report healthy, spot-check that your data is actually there:

etcdctl --endpoints=$ENDPOINTS get / --prefix --keys-only | head

Only delete /var/lib/etcd.old on each node once you have confirmed the restored cluster is fully healthy and serving the data you expect.

Common Mistakes & Troubleshooting

“snapshot save” hangs or times out. This usually means the endpoint you pointed it at is unreachable or the member is unhealthy. Run etcdctl endpoint health against that specific endpoint first, and try snapshotting from a different member if one is available.

Restoring the same snapshot on all nodes without changing --name or --initial-advertise-peer-urls. Every node needs its own identity even though they share the same underlying data. If two nodes end up with the same name or peer URL, the cluster will refuse to form correctly and you will see repeated “member already exists” or peer connection errors in the logs.

Forgetting to stop etcd before restoring. snapshot restore builds a new data directory and does not touch a running etcd process, but if you restore into the same path an active etcd instance is already using, you will end up with stale file handles and a directory etcd cannot cleanly start against. Always restore into a new directory, then swap it in after stopping the service.

Losing quorum and reaching for a snapshot restore when a single member rejoin would have worked. Check member list and count healthy members first. A full cluster restore should be your last resort, not your default recovery path, since it discards any writes that happened after the snapshot was taken.

Backups piling up with no retention policy. Snapshot files are small individually but add up, and more importantly, a directory full of snapshots nobody has verified in months gives false confidence. The find ... -mtime +7 -delete line in the backup script above keeps only a week of local snapshots; pair that with off-node copies for longer retention.

Best Practices

• Copy snapshots off the etcd node immediately. A backup stored only on the machine that might catch fire with the original data is not a backup, it is a second copy of the same risk. Sync backups to a separate host, an S3-compatible bucket (see our guide on self-hosted S3 storage with MinIO if you want to run your own target), or your cloud provider’s object storage.
• Verify snapshots automatically, not just on the day you need them. Run snapshot status right after every backup, as shown in Step 4, and alert if it fails or reports zero keys.
• Take a snapshot before any risky maintenance, such as a Kubernetes version upgrade, a certificate rotation, or a bulk key deletion. A five-second snapshot save before a risky change is cheap insurance.
• Practice the restore procedure on a non-production cluster periodically. A backup strategy you have never tested restoring is a guess, not a plan. Spin up three throwaway VMs, restore your latest production snapshot into them, and confirm the data is what you expect.
• Keep your snapshot retention aligned with your actual recovery point objective. If losing 6 hours of writes is unacceptable, a nightly-only backup schedule will not save you, tighten the timer interval in Step 4 accordingly.
• Secure the backup files themselves. A snapshot contains your full etcd keyspace, which for a Kubernetes cluster includes Secrets in plaintext (etcd’s own encryption at rest, if configured, is preserved in the snapshot, but an unencrypted cluster’s secrets are fully readable from the .db file). Restrict filesystem permissions on /var/backups/etcd and encrypt backups before shipping them off-site.

Conclusion

You now have a complete etcd backup and recovery workflow: manual and automated snapshots with etcdctl snapshot save, a verification step so you catch bad backups before you need them, the fast single-member rejoin path for the common case of losing one node with quorum intact, and the full snapshot restore procedure for the worst case of losing quorum entirely. Combined with the systemd timer from Step 4, backups now happen on a schedule instead of “whenever someone remembers.”

The next thing worth doing is not more etcd configuration, it is discipline: schedule a quarterly recovery drill on a disposable set of VMs, and treat a failed drill as a bug to fix, not a bad day. If your etcd cluster backs a Kubernetes control plane, also read through our guide on renewing Kubernetes certificates, since certificate expiry and etcd data loss are the two failure modes most likely to take down a control plane at the worst possible time, and both are far less scary once you have practiced fixing them before they happen for real.