Kubernetes etcd Backup and Restore Error Fix

In this tutorial, you'll learn about Kubernetes etcd Backup and Restore Error Fix. We cover key concepts, practical examples, and best practices.

The Problem

Backing up or restoring etcd fails with:

Error: context deadline exceeded

Or:

Error:  error connecting to etcd: dial tcp 127.0.0.1:2379: connect: connection refused

Etcd is the backing store for all Kubernetes objects. A failed backup means you cannot recover from cluster disasters. A failed restore can leave the cluster in a broken state.

Quick Fix

Step 1: Check etcd is running

# On the control plane node
systemctl status etcd

● etcd.service - etcd - highly-available key value store
   Loaded: loaded (/lib/systemd/system/etcd.service; enabled; vendor preset: enabled)
   Active: active (running) since ...

If etcd is not running:

sudo systemctl restart etcd

Check logs:

sudo journalctl -u etcd --no-pager -n 50

Step 2: Verify etcd endpoint and health

# Check the etcd endpoint
ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt
ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key

etcdctl endpoint health --cluster

https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 4.345ms

WRONG -- connection refused:

Error:  dial tcp 127.0.0.1:2379: connect: connection refused

RIGHT -- etcd is listening on the correct interface:

# Check what interfaces etcd binds to
ss -tlnp | grep 2379

If etcd listens on 0.0.0.0:2379 but you connect to 127.0.0.1, change the endpoint to the correct IP.

Step 3: Take a snapshot

sudo ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot-$(date +%Y%m%d).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Snapshot saved at /backup/etcd-snapshot-20260624.db

WRONG -- missing TLS certificates:

Error: open /etc/kubernetes/pki/etcd/server.crt: no such file or directory

RIGHT -- locate the correct certificate paths:

# Find etcd certs
ls /etc/kubernetes/pki/etcd/
# ca.crt  server.crt  server.key

Step 4: Verify the snapshot

sudo ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-snapshot-20260624.db

da39a3ee5e6b4b0d3255bfef95601890afd80709, 0, 0, 0 KB

A zero-size snapshot means the backup failed silently. Check disk space:

df -h /backup

Step 5: Restore from snapshot

sudo ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-snapshot-20260624.db \
  --data-dir=/var/lib/etcd-restored \
  --initial-cluster=master=https://127.0.0.1:2380 \
  --initial-cluster-token=etcd-cluster \
  --initial-advertise-peer-urls=https://127.0.0.1:2380

Then point etcd to the restored data:

sudo systemctl stop etcd
sudo mv /var/lib/etcd /var/lib/etcd.old
sudo mv /var/lib/etcd-restored /var/lib/etcd
sudo systemctl start etcd

WRONG -- restoring to the same data directory:

sudo ETCDCTL_API=3 etcdctl snapshot restore snapshot.db --data-dir=/var/lib/etcd

This fails because the directory is not empty or contains old data.

RIGHT -- restore to a new directory, then swap.

Step 6: Set up automated backups

# Add to crontab
0 */6 * * * ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +\%Y\%m\%d-\%H\%M).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Use DodaTech's Backup Manager to schedule etcd snapshots, verify integrity, and restore with a single command.

Prevention

• Take etcd snapshots every 6 hours.
• Store backups off-cluster (S3, NFS, or object storage).
• Verify snapshot integrity with etcdctl snapshot status.
• Test restores in a staging cluster at least once per quarter.
• Use at least 3 etcd members for HA in production.
• Secure etcd client-server communication with TLS.

Common Mistakes with etcd backup

1. Non-exhaustive pattern matches that compile with warnings then crash at runtime
2. Misunderstanding that String is [Char] with poor performance for large text operations
3. Using foldl instead of foldl' causing stack overflow on large lists

These mistakes appear frequently in real-world K8S code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ