How to Back Up and Restore Kubernetes with Velero

In this tutorial, you'll learn about How to Back Up and Restore Kubernetes with Velero. We cover key concepts, practical examples, and best practices.

The Problem

Your Velero backup fails with Backup failed: error getting volume snapshotter, the restore only restores some resources, or the backup is stuck in InProgress for hours. Without reliable backups, cluster data is at risk during disasters.

Quick Fix

Fix 1: Backup Stuck InProgress

velero get backup daily-backup
# NAME           STATUS       CREATED                         EXPIRES
# daily-backup   InProgress   2024-03-15 10:00:00 +0000 UTC   ...
# (stuck for 30+ minutes)

RIGHT — check the backup logs:

velero backup logs daily-backup | tail -20
# If stuck on a specific resource, verify the resource exists:
kubectl get <resource> -n <namespace>

Cancel and retry with excludes:

velero backup delete daily-backup
velero create backup daily-backup --exclude-resources jobs,events

Fix 2: Volume Snapshot Provider Not Found

# Error: error getting volume snapshotter: no volume snapshotter configured

WRONG — no VolumeSnapshotLocation configured:

velero snapshot-location get
# No snapshot locations found

RIGHT — configure the volume snapshot location:

# For AWS:
velero snapshot-location create aws-snapshot \
    --provider aws \
    --config region=us-east-1

# For GCP:
velero snapshot-location create gcp-snapshot \
    --provider gcp

Fix 3: Backup Fails with Permission Error

# Error: AccessDenied: Access Denied
# (the cloud provider credentials lack permissions)

RIGHT — verify IAM permissions:

# AWS:
aws iam list-attached-role-policies --role-name velero-role
# Expected: VeleroBackupPolicy with s3:PutObject, s3:GetObject, etc.

# GCP:
gcloud projects get-iam-policy PROJECT_ID --flatten="bindings[].members" --format="table(bindings.role)"

Fix 4: Restore Only Partial Resources

velero restore create --from-backup daily-backup
# Restore completed with 45 of 50 resources restored

RIGHT — check restore details:

velero restore describe restore-20240315
# Warnings:
#   - namespace "custom-ns" not found, skipping resources
#   - pvc "data-pvc" already exists, skipping

Create missing namespaces first:

kubectl create namespace custom-ns
velero restore create --from-backup daily-backup --namespace-mappings custom-ns:custom-ns

Fix 5: Schedule Not Running

velero get schedules
# NAME           STATUS    CREATED
# daily-backup   Enabled   2024-03-14
# But no backups are created

RIGHT — check the schedule definition:

velero schedule get daily-backup -o yaml
# Ensure the schedule syntax is correct:
spec:
  schedule: "0 2 * * *"  # daily at 2 AM
  template:
    includedNamespaces:
    - "*"
    ttl: 720h  # 30 days

Fix 6: Restore to Different Cluster

# Restore to a different cluster with different storage class:
velero restore create --from-backup daily-backup \
    --storage-class-mappings gp2:gp3 \
    --namespace-mappings prod:staging

Use DodaTech's Disaster Recovery Planner to design backup schedules, test restore procedures, and validate RPO/RTO requirements.

Prevention

• Schedule regular backups with velero schedule create.
• Test restores in a non-production environment monthly.
• Monitor backup status with alerts on failures.
• Store backups in a different region from the cluster.
• Document restore procedures before they are needed.

Common Mistakes with velero backup

1. Using return to exit a function early instead of wrapping a pure value in the monad
2. Mixing let bindings with <- bindings in do notation, producing type errors
3. Overlapping type class instances that cause GHC to reject the program with ambiguous dispatch errors

These mistakes appear frequently in real-world K8S code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ