Kubernetes Storage — Persistent Volumes, Persistent Volume Claims, and CSI Drivers Explained

In this tutorial, you'll learn about Kubernetes Storage. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Kubernetes storage is managed through PersistentVolumes (PVs), PersistentVolumeClaims (PVCs), and StorageClasses that abstract infrastructure details from application developers, enabling stateful workloads like databases and file stores to run reliably on any platform.

What You'll Learn

Why It Matters

Container storage is ephemeral by default -- data disappears when the Pod restarts. Stateful applications like PostgreSQL, Redis, and file servers require persistent storage that survives Pod crashes, rescheduling on different nodes, and cluster upgrades. Without understanding PVs, PVCs, and CSI drivers, teams either lose data during restarts or hard-code node-specific paths that break portability.

Real-World Use

DodaTech runs Durga Antivirus Pro's threat intelligence database on a StatefulSet with 100GB PVCs backed by AWS EBS CSI driver, with daily snapshots and cross-region Replication. The storage configuration is identical across dev, staging, and production -- only the StorageClass changes.

flowchart TD
    A["Developer creates PVC"] --> B["Kubernetes API"]
    B --> C{"StorageClass exists?"}
    C -->|"Yes"| D["Dynamic Provisioner"]
    C -->|"No"| E["Static PV required"]
    D --> F["CSI Driver provisions volume"]
    F --> G["PersistentVolume created"]
    G --> H["PVC binds to PV"]
    H --> I["Pod mounts volume"]
    I --> J["Application reads/writes data"]
    E --> K["Administrator creates PV"]
    K --> G
    style D fill:#326CE5,color:#fff
    style F fill:#326CE5,color:#fff

Static Provisioning -- Manual PV Creation

In static provisioning, the cluster administrator creates PersistentVolumes that map to pre-existing storage volumes. The PVC then binds to an available PV that matches its requirements.

# static-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: manual-pv
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manual
  hostPath:
    path: /data/k8s-pv

# static-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: manual

# pod-using-pvc.yaml
apiVersion: v1
kind: Pod
metadata:
  name: storage-pod
spec:
  containers:
    - name: app
      image: nginx:1.25-alpine
      volumeMounts:
        - name: data
          mountPath: /usr/share/nginx/html
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: data-claim

Expected behavior: The PVC data-claim binds to PV manual-pv because they match in storage size (10Gi) and access mode (ReadWriteOnce). The Pod mounts the volume at /usr/share/nginx/html. Data written by the Pod persists even if the Pod is deleted and recreated.

# Apply the manifests
kubectl apply -f static-pv.yaml -f static-pvc.yaml -f pod-using-pvc.yaml

# Expected output:
# persistentvolume/manual-pv created
# persistentvolumeclaim/data-claim created
# pod/storage-pod created

# Verify binding
kubectl get pv,pvc

# Expected output:
# NAME                        CAPACITY   ACCESS MODES   STATUS   CLAIM
# persistentvolume/manual-pv  10Gi       RWO            Bound    default/data-claim
# NAME                              STATUS   VOLUME     CAPACITY
# persistentvolumeclaim/data-claim  Bound    manual-pv  10Gi

Dynamic Provisioning with StorageClass

Dynamic provisioning eliminates manual PV creation. The StorageClass defines the provisioner (CSI driver) and parameters for creating volumes on demand.

# storage-class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
  encrypted: "true"
  csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

# dynamic-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dynamic-data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: fast-ssd

Expected behavior: When the PVC is created, the CSI driver (EBS in this case) provisions a new 50GB gp3 volume. The WaitForFirstConsumer binding mode delays volume creation until a Pod that uses the PVC is scheduled, ensuring the volume is created in the same availability zone as the Pod.

# Apply the PVC
kubectl apply -f dynamic-pvc.yaml

# Expected output:
# persistentvolumeclaim/dynamic-data created

# Check status (may show Pending until a Pod uses it)
kubectl get pvc dynamic-data

# Expected output (before Pod creation):
# NAME           STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS
# dynamic-data   Pending                                      fast-ssd

# After a Pod consumes the PVC:
# NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES
# dynamic-data   Bound   pvc-abc123-def456-7890-xxxx                50Gi       RWO

Access Modes and Volume Modes

Volume mode determines whether the volume is presented as a filesystem (Filesystem) or a raw block device (Block).

StatefulSet with Storage

StatefulSets provide stable network identities and persistent storage for stateful applications. Each replica gets its own PVC.

# statefulset-storage.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16-alpine
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-secret
                  key: password
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 50Gi
        storageClassName: fast-ssd

Expected behavior: The StatefulSet creates postgres-0, postgres-1, and postgres-2. Each Pod has a unique PVC named data-postgres-0, data-postgres-1, data-postgres-2. When a Pod is rescheduled to a different node, it reconnects to the same PVC, preserving its data.

# Deploy the StatefulSet
kubectl apply -f statefulset-storage.yaml

# Expected output:
# statefulset.apps/postgres created

# Verify PVCs per Pod
kubectl get pvc

# Expected output:
# NAME                 STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS
# data-postgres-0      Bound    pvc-xxx  50Gi       RWO            fast-ssd
# data-postgres-1      Bound    pvc-yyy  50Gi       RWO            fast-ssd
# data-postgres-2      Bound    pvc-zzz  50Gi       RWO            fast-ssd

CSI Drivers

The Container Storage Interface allows any storage vendor to provide a Kubernetes volume plugin without modifying Kubernetes core code. Popular CSI drivers include AWS EBS, GCE Persistent Disk, Azure Disk, NFS, Ceph RBD, and Portworx.

# csi-driver-example.yaml
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: ebs.csi.aws.com
spec:
  attachRequired: true
  podInfoOnMount: false
  volumeLifecycleModes:
    - Persistent
    - Ephemeral

Common Errors

1. PVC stuck in Pending state: This typically means no PV matches the PVC request or the StorageClass provisioner is not installed. Check kubectl describe pvc <name> for events. Common causes: missing CSI driver, insufficient storage capacity, or access mode mismatch.

2. Volume mounted but Pod crashes with permission errors: The container user (often non-root) may not own the mounted volume directory. Set the fsGroup in the Pod's securityContext to change volume ownership at mount time.

3. Using ReadWriteMany with a CSI driver that does not support it: Not all storage backends support ReadWriteMany (RWX). AWS EBS is RWO only -- attempting RWX on EBS causes mount failures. Use EFS or NFS for RWX workloads.

4. Deleting PVC before deleting the StatefulSet: If a PVC is deleted while the StatefulSet still references it, the underlying PV is also deleted (if reclaimPolicy: Delete), causing permanent data loss. Always scale down the StatefulSet to 0 replicas before deleting PVCs.

5. StorageClass volumeBindingMode set to Immediate for regional clusters: With Immediate binding, a PVC binds to a PV in a random zone. When the Pod later schedules to a different zone, it cannot mount the volume. Always use WaitForFirstConsumer for zonal volume provisioning.

Practice Questions

1. What is the difference between a PersistentVolume and a PersistentVolumeClaim? Answer: A PV is cluster-wide storage resource provisioned by an administrator or dynamically by a StorageClass. A PVC is a request for storage by a user. The PVC binds to a PV that matches its size, access mode, and storage class requirements.

2. What happens to a PV when its PVC is deleted? Answer: It depends on the persistentVolumeReclaimPolicy. Retain keeps the PV and its data (manual cleanup required). Delete removes both the PV object and the underlying storage volume. Recycle (deprecated) scrubs the volume and makes it available again.

3. How does WaitForFirstConsumer volume binding work? Answer: The PVC remains unbound until a Pod that uses it is scheduled to a node. At that point, the provisioner creates the volume in the same availability zone as the node. This ensures the volume is accessible from where the Pod runs.

4. Why does a StatefulSet use volumeClaimTemplates instead of a regular PVC? Answer: volumeClaimTemplates ensure each StatefulSet replica gets its own unique PVC with a predictable name (<volume-name>-<statefulset-name>-<ordinal>). This guarantees that when a Pod is rescheduled, it reconnects to its original data volume.

Challenge

Deploy a WordPress site with MySQL backend using Kubernetes storage. The MySQL Pod should use a StatefulSet with 10GB persistent storage via volumeClaimTemplates and a fast-ssd StorageClass. The WordPress Pod should use a Deployment with a 5GB PVC shared via ReadWriteMany (use an NFS CSI driver or local NFS provisioner). Configure fsGroup in the Pod securityContext so both containers can write to their mounted volumes. Verify data persistence by deleting both Pods and confirming data remains after recreation.

Mini Project

Set up a production-grade storage infrastructure: install the AWS EBS CSI driver on an EKS cluster, configure two StorageClasses (standard with gp3 volumes, high-iops with io2 volumes at 5000 IOPS), deploy a Cassandra StatefulSet with 3 replicas using volumeClaimTemplates and the high-iops StorageClass, deploy a file server using NFS CSI driver with ReadWriteMany access, set up daily volume snapshots using VolumeSnapshot and VolumeSnapshotClass, test a restore by creating a new PVC from a snapshot, and document the storage architecture with access modes and reclaim policies for each workload type.

Related Resources

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.