Persistent Volumes and PersistentVolumeClaims

Introduction & Overview

Understand Kubernetes PersistentVolumes and PVCs: static vs dynamic provisioning, access modes, reclaim policies, expansion, and why a PVC stays Pending.

A PersistentVolume (PV) is storage that exists in the cluster. A PersistentVolumeClaim (PVC) is a namespaced request for some of it. A pod mounts the claim, never the volume.

The indirection is the point: an application manifest asks for "20Gi, ReadWriteOnce, fast" and works unchanged whether that resolves to an EBS volume, a Ceph RBD image, or a directory on a laptop.

Scope and ownership

Because a PV is cluster-scoped and a PVC is namespaced, a PVC in staging can never bind a PV already claimed by production. Bindings are exclusive and one-to-one.

Static versus dynamic provisioning

Static means an administrator creates PVs ahead of time and claims bind to whichever one fits. You will see this in tutorials and on bare metal, and rarely anywhere else:

Dynamic means a StorageClass creates the PV on demand when a PVC appears. This is how essentially every real cluster works:

Omitting storageClassName entirely uses the cluster's default StorageClass. Setting it to "" explicitly disables dynamic provisioning and forces binding to a pre-created PV. A distinction that catches people out, because the two are not the same thing.

Pod → PVC → PV → Storage backend

cluster-wide resource

apiVersion: v1
kind: PersistentVolume
metadata:
  name: demo-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /data/demo-pv        # local clusters only, never production

Binding states

Released is the state worth understanding. A released PV is not reusable as-is: it still carries a claimRef pointing at the deleted PVC, so no new claim can bind to it even though the data is intact. Recovering it means clearing that reference. See below.

kubectl get pv,pvc

Access modes

Two things people get wrong here. RWO is a node constraint, so several pods on the same node can share an RWO volume, which makes the failure look intermittent, working until the scheduler spreads replicas. And block storage cannot do RWX at all: EBS, GCP Persistent Disk and Azure Disk are RWO only, so shared read-write requires EFS, Azure Files, Filestore or NFS.

ReadWriteOncePod is the strict one, useful when even two pods on the same node writing concurrently would corrupt data.

Reclaim policies

Dynamic provisioning defaults to Delete on nearly every cloud StorageClass. That means deleting a namespace destroys its database volumes with it, permanently.

For anything valuable, use a StorageClass with reclaimPolicy: Retain. You can also flip an existing PV:

This is a five-second change that has saved a great many people.

kubectl patch pv pvc-a1b2 -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

Expanding a volume

Requirements and caveats:

kubectl patch pvc payments-data -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'

Recovering data from a Released PV

The scenario: someone deleted a PVC, the policy was Retain, and the data is still on the volume. The PV is Released and will not bind to anything.

# 1. Confirm the data is still there
kubectl get pv pvc-a1b2 -o yaml

# 2. Remove the stale claim reference so it returns to Available
kubectl patch pv pvc-a1b2 -p '{"spec":{"claimRef": null}}'

# 3. Create a PVC that targets it explicitly

A PVC stuck in Terminating

Almost always a finalizer waiting on something still using the volume.

The kubernetes.io/pvc-protection finalizer holds the PVC while any pod still references it. Find and remove the consumer:

Force-removing the finalizer is possible and is a genuinely bad idea unless you are certain nothing is attached. You can end up with a volume attached to a node that Kubernetes no longer tracks.

kubectl get pvc payments-data -o jsonpath='{.metadata.finalizers}'

Storage for StatefulSets

A Deployment sharing one PVC does not scale past one replica on RWO storage. When each replica needs its own volume, that is a StatefulSet with a volumeClaimTemplate:

This creates data-postgres-0, data-postgres-1, data-postgres-2, each bound to its own volume and reattached to the same pod ordinal after rescheduling.

One thing to know: deleting a StatefulSet does not delete its PVCs. That is deliberate protection, and it also means scaling down then up reuses the old data, and that orphaned PVCs keep costing money until someone removes them.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 3
  selector:
    matchLabels: { app: postgres }
  template:
    metadata:
      labels: { app: postgres }
    spec:
      containers:
        - name: postgres
          image: postgres:17
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: gp3
        resources:
          requests:
            storage: 20Gi

Key takeaways

A PVC was deleted by mistake. The PV has reclaimPolicy Retain and shows status Released, and the data is intact. A new PVC with identical specs stays Pending. Why?

You delete a StatefulSet that ran three Postgres replicas with a volumeClaimTemplate. What happens to the three PVCs?