Handling PV Tantrum — Unable to attach or mount volumes

Sometimes the persistent volumes won’t be attached and your StatefulSet won’t run.

Unable to attach or mount volumes: unmounted volumes=[ingester-data ingester-wal], unattached volumes=[ingester-data ingester-wal loki overrides kube-api-access-fkfld]: timed out waiting for the condition

In this case, you can try delete the VolumeAttachement and delete the hanging Pod.

# Find the VA with the PV name
$ k get volumeattachment | grep csi-22f441d0-e18b-4f83-852b-b2f410ca81f3
NAME ATTACHER  PV NODE ATTACHED   AGE
csi-0227e14bf244fa6836e60df779411a8a73e5da91727e89ad01f928233f85cd18   blockvolume.csi.oraclecloud.com   csi-bafd73d0-43c9-4d6c-96c2-63ef8aa69fba   10.0.21.70    true       3h3m
csi-040393cddea13b7c19ea17fb5266c4c252ea7975ea2077d5b75524e3a141c52d   blockvolume.csi.oraclecloud.com   csi-cf4a21f6-09fa-4c13-9f4d-f00222432bc6   10.0.19.228   true       3h3m
csi-1c3fadea68bb837de152639da3ea7b44d2e9c20c7bd42b2ff562de491a6eaa69   blockvolume.csi.oraclecloud.com   csi-22f441d0-e18b-4f83-852b-b2f410ca81f3   10.0.19.228   true       26m=

# And delete the VA
$ kubectl delete volumeattachment/csi-1c3fadea68bb837de152639da3ea7b44d2e9c20c7bd42b2ff562de491a6eaa69
# Then delete the stuck Pod and wait.