In the recent 1.4 release of Velero, we announced a new feature of supporting CSI snapshotting using the Kubernetes CSI Snapshot Beta APIs. With this capability of CSI volume snapshotting, Velero can now support any volume provider that has a CSI driver with snapshotting capability, without requiring a Velero-specific volume snapshotter plugin to be available.
This post has the necessary instructions for you to start using this feature.
Using the CSI volume snapshotting features in Velero involves the following steps.
csi-app: a stateful application that uses CSI backed volumes that we will backup and restore.
Kubernetes CSI Snapshot Beta API is available starting from Kubernetes
1.17, you need to run Kubernetes
1.17 or later.
This post uses an AKS cluster running Kubernetes
1.17, with Azure disk CSI driver as an example.
Following instructions to install the Azure disk CSI driver from here run the below command
curl -skSL https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/deploy/install-driver.sh | bash -s master snapshot --
This script will deploy the following CSI components, CRDs, and necessary RBAC:
The CSI volume snapshot capability is currently, as of Velero 1.4, a beta feature behind the
EnableCSI feature flag and is not enabled by default.
Following instructions from our docs website, install Velero with the velero-plugin-for-csi and using the Azure Blob Store as our BackupStorageLocation. Please refer to our velero-plugin-for-microsoft-azure documentation for instructions on setting up the BackupStorageLocation. Please note that the BackupStorageLocation should be set up before installing Velero.
Install Velero by running the below command
velero install \ --provider azure \ --plugins velero/velero-plugin-for-microsoft-azure:v1.1.0,velero/velero-plugin-for-csi:v0.1.1 \ --bucket $BLOB_CONTAINER \ --secret-file <PATH_TO_CREDS_FILE>/aks-creds \ --backup-location-config resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID \ --snapshot-location-config apiTimeout=5m,resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID \ --image velero/velero:v1.4.0 \ --features=EnableCSI
Before installing the stateful application with CSI backed volumes, install the storage class and the volume snapshot class for the Azure disk CSI driver by applying the below
yaml to our cluster.
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: disk.csi.azure.com provisioner: disk.csi.azure.com parameters: skuname: StandardSSD_LRS reclaimPolicy: Delete volumeBindingMode: Immediate allowVolumeExpansion: true --- apiVersion: snapshot.storage.k8s.io/v1beta1 kind: VolumeSnapshotClass metadata: name: csi-azuredisk-vsc driver: disk.csi.azure.com deletionPolicy: Retain parameters: tags: 'foo=aaa,bar=bbb'
NOTE: The above
yaml was sourced from
Deploy the stateful application that is using CSI backed PVCs, in the
csi-app namespace by applying the below yaml.
apiVersion: v1 kind: Namespace metadata: creationTimestamp: null name: csi-app --- kind: Pod apiVersion: v1 metadata: namespace: csi-app name: csi-nginx spec: nodeSelector: kubernetes.io/os: linux containers: - image: nginx name: nginx command: [ "sleep", "1000000" ] volumeMounts: - name: azuredisk01 mountPath: "/mnt/azuredisk" volumes: - name: azuredisk01 persistentVolumeClaim: claimName: pvc-azuredisk --- apiVersion: v1 kind: PersistentVolumeClaim metadata: namespace: csi-app name: pvc-azuredisk spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: disk.csi.azure.com ---
For demonstration purposes, instead of relying on the application writing data to the mounted CSI volume, exec into the pod running the stateful application to write data into
/mnt/azuredisk, where the CSI volume is mounted.
This is to let us get a consistent checksum value of the data and verify that the data on restore is exactly same as that in the backup.
$ kubectl -n csi-app exec -ti csi-nginx bash root@csi-nginx:/# while true; do echo -n "FOOBARBAZ " >> /mnt/azuredisk/foobar; done ^C root@csi-nginx:/# cksum /mnt/azuredisk/foobar 2279846381 1726530 /mnt/azuredisk/foobar
Back up the
csi-app namespace by running the below command
$ velero backup create csi-b2 --include-namespaces csi-app --wait Backup request "csi-b2" submitted successfully. Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup will continue in the background. .................. Backup completed with status: Completed. You may check for more information using the commands `velero backup describe csi-b2` and `velero backup logs csi-b2`.
Before restoring from the backup simulate a disaster by running
kubectl delete ns csi-app
Once the namespace has been deleted, restore the
csi-app from the backup
$ velero create restore --from-backup csi-b2 --wait Restore request "csi-b2-20200518085136" submitted successfully. Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background. .... Restore completed with status: Completed. You may check for more information using the commands `velero restore describe csi-b2-20200518085136` and `velero restore logs csi-b2-20200518085136`.
Now that the restore has completed and our
csi-nginx pod is
Running, confirm that contents of
/mnt/azuredisk/foobar have been correctly restored.
$ kubectl -n csi-app exec -ti csi-nginx bash root@csi-nginx:/# cksum /mnt/azuredisk/foobar 2279846381 1726530 /mnt/azuredisk/foobar root@csi-nginx:/#
The stateful application that we deployed has been successfully restored with its data intact. And that’s all it takes to backup and restore a stateful application that uses CSI backed volumes!
Please try out the CSI support in Velero 1.4. Feature requests, suggestions, bug reports, PRs are all welcome.
Get in touch with us on Kubernetes Slack #velero, Twitter, or Our weekly community calls
More details about CSI volume snapshotting and its support in Velero may be found in the following links: