Velero serves to both restore and migrate Kubernetes applications. Typically, backup and restore does not involve upgrading Kubernetes API group versions. However, when migrating from a source cluster to a destination cluster, it is not unusual to see the API group versions differing between clusters.
NOTE: Kubernetes applications are made up of various resources. Common resources are pods, jobs, and deployments. Custom resources are created via custom resource definitions (CRDs). Every resource, whether custom or not, is part of a group, and each group has a version called the API group version.
Kubernetes by default allows changing API group versions between clusters as long as the upgrade is a single version, for example, v1 -> v2beta1. Jumping multiple versions, for example, v1 -> v3, is not supported out of the box. This is where the Velero Enable API Group Version feature can help you during an upgrade.
Currently, the Enable API Group Version feature is in beta and can be enabled by installing Velero with a
For the most up-to-date information on Kubernetes API version compatibility, you should always review the Kubernetes release notes for the source and destination cluster version to before starting an upgrade, migration, or restore. If there is a difference between Kubernetes API versions, use the Enable API Group Version feature to help mitigate compatibility issues.
When the Enable API Group Versions feature is enabled on the source cluster, Velero will not only back up Kubernetes preferred API group versions, but it will also back up all supported versions on the cluster. As an example, consider the resource
horizontalpodautoscalers which falls under the
autoscaling group. Without the feature flag enabled, only the preferred API group version for autoscaling,
v1 will be backed up. With the feature enabled, the remaining supported versions,
v2beta2 will also be backed up. Once the versions are stored in the backup tarball file, they will be available to be restored on the destination cluster.
When the Enable API Group Versions feature is enabled on the destination cluster, Velero restore will choose the version to restore based on an API group version priority order.
The version priorities are listed from highest to lowest priority below:
The highest priority (Priority 1) will be the destination cluster’s preferred API group version. If the destination preferred version is found in the backup tarball, it will be the API group version chosen for restoration for that resource. However, if the destination preferred version is not found in the backup tarball, the next version in the list will be selected: the source cluster preferred version (Priority 2).
If the source cluster preferred version is found to be supported by the destination cluster, it will be chosen as the API group version to restore. However, if the source preferred version is not supported by the destination cluster, then the next version in the list will be considered: a non-preferred common supported version (Priority 3).
In the case that there are more than one non-preferred common supported version, which version will be chosen? The answer requires understanding the Kubernetes version priority order. Kubernetes prioritizes group versions by making the latest, most stable version the highest priority. The highest priority version is the Kubernetes preferred version. Here is a sorted version list example from the Kubernetes.io documentation:
Of the non-preferred common versions, the version that has the highest Kubernetes version priority will be chosen. See the example for Priority 3 below.
To better understand which API group version will be chosen, the following provides some concrete examples. The examples use the term “target cluster” which is synonymous to “destination cluster”.
--features=EnableAPIGroupVersions. For the enable API group versions feature to work, the feature flag needs to be used for Velero installations on both the source and destination clusters.
Optionally, users can create a config map to override the default API group prioritization for some or all of the resources being migrated. For each resource that is specified by the user, Velero will search for the version in both the backup tarball and the destination cluster. If there is a match, the user-specified API group version will be restored. If the backup tarball and the destination cluster does not have or support any of the user-specified versions, then the default version prioritization will be used.
Here are the steps for creating a config map that allows users to override the default version prioritization. These steps must happen on the destination cluster before a Velero restore is initiated.
Create a file called
restoreResourcesVersionPriority. The file name will become a key in the
data field of the config map.
<resource>.<group>=<highest user priority version>,<next highest>
rockbands.music.example.io=v2beta1,v2beta2 orchestras.music.example.io=v2,v3alpha1 subscriptions.operators.coreos.com=v2,v1
Apply config map with
kubectl create configmap enableapigroupversions --from-file=<absolute path>/restoreResourcesVersionPriority -n velero
See the config map with
kubectl describe configmap enableapigroupversions -n velero
The config map should look something like
Name: enableapigroupversions Namespace: velero Labels: <none> Annotations: <none> Data ==== restoreResourcesVersionPriority: ---- rockbands.music.example.io=v2beta1,v2beta2 orchestras.music.example.io=v2,v3alpha1 subscriptions.operators.coreos.com=v2,v1 Events: <none>
"error restoring rockbands.music.example.io/rockstars/beatles: the server could not find the requested resource".