version/2.3

Legacy Upgrade Support

This website/page will be End-of-life (EOL) after 31 August 2024. We recommend you to visit OpenEBS Documentation for the latest Mayastor documentation (v2.6 and above).

Mayastor is now also referred to as OpenEBS Replicated PV Mayastor.

A legacy installation of Mayastor (1.0.5 and below) cannot be seamlessly upgraded and needs manual intervention. Follow the below steps if you wish to upgrade from Mayastor 1.0.x to Mayastor 2.1.0 and above. Mayastor uses etcd as a persistent datastore for its configuration. As a first step, take a snapshot of the etcd. The detailed steps for taking a snapshot can be found in the etcd documentation.

As compared to Mayastor 1.0, the Mayastor 2.0 feature-set introduces breaking changes in some of the components, due to which the upgrade process from 1.0 to 2.0 is not seamless. The list of such changes are given below: ETCD:

  • Control Plane: The prefixes for control plane have changed from /namespace/$NAMESPACE/control-plane/ to /openebs.io/mayastor/apis/v0/clusters/$KUBE_SYSTEM_UID/namespaces/$NAMESPACE/

  • Data Plane: The Data Plane nexus information containing a list of healthy children has been moved from $nexus_uuid to /openebs.io/mayastor/apis/v0/clusters/$KUBE_SYSTEM_UID/namespaces/$NAMESPACE/volume/$volume_uuid/nexus/$nexus_uuid/info

RPC:

  • Control Plane: The RPC for the control plane has been changed from NATS to gRPC.

  • Data Plane: The registration heartbeat has been changed from NATS to gRPC.

Pool CRDs:

  • The pool CRDs have been renamed DiskPools (previously, MayastorPools).

  1. In order to start the upgrade process, the following previously deployed components have to be deleted.

  • To delete the control-plane components, execute:

kubectl delete deploy core-agents -n mayastor
kubectl delete deploy csi-controller -n mayastor
kubectl delete deploy msp-operator -n mayastor
kubectl delete deploy rest -n mayastor
kubectl delete ds mayastor-csi -n mayastor

  • Next, delete the associated RBAC operator. To do so, execute:

kubectl delete -f https://raw.githubusercontent.com/openebs/mayastor-control-plane/<version>/deploy/operator-rbac.yaml

In the above command, add the previously installed Mayastor version in the format v1.x.x

  1. Once all the above components have been successfully removed, fetch the latest helm chart from Mayastor-extension repo and save it to a file, say helm_templates.yaml. To do so, execute:

helm template mayastor . -n mayastor --set etcd.persistence.storageClass="manual" --set loki-stack.loki.persistence.storageClassName="manual" --set etcd.initialClusterState=existing > helm_templates.yaml

  • Next, update the helm_template.yaml file, add the following helm label to all the resources that are being created.

metadata: 
  annotations: 
    meta.helm.sh/release-name: $RELEASE_NAME 
    meta.helm.sh/release-namespace: $RELEASE_NAMESPACE 
  labels: 
    app.kubernetes.io/managed-by: Helm

  • Copy the etcd and io-engine spec from the helm_templates.yaml and save it in two different files say, mayastor_2.0_etcd.yaml and mayastor_io_v2.0.yaml. Once done, remove the etcd and io-engine specs from helm_templates.yaml. These components need to be upgraded separately.

  1. Install the new control-plane components using the helm-templates.yaml file.

kubectl apply -f helm_templates.yaml -n mayastor

In the above method of installation, the nexus (target) High Availability (HA) is disabled by default. The steps to enable HA are described later in this document.

  • Verify the status of the pods. Upon successful deployment, all the pods will be in a running state.

kubectl get pods -n mayastor

  • Verify the etcd prefix and compat mode.

kubectl exec -it mayastor-etcd-0 -n mayastor -- bash
Defaulted container "etcd" out of: etcd, volume-permissions (init)
I have no name!@mayastor-etcd-0:/opt/bitnami/etcd$ export ETCDCTL_API=3
I have no name!@mayastor-etcd-0:/opt/bitnami/etcd$ etcdctl get --prefix ""

  • Verify if the DiskPools are online.

kubectl get dsp -n mayastor

  • Next, verify the status of the volumes.

kubectl mayastor get volumes

  1. After upgrading control-plane components, the data-plane pods have to be upgraded. To do so, deploy the io-engine DaemonSet from Mayastor's new version.

Using the command given below, the data-plane pods (now io-engine pods) will be upgraded to Mayastor v2.0.

kubectl apply -f mayastor_io_v2.0.yaml -n mayastor

  • Delete the previously deployed data-plane pods (mayastor-xxxxx). The data-plane pods need to be manually deleted as their update-strategy is set to delete. Upon successful deletion, the new io-engine pods will be up and running.

  • NATS has been replaced by gRPC for Mayastor versions 2.0 or later. Hence, the NATS components (StatefulSets and services) have to be removed from the cluster.

kubectl delete sts nats -n mayastor
kubectl delete svc nats -n mayastor

  1. After control-plane and io-engine, the etcd has to be upgraded. Before starting the etcd upgrade, label the etcd PV and PVCs with helm. Use the below example to create a labels.yaml file. This will be needed to make them helm compatible.

metadata:
  annotations:
    meta.helm.sh/release-name: mayastor
    meta.helm.sh/release-namespace: mayastor
  labels:
    app.kubernetes.io/managed-by: Helm
kubectl patch pvc <data-mayastor-etcd-x> --patch-file labels.yaml n mayastor 
kubectl patch pv <etcd-volume-x> --patch-file labels.yaml -n mayastor

  • Next, deploy the etcd YAML. To deploy, execute:

kubectl apply -f mayastor_2.0_etcd.yaml -n mayastor

Now, verify the etcd space and compat mode, execute:

kubectl exec -it mayastor-etcd-0 -n mayastor -- bash
Defaulted container "etcd" out of: etcd, volume-permissions (init)
I have no name!@mayastor-etcd-0:/opt/bitnami/etcd$ export ETCDCTL_API=3
I have no name!@mayastor-etcd-0:/opt/bitnami/etcd$ etcdctl get --prefix ""

  1. Once all the components have been upgraded, the HA module can now be enabled via the helm upgrade command.

helm upgrade --install mayastor . -n mayastor --set etcd.persistence.storageClass="manual" --set loki-stack.loki.persistence.storageClassName="manual" --set agents.ha.enabled="true"

  1. This concludes the process of legacy upgrade. Run the below commands to verify the upgrade,

helm list -n mayastor
kubectl get pods -n mayastor
kubectl mayastor get volumes
PreviousUpgradeNextTips and Tricks

Last updated