Deploying and operating Mayastor in production contexts requires a foundational knowledge of Mayastor internals and best practices, found elsewhere within this documentation.

The objective of this section is to provide the user and evaluator of Mayastor with a topological view of the gross anatomy of a Mayastor deployment. A description will be made of the expected pod inventory of a correctly deployed cluster, the roles and functions of the constituent pods and related Kubernetes resource types, and of the high level interactions between them and the orchestration thereof.

Dramatis Personae

Component Roles

io-engine

The io-engine pod encapsulates Mayastor containers, which implement the I/O path from the block devices at the persistence layer, up to the relevant initiators on the worker nodes mounting volume claims. The Mayastor process running inside this container performs four major functions:

• Creates and manages DiskPools hosted on that node.

• Creates, exports, and manages volume controller objects hosted on that node.

• Creates and exposes replicas from DiskPools hosted on that node over NVMe-TCP.

• Provides a gRPC interface service to orchestrate the creation, deletion and management of the above objects, hosted on that node. Before the io-engine pod starts running, an init container attempts to verify connectivity to the agent-core in the namespace where Mayastor has been deployed. If a connection is established, the io-engine pod registers itself over gRPC to the agent-core. In this way, the agent-core maintains a registry of nodes and their supported api-versions. The scheduling of these pods is determined declaratively by using a DaemonSet specification. By default, a nodeSelector field is used within the pod spec to select all worker nodes to which the user has attached the label openebs.io/engine=mayastor as recipients of an io-engine pod. In this way, the node count and location are set appropriately to the hardware configuration of the worker nodes, and the capacity and performance demands of the cluster.

csi-node

The csi-node pods within a cluster implement the node plugin component of Mayastor's CSI driver. As such, their function is to orchestrate the mounting of Mayastor-provisioned volumes on worker nodes on which application pods consuming those volumes are scheduled. By default, a csi-node pod is scheduled on every node in the target cluster, as determined by a DaemonSet resource of the same name. Each of these pods encapsulates two containers, csi-node, and csi-driver-registrar. The node plugin does not need to run on every worker node within a cluster and this behavior can be modified, if desired, through the application of appropriate node labeling and the addition of a corresponding nodeSelector entry within the pod spec of the csi-node DaemonSet. However, it should be noted that if a node does not host a plugin pod, then it will not be possible to schedule an application pod on it, which is configured to mount Mayastor volumes.

etcd

etcd is a distributed reliable key-value store for the critical data of a distributed system. Mayastor uses etcd as a reliable persistent store for its configuration and state data.

Supportability:

The supportability tool is used to create support bundles (archive files) by interacting with multiple services present in the cluster where Mayastor is installed. These bundles contain information about Mayastor resources like volumes, pools and nodes, and can be used for debugging. The tool can collect the following information:

• Topological information of Mayastor's resource(s) by interacting with the REST service

• Historical logs by interacting with Loki. If Loki is unavailable, it interacts with the kube-apiserver to fetch logs.

• Mayastor-specific Kubernetes resources by interacting with the kube-apiserver

• Mayastor-specific information from etcd (internal) by interacting with the etcd server.

Loki:

Promtail:

Verify / Enable Huge Page Support

2MiB-sized Huge Pages must be supported and enabled on the mayastor storage nodes. A minimum number of 1024 such pages (i.e. 2GiB total) must be available exclusively to the Mayastor pod on each node, which should be verified thus:

grep HugePages /proc/meminfo

AnonHugePages:         0 kB
ShmemHugePages:        0 kB
HugePages_Total:    1024
HugePages_Free:      671
HugePages_Rsvd:        0
HugePages_Surp:        0

If fewer than 1024 pages are available then the page count should be reconfigured on the worker node as required, accounting for any other workloads which may be scheduled on the same node and which also require them. For example:

echo 1024 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

This change should also be made persistent across reboots by adding the required value to the file/etc/sysctl.conf like so:

echo vm.nr_hugepages = 1024 | sudo tee -a /etc/sysctl.conf

Label Mayastor Node Candidates

All worker nodes which will have Mayastor pods running on them must be labelled with the OpenEBS engine type "mayastor". This label will be used as a node selector by the Mayastor Daemonset, which is deployed as a part of the Mayastor data plane components installation. To add this label to a node, execute:

kubectl label node <node_name> openebs.io/engine=mayastor

General

All worker nodes must satisfy the following requirements:

• x86-64 CPU cores with SSE4.2 instruction support

• (Tested on) Linux kernel 5.15 (Recommended) Linux kernel 5.13 or higher. The kernel should have the following modules loaded:

  • nvme-tcp

  • ext4 and optionally xfs
• Each worker node which will host an instance of an io-engine pod must have the following resources free and available for exclusive use by that pod:

  • Two CPU cores

  • 1GiB RAM

  • HugePage support

    • A minimum of 2GiB of 2MiB-sized pages

Networking requirements

• Ensure that the following ports are not in use on the node:

  • 10124: Mayastor gRPC server will use this port.

  • 8420 / 4421: NVMf targets will use these ports.

• The firewall settings should not restrict connection to the node.

Recommended resource requirements

resources:
  limits:
    cpu: "2"
    memory: "1Gi"
    hugepages-2Mi: "2Gi"
  requests:
    cpu: "2"
    memory: "1Gi"
    hugepages-2Mi: "2Gi"

resources:
  limits:
    cpu: "100m"
    memory: "50Mi"
  requests:
    cpu: "100m"
    memory: "50Mi"

resources:
  limits:
    cpu: "32m"
    memory: "128Mi"
  requests:
    cpu: "16m"
    memory: "64Mi"

resources:
  limits:
    cpu: "100m"
    memory: "64Mi"
  requests:
    cpu: "50m"
    memory: "32Mi"

resources:
  limits:
    cpu: "1000m"
    memory: "32Mi"
  requests:
    cpu: "500m"
    memory: "16Mi"

resources:
  limits:
    cpu: "100m"
    memory: "32Mi"
  requests:
    cpu: "50m"
    memory: "16Mi"

DiskPool requirements

• Disks must be unpartitioned, unformatted, and used exclusively by the DiskPool.

• The minimum capacity of the disks should be 10 GB.

RBAC permission requirements

• Kubernetes core v1 API-group resources: Pod, Event, Node, Namespace, ServiceAccount, PersistentVolume, PersistentVolumeClaim, ConfigMap, Secret, Service, Endpoint, Event.

• Kubernetes batch API-group resources: CronJob, Job

• Kubernetes apps API-group resources: Deployment, ReplicaSet, StatefulSet, DaemonSet

• Kubernetes storage.k8s.io API-group resources: StorageClass, VolumeSnapshot, VolumeSnapshotContent, VolumeAttachment, CSI-Node

• Kubernetes apiextensions.k8s.io API-group resources: CustomResourceDefinition

• Mayastor Custom Resources that is openebs.io API-group resources: DiskPool

• Custom Resources from Helm chart dependencies of Jaeger that is helpful for debugging:

  • ConsoleLink Resource from console.openshift.io API group

  • ElasticSearch Resource from logging.openshift.io API group

  • Kafka and KafkaUsers from kafka.strimzi.io API group

  • ServiceMonitor from monitoring.coreos.com API group

  • Ingress from networking.k8s.io API group and from extensions API group

  • Route from route.openshift.io API group

  • All resources from jaegertracing.io API group

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: {{ .Release.Name }}-service-account
  namespace: {{ .Release.Namespace }}
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mayastor-cluster-role
rules:
- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["create", "list"]
  # must read diskpool info
- apiGroups: ["datacore.com"]
  resources: ["diskpools"]
  verbs: ["get", "list", "watch", "update", "replace", "patch"]
  # must update diskpool status
- apiGroups: ["datacore.com"]
  resources: ["diskpools/status"]
  verbs: ["update", "patch"]
  # external provisioner & attacher
- apiGroups: [""]
  resources: ["persistentvolumes"]
  verbs: ["get", "list", "watch", "update", "create", "delete", "patch"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list", "watch"]

  # external provisioner
- apiGroups: [""]
  resources: ["persistentvolumeclaims"]
  verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
  resources: ["storageclasses"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["events"]
  verbs: ["list", "watch", "create", "update", "patch"]
- apiGroups: ["snapshot.storage.k8s.io"]
  resources: ["volumesnapshots"]
  verbs: ["get", "list"]
- apiGroups: ["snapshot.storage.k8s.io"]
  resources: ["volumesnapshotcontents"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list", "watch"]

  # external attacher
- apiGroups: ["storage.k8s.io"]
  resources: ["volumeattachments"]
  verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["storage.k8s.io"]
  resources: ["volumeattachments/status"]
  verbs: ["patch"]
  # CSI nodes must be listed
- apiGroups: ["storage.k8s.io"]
  resources: ["csinodes"]
  verbs: ["get", "list", "watch"]

  # get kube-system namespace to retrieve Uid
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mayastor-cluster-role-binding
subjects:
- kind: ServiceAccount
  name: {{ .Release.Name }}-service-account
  namespace: {{ .Release.Namespace }}
roleRef:
  kind: ClusterRole
  name: mayastor-cluster-role
  apiGroup: rbac.authorization.k8s.io

Minimum worker node count

The minimum supported worker node count is three nodes. When using the synchronous replication feature (N-way mirroring), the number of worker nodes to which Mayastor is deployed should be no less than the desired replication factor.

Transport protocols

Mayastor supports the export and mounting of volumes over NVMe-oF TCP only. Worker node(s) on which a volume may be scheduled (to be mounted) must have the requisite initiator support installed and configured. In order to reliably mount Mayastor volumes over NVMe-oF TCP, a worker node's kernel version must be 5.13 or later and the nvme-tcp kernel module must be loaded.

Objective

Define the PVC

For the purposes of this quickstart guide, it is suggested to name the PVC "ms-volume-claim", as this is what will be illustrated in the example steps which follow.

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ms-volume-claim
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: mayastor-1
EOF

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ms-volume-claim
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: INSERT_YOUR_STORAGECLASS_NAME_HERE

If you used the storage class example from previous stage, then volume binding mode is set to WaitForFirstConsumer. That means, that the volume won't be created until there is an application using the volume. We will go ahead and create the application pod and then check all resources that should have been created as part of that in kubernetes.

Deploy the FIO Test Pod

The Mayastor CSI driver will cause the application pod and the corresponding Mayastor volume's NVMe target/controller ("Nexus") to be scheduled on the same Mayastor Node, in order to assist with restoration of volume and application availabilty in the event of a storage node failure.

kubectl apply -f https://raw.githubusercontent.com/openebs/Mayastor/v1.0.2/deploy/fio.yaml

kind: Pod
apiVersion: v1
metadata:
  name: fio
spec:
  nodeSelector:
    openebs.io/engine: mayastor
  volumes:
    - name: ms-volume
      persistentVolumeClaim:
        claimName: ms-volume-claim
  containers:
    - name: fio
      image: nixery.dev/shell/fio
      args:
        - sleep
        - "1000000"
      volumeMounts:
        - mountPath: "/volume"
          name: ms-volume

Verify the Volume and the Deployment

We will now verify the Volume Claim and that the corresponding Volume and Mayastor Volume resources have been created and are healthy.

Verify the Volume Claim

The status of the PVC should be "Bound".

kubectl get pvc ms-volume-claim

NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     AGE
ms-volume-claim     Bound    pvc-fe1a5a16-ef70-4775-9eac-2f9c67b3cd5b   1Gi        RWO            mayastor-1       15s

Verify the Persistent Volume

kubectl get pv pvc-fe1a5a16-ef70-4775-9eac-2f9c67b3cd5b

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                       STORAGECLASS     REASON   AGE
pvc-fe1a5a16-ef70-4775-9eac-2f9c67b3cd5b   1Gi        RWO            Delete           Bound    default/ms-volume-claim     mayastor-1       16m

Verify the Mayastor Volume

The status of the volume should be "online".

kubectl mayastor get volumes

ID                                    REPLICAS  TARGET-NODE                ACCESSIBILITY STATUS  SIZE

18e30e83-b106-4e0d-9fb6-2b04e761e18a  3         aks-agentpool-12194210-0   nvmf           Online  1073741824 

Verify the application

Verify that the pod has been deployed successfully, having the status "Running". It may take a few seconds after creating the pod before it reaches that status, proceeding via the "ContainerCreating" state.

kubectl get pod fio

NAME   READY   STATUS    RESTARTS   AGE
fio    1/1     Running   0          34s

Run the FIO Tester Application

We now execute the FIO Test utility against the Mayastor PV for 60 seconds, checking that I/O is handled as expected and without errors. In this quickstart example, we use a pattern of random reads and writes, with a block size of 4k and a queue depth of 16.

kubectl exec -it fio -- fio --name=benchtest --size=800m --filename=/volume/test --direct=1 --rw=randrw --ioengine=libaio --bs=4k --iodepth=16 --numjobs=8 --time_based --runtime=60

benchtest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.20
Starting 1 process
benchtest: Laying out IO file (1 file / 800MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=376KiB/s,w=340KiB/s][r=94,w=85 IOPS][eta 00m:00s]
benchtest: (groupid=0, jobs=1): err= 0: pid=19: Thu Aug 27 20:31:49 2020
  read: IOPS=679, BW=2720KiB/s (2785kB/s)(159MiB/60011msec)
    slat (usec): min=6, max=19379, avg=33.91, stdev=270.47
    clat (usec): min=2, max=270840, avg=9328.57, stdev=23276.01
     lat (msec): min=2, max=270, avg= 9.37, stdev=23.29
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    4], 20.00th=[    4],
     | 30.00th=[    4], 40.00th=[    4], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    7], 95.00th=[   45],
     | 99.00th=[  136], 99.50th=[  153], 99.90th=[  165], 99.95th=[  178],
     | 99.99th=[  213]
   bw (  KiB/s): min=  184, max= 9968, per=100.00%, avg=2735.00, stdev=3795.59, samples=119
   iops        : min=   46, max= 2492, avg=683.60, stdev=948.92, samples=119
  write: IOPS=678, BW=2713KiB/s (2778kB/s)(159MiB/60011msec); 0 zone resets
    slat (usec): min=6, max=22191, avg=45.90, stdev=271.52
    clat (usec): min=454, max=241225, avg=14143.39, stdev=34629.43
     lat (msec): min=2, max=241, avg=14.19, stdev=34.65
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    3], 80.00th=[    4], 90.00th=[   22], 95.00th=[  110],
     | 99.00th=[  155], 99.50th=[  157], 99.90th=[  169], 99.95th=[  197],
     | 99.99th=[  228]
   bw (  KiB/s): min=  303, max= 9904, per=100.00%, avg=2727.41, stdev=3808.95, samples=119
   iops        : min=   75, max= 2476, avg=681.69, stdev=952.25, samples=119
  lat (usec)   : 4=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.02%, 4=82.46%, 10=7.20%, 20=1.62%, 50=1.50%
  lat (msec)   : 100=2.58%, 250=4.60%, 500=0.01%
  cpu          : usr=1.19%, sys=3.28%, ctx=134029, majf=0, minf=17
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=40801,40696,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2720KiB/s (2785kB/s), 2720KiB/s-2720KiB/s (2785kB/s-2785kB/s), io=159MiB (167MB), run=60011-60011msec
  WRITE: bw=2713KiB/s (2778kB/s), 2713KiB/s-2713KiB/s (2778kB/s-2778kB/s), io=159MiB (167MB), run=60011-60011msec

Disk stats (read/write):
  sdd: ios=40795/40692, merge=0/9, ticks=375308/568708, in_queue=891648, util=99.53%

If no errors are reported in the output then Mayastor has been correctly configured and is operating as expected. You may create and consume additional Persistent Volumes with your own test applications.

What is Mayastor?

Design goals for Mayastor include:

• Highly available, durable persistence of data

• To be readily deployable and easily managed by autonomous SRE or development teams

• To be a low-overhead abstraction for NVMe-based storage devices

By comparison, most "shared everything" storage systems are widely thought to impart an overhead of at least 40% (and sometimes as much as 80% or more) as compared to the capabilities of the underlying devices or cloud volumes; additionally traditional shared storage scales in an unpredictable manner as I/O from many workloads interact and compete for resources.

While Mayastor utilizes NVMe-oF it does not require NVMe devices or cloud volumes to operate and can work well with other device types.

Where can I find Mayastor?

Mayastor's source code and documentation are distributed amongst a number of GitHub repositories under the OpenEBS organisation. The following list describes some of the main repositories but is not exhaustive.

Overview

The steps and commands which follow are intended only for use in conjunction with Mayastor version(s) 2.1.x and above.

Installation via helm

1. Add the OpenEBS Mayastor Helm repository.

helm repo add mayastor https://openebs.github.io/mayastor-extensions/ 

"mayastor" has been added to your repositories

Run the following command to discover all the stable versions of the added chart repository:

helm search repo mayastor --versions

 NAME             	CHART VERSION	APP VERSION  	DESCRIPTION                       
mayastor/mayastor	2.4.0        	2.4.0       	Mayastor Helm chart for Kubernetes

1. Run the following command to install Mayastor _version 2.4.

helm install mayastor mayastor/mayastor -n mayastor --create-namespace --version 2.4.0

NAME: mayastor
LAST DEPLOYED: Thu Sep 22 18:59:56 2022
NAMESPACE: mayastor
STATUS: deployed
REVISION: 1
NOTES:
OpenEBS Mayastor has been installed. Check its status by running:
$ kubectl get pods -n mayastor

For more information or to view the documentation, visit our website at https://openebs.io.

Verify the status of the pods by running the command:

kubectl get pods -n mayastor

NAME                                         READY   STATUS    RESTARTS   AGE
mayastor-agent-core-6c485944f5-c65q6         2/2     Running   0          2m13s
mayastor-agent-ha-node-42tnm                 1/1     Running   0          2m14s
mayastor-agent-ha-node-45srp                 1/1     Running   0          2m14s
mayastor-agent-ha-node-tzz9x                 1/1     Running   0          2m14s
mayastor-api-rest-5c79485686-7qg5p           1/1     Running   0          2m13s
mayastor-csi-controller-65d6bc946-ldnfb      3/3     Running   0          2m13s
mayastor-csi-node-f4fgd                      2/2     Running   0          2m13s
mayastor-csi-node-ls9m4                      2/2     Running   0          2m13s
mayastor-csi-node-xtcfc                      2/2     Running   0          2m13s
mayastor-etcd-0                              1/1     Running   0          2m13s
mayastor-etcd-1                              1/1     Running   0          2m13s
mayastor-etcd-2                              1/1     Running   0          2m13s
mayastor-io-engine-f2wm6                     2/2     Running   0          2m13s
mayastor-io-engine-kqxs9                     2/2     Running   0          2m13s
mayastor-io-engine-m44ms                     2/2     Running   0          2m13s
mayastor-loki-0                              1/1     Running   0          2m13s
mayastor-obs-callhome-5f47c6d78b-fzzd7       1/1     Running   0          2m13s
mayastor-operator-diskpool-b64b9b7bb-vrjl6   1/1     Running   0          2m13s
mayastor-promtail-cglxr                      1/1     Running   0          2m14s
mayastor-promtail-jc2mz                      1/1     Running   0          2m14s
mayastor-promtail-mr8nf                      1/1     Running   0          2m14s

Create DiskPool(s)

What is a DiskPool?

When a node allocates storage capacity for a replica of a persistent volume (PV) it does so from a DiskPool. Each node may create and manage zero, one, or more such pools. The ownership of a pool by a node is exclusive. A pool can manage only one block device, which constitutes the entire data persistence layer for that pool and thus defines its maximum storage capacity.

A pool is defined declaratively, through the creation of a corresponding DiskPool custom resource on the cluster. The DiskPool must be created in the same namespace where Mayastor has been deployed. User configurable parameters of this resource type include a unique name for the pool, the node name on which it will be hosted and a reference to a disk device which is accessible from that node. The pool definition requires the reference to its member block device to adhere to a discrete range of schemas, each associated with a specific access mechanism/transport/ or device type.

Permissible Schemes for spec.disks under DiskPool CR

DEVNAME       DEVTYPE  SIZE      AVAILABLE  MODEL                             DEVPATH                                                           MAJOR  MINOR  DEVLINKS 
/dev/nvme0n1  disk     894.3GiB  yes        Dell DC NVMe PE8010 RI U.2 960GB  /devices/pci0000:30/0000:30:02.0/0000:31:00.0/nvme/nvme0/nvme0n1  259    0      "/dev/disk/by-id/nvme-eui.ace42e00164f0290", "/dev/disk/by-path/pci-0000:31:00.0-nvme-1", "/dev/disk/by-dname/nvme0n1", "/dev/disk/by-id/nvme-Dell_DC_NVMe_PE8010_RI_U.2_960GB_SDA9N7266I110A814"

Once a node has created a pool it is assumed that it henceforth has exclusive use of the associated block device; it should not be partitioned, formatted, or shared with another application or process. Any pre-existing data on the device will be destroyed.

Configure Pools for Use with this Quickstart

To get started, it is necessary to create and host at least one pool on one of the nodes in the cluster. The number of pools available limits the extent to which the synchronous N-way mirroring (replication) of PVs can be configured; the number of pools configured should be equal to or greater than the desired maximum replication factor of the PVs to be created. Also, while placing data replicas ensure that appropriate redundancy is provided. Mayastor's control plane will avoid placing more than one replica of a volume on the same node. For example, the minimum viable configuration for a Mayastor deployment which is intended to implement 3-way mirrored PVs must have three nodes, each having one DiskPool, with each of those pools having one unique block device allocated to it.

Using one or more the following examples as templates, create the required type and number of pools.

cat <<EOF | kubectl create -f -
apiVersion: "openebs.io/v1beta1"
kind: DiskPool
metadata:
  name: pool-on-node-1
  namespace: mayastor
spec:
  node: workernode-1-hostname
  disks: ["/dev/disk/by-id/<id>"]
EOF

apiVersion: "openebs.io/v1beta1"
kind: DiskPool
metadata:
  name: INSERT_POOL_NAME_HERE
  namespace: mayastor
spec:
  node: INSERT_WORKERNODE_HOSTNAME_HERE
  disks: ["INSERT_DEVICE_URI_HERE"]

Verify Pool Creation and Status

The status of DiskPools may be determined by reference to their cluster CRs. Available, healthy pools should report their State as online. Verify that the expected number of pools have been created and that they are online.

kubectl get dsp -n mayastor

NAME             NODE          STATE    POOL_STATUS   CAPACITY      USED   AVAILABLE
pool-on-node-1   node-1-14944  Created   Online        10724835328   0      10724835328
pool-on-node-2   node-2-14944  Created   Online        10724835328   0      10724835328
pool-on-node-3   node-3-14944  Created   Online        10724835328   0      10724835328

Create Mayastor StorageClass(s)

We illustrate this quickstart guide with two examples of possible use cases; one which offers no data redundancy (i.e. a single data replica), and another having three data replicas.

cat <<EOF | kubectl create -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: mayastor-1
parameters:
  ioTimeout: "30"
  protocol: nvmf
  repl: "1"
  thin: "true"
provisioner: io.openebs.csi-mayastor
EOF

cat <<EOF | kubectl create -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: mayastor-3
parameters:
  ioTimeout: "30"
  protocol: nvmf
  repl: "3"
  thin: "true"
provisioner: io.openebs.csi-mayastor
EOF

To start the drain operation, execute:

kubectl-mayastor drain node <node_name> <label>

To get the list of nodes on which the drain operation has been performed, execute:

kubectl-mayastor get drain nodes

To halt the drain operation or to make the node schedulable again, execute:

kubectl-mayastor uncordon node <node_name> <label>

Storage class resource in Kubernetes is used to supply parameters to volumes when they are created. It is a convenient way of grouping volumes with common characteristics. All parameters take a string value. Brief explanation of each supported Mayastor parameter follows.

"fsType"

File system that will be used when mounting the volume. The supported file systems are ext4, xfs and btrfs and the default file system when not specified is ext4. We recommend to use xfs that is considered to be more advanced and performant. Please ensure the requested filesystem driver is installed on all worker nodes in the cluster before using it.

"protocol"

The parameter 'protocol' takes the value nvmf(NVMe over TCP protocol). It is used to mount the volume (target) on the application node.

"repl"

The string value should be a number and the number should be greater than zero. Mayastor control plane will try to keep always this many copies of the data if possible. If set to one then the volume does not tolerate any node failure. If set to two, then it tolerates one node failure. If set to three, then two node failures, etc.

"thin"

The agents.core.capacity.thin spec present in the Mayastor helm chart consists of the following configurable parameters that can be used to control the scheduling of thinly provisioned replicas:

1. poolCommitment parameter specifies the maximum allowed pool commitment limit (in percent).

2. volumeCommitment parameter specifies the minimum amount of free space that must be present in each replica pool in order to create new replicas for an existing volume. This value is specified as a percentage of the volume size.

3. volumeCommitmentInitial minimum amount of free space that must be present in each replica pool in order to create new replicas for a new volume. This value is specified as a percentage of the volume size.

"stsAffinityGroup"

stsAffinityGroup represents a collection of volumes that belong to instances of Kubernetes StatefulSet. When a StatefulSet is deployed, each instance within the StatefulSet creates its own individual volume, which collectively forms the stsAffinityGroup. Each volume within the stsAffinityGroup corresponds to a pod of the StatefulSet.

This feature enforces the following rules to ensure the proper placement and distribution of replicas and targets so that there isn't any single point of failure affecting multiple instances of StatefulSet.

1. Anti-Affinity among single-replica volumes : This rule ensures that replicas of different volumes are distributed in such a way that there is no single point of failure. By avoiding the colocation of replicas from different volumes on the same node.

2. Anti-Affinity among multi-replica volumes :

If the affinity group volumes have multiple replicas, they already have some level of redundancy. This feature ensures that in such cases, the replicas are distributed optimally for the stsAffinityGroup volumes.

1. Anti-affinity among targets :

By default, the stsAffinityGroup feature is disabled. To enable it, modify the storage class YAML by setting the parameters.stsAffinityGroup parameter to true.

"cloneFsIdAsVolumeId"

cloneFsIdAsVolumeId is a setting for volume clones/restores with two options: true and false. By default, it is set to false.

• When set to true, the created clone/restore's filesystem uuid will be set to the restore volume's uuid. This is important because some file systems, like XFS, do not allow duplicate filesystem uuid on the same machine by default.

• When set to false, the created clone/restore's filesystem uuid will be same as the orignal volume uuid, but it will be mounted using the nouuid flag to bypass duplicate uuid validation.

The Mayastor kubectl plugin can be used to view and manage Mayastor resources such as nodes, pools and volumes. It is also used for operations such as scaling the replica count of volumes.

Install kubectl plugin

• Add the downloaded Mayastor kubectl plugin under $PATH.

To verify the installation, execute:

kubectl mayastor -V

kubectl-plugin 1.0.0

Use kubectl plugin to retrieve data

Sample command to use kubectl plugin:

USAGE:
    kubectl-mayastor [OPTIONS] <SUBCOMMAND>

OPTIONS:
    -h, --help
            Print help information
    -j, --jaeger <JAEGER>
            Trace rest requests to the Jaeger endpoint agent
    -k, --kube-config-path <KUBE_CONFIG_PATH>
            Path to kubeconfig file
    -n, --namespace <NAMESPACE>
            Kubernetes namespace of mayastor service, defaults to mayastor [default: mayastor]
    -o, --output <OUTPUT>
            The Output, viz yaml, json [default: none]
    -r, --rest <REST>
            The rest endpoint to connect to
    -t, --timeout <TIMEOUT>
            Timeout for the REST operations [default: 10s]
    -V, --version
            Print version information
SUBCOMMANDS:
    cordon      'Cordon' resources
    drain       'Drain' resources
    dump        `Dump` resources
    get         'Get' resources
    help        Print this message or the help of the given subcommand(s)
    scale       'Scale' resources
    uncordon    'Uncordon' resources

You can use the plugin with the following options:

Get Mayastor Volumes

kubectl mayastor get volumes

ID                                    REPLICAS  TARGET-NODE  ACCESSIBILITY STATUS  SIZE

18e30e83-b106-4e0d-9fb6-2b04e761e18a  4         mayastor-1   nvmf          Online  10485761
0c08667c-8b59-4d11-9192-b54e27e0ce0f  4         mayastor-2   <none>        Online  10485761

Get Mayastor Pools

kubectl mayastor get pools

ID               TOTAL CAPACITY  USED CAPACITY  DISKS                                                     NODE      STATUS  MANAGED

mayastor-pool-1  5360320512      1111490560     aio:///dev/vdb?uuid=d8a36b4b-0435-4fee-bf76-f2aef980b833  kworker1  Online  true
mayastor-pool-2  5360320512      2172649472     aio:///dev/vdc?uuid=bb12ec7d-8fc3-4644-82cd-dee5b63fc8c5  kworker1  Online  true
mayastor-pool-3  5360320512      3258974208     aio:///dev/vdb?uuid=f324edb7-1aca-41ec-954a-9614527f77e1  kworker2  Online  false
    

Get Mayastor Nodes

kubectl mayastor get nodes

ID          GRPC ENDPOINT   STATUS
mayastor-2  10.1.0.7:10124  Online
mayastor-1  10.1.0.6:10124  Online
mayastor-3  10.1.0.8:10124  Online

Scale the replica count of a volume

kubectl mayastor scale volume <volume_id> <size>

Volume 0c08667c-8b59-4d11-9192-b54e27e0ce0f Scaled Successfully 🚀

Retrieve resource in any of the output formats (table, JSON or YAML)

Table is the default output format.

kubectl mayastor -ojson get <resource_type>

[{"spec":{"num_replicas":2,"size":67108864,"status":"Created","target":{"node":"ksnode-2","protocol":"nvmf"},"uuid":"5703e66a-e5e5-4c84-9dbe-e5a9a5c805db","topology":{"explicit":{"allowed_nodes":["ksnode-1","ksnode-3","ksnode-2"],"preferred_nodes":["ksnode-2","ksnode-3","ksnode-1"]}},"policy":{"self_heal":true}},"state":{"target":{"children":[{"state":"Online","uri":"bdev:///ac02cf9e-8f25-45f0-ab51-d2e80bd462f1?uuid=ac02cf9e-8f25-45f0-ab51-d2e80bd462f1"},{"state":"Online","uri":"nvmf://192.168.122.6:8420/nqn.2019-05.io.openebs:7b0519cb-8864-4017-85b6-edd45f6172d8?uuid=7b0519cb-8864-4017-85b6-edd45f6172d8"}],"deviceUri":"nvmf://192.168.122.234:8420/nqn.2019-05.io.openebs:nexus-140a1eb1-62b5-43c1-acef-9cc9ebb29425","node":"ksnode-2","rebuilds":0,"protocol":"nvmf","size":67108864,"state":"Online","uuid":"140a1eb1-62b5-43c1-acef-9cc9ebb29425"},"size":67108864,"status":"Online","uuid":"5703e66a-e5e5-4c84-9dbe-e5a9a5c805db"}}]

Retrieve replica topology for specific volumes

kubectl mayastor get volume-replica-topology <volume_id>

 ID                                    NODE         POOL    STATUS  CAPACITY  ALLOCATED  SNAPSHOTS  CHILD-STATUS  REASON  REBUILD 
 a34dbaf4-e81a-4091-b3f8-f425e5f3689b  io-engine-1  pool-1  Online  12MiB     0 B        12MiB      <none>        <none>  <none> 

List available volume snapshots

kubectl mayastor get volume-snapshots

 ID                                    TIMESTAMP             SOURCE-SIZE  ALLOCATED-SIZE  TOTAL-ALLOCATED-SIZE  SOURCE-VOL 
 25823425-41fa-434a-9efd-a356b70b5d7c  2023-07-07T13:20:17Z  10MiB        12MiB           12MiB                 ec4e66fd-3b33-4439-b504-d49aba53da26 

Limitations of kubectl plugin

• The plugin currently does not have authentication support.

• The plugin can operate only over HTTP.

Mayastor 2.0 enhances High Availability (HA) of the volume target with the nexus switch-over feature. In the event of the target failure, the switch-over feature quickly detects the failure and spawns a new nexus to ensure I/O continuity. The HA feature consists of two components: the HA node agent (which runs in each csi- node) and the cluster agent (which runs alongside the agent-core). The HA node agent looks for io-path failures from applications to their corresponding targets. If any such broken path is encountered, the HA node agent informs the cluster agent. The cluster-agent then creates a new target on a different (live) node. Once the target is created, the node-agent establishes a new path between the application and its corresponding target. The HA feature restores the broken path within seconds, ensuring negligible downtime.

How do I disable this feature?

The HA feature is enabled by default; to disable it, pass the parameter --set=agents.ha.enabled=false with the helm install command.

With the previous versions, the control plane ensured replica redundancy by monitoring all volume targets and checking for any volume targets that were in Degraded state, indicating that one or more replicas of that volume targets were faulty. When a matching volume targets is found, the faulty replica is removed. Then, a new replica is created and added to the volume targets object. As part of adding the new child data-plane, a full rebuild was initiated from one of the existing Online replicas. However, the drawback to the above approach was that even if a replica was inaccessible for a short period (e.g., due to a node restart), a full rebuild was triggered. This may not have a significant impact on replicas with small sizes, but it is not desirable for large replicas.

The partial rebuild feature, overcomes the above problem and helps in achieving faster rebuild times. When volume target encounters IO error on a child/replica, it marks the child as Faulted (removing it from the I/O path) and begins to maintain a write log for all subsequent writes. The Core agent starts a default 10 minute wait for the replica to come back. If the child's replica is online again within timeout, the control-plane requests the volume target to online the child and add it to the IO path along with a partial rebuild process using the aforementioned write log.

Replica Rebuild History

The data-plane handles both full and partial replica rebuilds. To view history of the rebuilds that an existing volume target has undergone during its lifecycle until now, you can use the given kubectl command.

To get the output in table format:

To get the output in JSON format:

For example: kubectl mayastor get rebuild-history e898106d-e735-4edf-aba2-932d42c3c58d -ojson

Cordoning a node marks or taints the node as unschedulable. This prevents the scheduler from deploying new resources on that node. However, the resources that were deployed prior to cordoning off the node will remain intact.

This feature is in line with the node-cordon functionality of Kubernetes.

To add a label and cordon a node, execute:

To get the list of cordoned nodes, execute:

To view the labels associated with a cordoned node, execute:

How to uncordon a node?

In order to make a node schedulable again, execute:

Pool metrics exporter

The Mayastor pool metrics exporter runs as a sidecar container within every io-engine pod and exposes pool usage metrics in Prometheus format. These metrics are exposed on port 9502 using an HTTP endpoint /metrics and are refreshed every five minutes.

Supported pool metrics

Stats exporter metrics

Supported stats metrics

Integrating exporter with Prometheus monitoring stack

1. To install, add the Prometheus-stack helm chart and update the repo.

Then, install the Prometheus monitoring stack and set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues to false. This enables Prometheus to discover custom ServiceMonitor for Mayastor.

1. Next, install the ServiceMonitor resource to select services and specify their underlying endpoint objects.

CSI metrics exporter

Volume snapshots are copies of a persistent volume at a specific point in time. They can be used to restore a volume to a previous state or create a new volume. Mayastor provides support for industry standard copy-on-write (COW) snapshots, which is a popular methodology for taking snapshots by keeping track of only those blocks that have changed. Mayastor incremental snapshot capability enhances data migration and portability in Kubernetes clusters across different cloud providers or data centers. Using standard kubectl commands, you can seamlessly perform operations on snapshots and clones in a fully Kubernetes-native manner.

Use cases for volume snapshots include:

• Efficient replication for backups.

• Utilization of clones for troubleshooting.

• Development against a read-only copy of data.

Volume snapshots allow the creation of read-only incremental copies of volumes, enabling you to maintain a history of your data. These volume snapshots possess the following characteristics:

• Consistency: The data stored within a snapshot remains consistent across all replicas of the volume, whether local or remote.

• Immutability: Once a snapshot is successfully created, the data contained within it cannot be modified.

Currently, Mayastor supports the following operations related to volume snapshots:

1. Creating a snapshot for a PVC

2. Listing available snapshots for a PVC

3. Deleting a snapshot for a PVC

Prerequisites

2. Create a Mayastor StorageClass with single replica.

Copy the PVC name, for example, ms-volume-claim.

Create a Snapshot

You can create a snapshot (with or without an application) using the PVC. Follow the steps below to create a volume snapshot:

Step 1: Create a Kubernetes VolumeSnapshotClass object

Apply VolumeSnapshotClass details

Step 2: Create the snapshot

Apply the snapshot

List Snapshots

To retrieve the details of the created snapshots, use the following command:

Delete a Snapshot

To delete a snapshot, use the following command:

Supportability

Using the supportability tool

To bundle Mayastor's complete system information, execute:

kubectl mayastor dump system -n mayastor -d <output_directory_path>

To view all the available options and sub-commands that can be used with the dump command, execute:

kubectl mayastor dump --help

`Dump` resources

Usage: kubectl-mayastor dump [OPTIONS] <COMMAND>

Commands:
  system  Collects entire system information
  etcd    Collects information from etcd
  help    Print this message or the help for the given subcommand(s)

Options:
  -r, --rest <REST>
          The rest endpoint to connect to
  -t, --timeout <TIMEOUT>
          Specifies the timeout value to interact with other modules of system [default: 10s]
  -k, --kube-config-path <KUBE_CONFIG_PATH>
          Path to kubeconfig file
  -s, --since <SINCE>
          Period states to collect all logs from last specified duration [default: 24h]
  -l, --loki-endpoint <LOKI_ENDPOINT>
          LOKI endpoint, if left empty then it will try to parse endpoint from Loki service(K8s service resource), if the tool is unable to parse from service then logs will be collected using Kube-apiserver
  -e, --etcd-endpoint <ETCD_ENDPOINT>
          Endpoint of ETCD service, if left empty then will be parsed from the internal service name
  -d, --output-directory-path <OUTPUT_DIRECTORY_PATH>
          Output directory path to store archive file [default: ./]
  -n, --namespace <NAMESPACE>
          Kubernetes namespace of mayastor service [default: mayastor]
  -o, --output <OUTPUT>
          The Output, viz yaml, json [default: none]
  -j, --jaeger <JAEGER>
          Trace rest requests to the Jaeger endpoint agent
  -h, --help
          Print help
          
Supportability - collects state & log information of services and dumps it to a tar file. 

The archive files generated by the dump command are stored in the specified output directories. The tables below specify the path and the content that will be stored in each archive file.

Topology Information

Historical Logs

Configuration details of Kubernetes resources

etcd dump information

Supportability tool logs

Does the supportability tool expose sensitive data?

The supportability tool generates support bundles, which are used for debugging purposes. These bundles are created in response to the user's invocation of the tool and can be transmitted only by the user. Below is the information collected by the supportability tool that might be identified as 'sensitive' based on the organization's data protection/privacy commitments and security policies. Logs: The default installation of Mayastor includes the deployment of a log aggregation subsystem based on Grafana Loki. All the pods deployed in the same namespace as Mayastor and labelled with openebs.io/logging=true will have their logs incorporated within this centralized collector. These logs may include the following information:

• Kubernetes (K8s) node hostnames

• IP addresses

  • container addresses

• API endpoints

  • Mayastor

  • K8s

• Container names

• K8s Persistent Volume names (provisioned by Mayastor)

• DiskPool names

• Block device details (except the content) K8s Definition Files: The support bundle includes definition files for all the Mayastor components. Some of these are listed below:

• Deployments

• DaemonSets

• StatefulSets

• VolumeSnapshotClass

• VolumeSnapshotContent

K8s Events: The archive files generated by the supportability tool contain information on all the events of the Kubernetes cluster present in the same namespace as Mayastor.

etcd Dump: The default installation of Mayastor deploys an etcd instance for its exclusive use. This key-value pair is used to persist state information for Mayastor-managed objects. These key-value pairs are required for diagnostic and troubleshooting purposes. The etcd dump archive file consists of the following information:

• Kubernetes node hostnames

• IP addresses

• PVC/PV names

• Container names

  • Mayastor

  • User applications within the mayastor namespace

• Block device details (except data content)

Supported upgrade paths

• From 2.0.x to 2.4.0

To upgrade Mayastor deployment on the Kubernetes cluster, execute:

kubectl mayastor upgrade

To view all the available options and sub-commands that can be used with the upgrade command, execute:

kubectl mayastor upgrade -h

`Upgrade` the deployment

Usage: kubectl-mayastor upgrade [OPTIONS]

Options:
  -d, --dry-run
          Display all the validations output but will not execute upgrade
  -r, --rest <REST>
          The rest endpoint to connect to
  -D, --skip-data-plane-restart
          If set then upgrade will skip the io-engine pods restart
  -k, --kube-config-path <KUBE_CONFIG_PATH>
          Path to kubeconfig file
  -S, --skip-single-replica-volume-validation
          If set then it will continue with upgrade without validating singla replica volume
  -R, --skip-replica-rebuild
          If set then upgrade will skip the repilca rebuild in progress validation
  -C, --skip-cordoned-node-validation
          If set then upgrade will skip the cordoned node validation
  -o, --output <OUTPUT>
          The Output, viz yaml, json [default: none]
  -j, --jaeger <JAEGER>
          Trace rest requests to the Jaeger endpoint agent
  -n, --namespace <NAMESPACE>
          Kubernetes namespace of mayastor service, defaults to mayastor [default: mayastor]
  -h, --help
          Print help

To view the status of upgrade, execute:

kubectl mayastor get upgrade-status

Upgrade From: 2.0.0
Upgrade To: 2.4.0
Upgrade Status: Successfully upgraded Mayastor

To view the logs of upgrade job, execute:

kubectl logs <upgrade-job-pod-name> -n <mayastor-namespace>

CPU isolation

Mayastor will fully utilize each CPU core that it was configured to run on. It will spawn a thread on each and the thread will run in an endless loop serving tasks dispatched to it without sleeping or blocking. There are also other Mayastor threads that are not bound to the CPU and those are allowed to block and sleep. However, the bound threads (also called reactors) rely on being interrupted by the kernel and other userspace processes as little as possible. Otherwise, the latency of IO may suffer.

Ideally, the only thing that interrupts Mayastor's reactor would be only kernel time-based interrupts responsible for CPU accounting. However, that is far from trivial. isolcpus option that we will be using does not prevent:

• kernel threads and

• other k8s pods to run on the isolated CPU

However, it prevents system services including kubelet from interfering with Mayastor.

Set Linux kernel boot parameter

Note that the best way to accomplish this step may differ, based on the Linux distro that you are using.

Add the isolcpus kernel boot parameter to GRUB_CMDLINE_LINUX_DEFAULT in the grub configuration file, with a value which identifies the CPUs to be isolated (indexing starts from zero here). The location of the configuration file to change is typically /etc/default/grub but may vary. For example when running Ubuntu 20.04 in AWS EC2 Cloud boot parameters are in /etc/default/grub.d/50-cloudimg-settings.cfg.

In the following example we assume a system with 4 CPU cores in total, and that the third and the fourth CPU cores are to be dedicated to Mayastor.

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=2,3"

Update grub

sudo update-grub

Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/40-force-partuuid.cfg'
Sourcing file `/etc/default/grub.d/50-cloudimg-settings.cfg'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.8.0-29-generic
Found initrd image: /boot/microcode.cpio /boot/initrd.img-5.8.0-29-generic
Found linux image: /boot/vmlinuz-5.4.0-1037-aws
Found initrd image: /boot/microcode.cpio /boot/initrd.img-5.4.0-1037-aws
Found Ubuntu 20.04.2 LTS (20.04) on /dev/xvda1
done

Reboot the system

sudo reboot

Verify isolcpus

Basic verification is by outputting the boot parameters of the currently running kernel:

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-5.8.0-29-generic root=PARTUUID=7213a253-01 ro console=tty1 console=ttyS0 nvme_core.io_timeout=4294967295 isolcpus=2,3 panic=-1

You can also print a list of isolated CPUs:

cat /sys/devices/system/cpu/isolated

2-3

Update mayastor helm chart for CPU core specification

To allot specific CPU cores for Mayastor's reactors, follow these steps:

2. Execute the following command to update Mayastor's configuration. Replace <namespace> with the appropriate Kubernetes namespace where Mayastor is deployed.

kubectl mayastor upgrade -n <namespace> --set-args 'io_engine.coreList={2,3}'

In the above command, io_engine.coreList={2,3} specifies that Mayastor's reactors should operate on the third and fourth CPU cores.

Glossary of Terms

Mayastor Instance

An instance of the mayastor binary running inside a Mayastor container, which is encapsulated by a Mayastor Pod.

Nexus

Pool / Storage Pool / Mayastor Storage Pool (MSP)

Mayastor's volume management abstraction. Block devices contributing storage capacity to a Mayastor deployment do so by their inclusion within configured storage pools. Each Mayastor node can host zero or more pools and each pool can "contain" a single base block device as a member. The total capacity of the pool is therefore determined by the size of that device. Pools can only be hosted on nodes running an instance of a mayastor pod.

Multiple volumes can share the capacity of one pool but thin provisioning is not supported. Volumes cannot span multiple pools for the purposes of creating a volume larger in size than could be accommodated by the free capacity in any one pool.

Bdev

• base bdev - Handles I/O directly, e.g. a representation of a physical SSD device

Replica

Mayastor terminology. An lvol bdev (a "logical volume", created within a pool and consuming pool capacity) which is being exported by a Mayastor instance, for consumption by a nexus (local or remote to the exporting instance) as a "child"

Child

Mayastor terminology. A NVMe controller created and owned by a given Nexus and which handles I/O downstream from the nexus' target, by routing it to a replica associated with that child.

A nexus has a minimum of one child, which must be local (local: exported as a replica from a pool hosted by the same mayastor instance as hosts the nexus itself). If the Mayastor volume being exported by the nexus is derived from a StorageClass with a replication factor greater than 1 (i.e. synchronous N-way mirroring is enabled), then the nexus will have additional children, up to the desired number of data copies.

Export

To allow the discovery of, and acceptance of I/O for, a volume by a client initiator, over a Mayastor storage target.

Basics of I/O Flow

Non-Replicated Volume I/O Path

 ____________________________________________________________
|                         Front-end                          |
|                          NVMe-oF                           |
|                        (user space)                        |
|____________________________________________________________|
                              |                              
 _____________________________v______________________________
|  [Nexus]                    |  I/O path                    |
|                             |                              |  
|                     ________V________                      |
|                    |        |         |                    |
|                    |    NexusChild    |                    |
|                    |                  |                    |
|                    |________|_________|                    |
|_____________________________|______________________________| 
                              |                     
                          <loopback>
                              |  
                        ______V________ 
                       |    Replica    |
                       |    (local)    |   
                       |==== pool =====|
                       |               |
                       |    +----+     |
                       |    |lvol|     |  
                       |    +----+     |   
                       |_______________| 
                              |
                        ______V________ 
                       |  base bdev    |
                       |_______________|
                              |
                              V
                          DISK DEVICE
                         e.g. /dev/sda                              

For volumes based on a StorageClass defined as having a replication factor of 1, a single data copy is maintained by Mayastor. The I/O path is largely (entirely, if using malloc:/// pool devices) constrained to within the bounds of a single mayastor instance, which hosts both the volume's nexus and the storage pool in use as its persistence layer.

Each mayastor instance presents a user-space storage target over NVMe-oF TCP. Worker nodes mounting a Mayastor volume for a scheduled application pod to consume are directed by Mayastor's CSI driver implementation to connect to the appropriate transport target for that volume and perform discovery, after which they are able to send I/O to it, directed at the volume in question. Regardless of how many volumes, and by extension how many nexus a mayastor instance hosts, all share the same target instances.

Application I/O received on a target for a volume is passed to the virtual bdev at the front-end of the nexus hosting that volume. In the case of a non-replicated volume, the nexus is composed of a single child, to which the I/O is necessarily routed. As a virtual bdev itself, the child handles the I/O by routing it to the next device, in this case the replica that was created for this child. In non-replicated scenarios, both the volume's nexus and the pool which hosts its replica are co-located within the same mayastor instance, hence the I/O is passed from child to replica using SPDK bdev routines, rather than a network level transport. At the pool layer, a blobstore maps the lvol bdev exported as the replica concerned to the base bdev on which the pool was constructed. From there, other than for malloc:/// devices, the I/O passes to the host kernel via either aio or io_uring, thence via the appropriate storage driver to the physical disk device.

The disk devices' response to the I/O request is returns back along the same path to the caller's initiator.

Replicated Volume I/O Path

 _______________________________________________________________    _
|                           Front-end                           |    |
|                              NVMe-oF                          |    |
|                          (user space)                         |    |
|_______________________________________________________________|    |
                                |                                    |
 _______________________________|_______________________________     |
|  [Nexus]                      |  I/O path                     |    |
|           ____________________|____________________           |    |
|          |                    |                    |          |    |
|  ________V________    ________V________    ________V________  |    |
| |child 1          |  |child 2          |  |child 3          | |    |
| |                 |  |                 |  |                 | |    |
| |      NVMe-oF    |  |     NVMe-oF     |  |      NVMe-oF    | |    |
| |                 |  |                 |  |                 | |    |  Mayastor
| |________|________|  |________|________|  |________|________| |    |  Instance
|__________|____________________|____________________|__________|    |    "A"
           |                    |                    |               | 
       <network>            <loopback>           <network>           |
           |                    |                    |               | 
           |              ______V________            |               |
           |             |    Replica    |           |               |
           |             |    (local)    |           |               |
           |             |==== pool =====|           |               |  
           |             |               |           |               |
           |             |    +----+     |           |               |
           |             |    |lvol|     |           |               |  
           |             |    +----+     |           |               |  
           |             |_______________|           |               |
           |                    |                    |               |
           |              ______V________            |               |
           |             |  base bdev    |           |               |
           |             |_______________|           |              _|
           |                    |                    |
           |                    V                    |
           |                DISK DEVICE              |               ]  Node "A"
           |                                         |
           |                                         |
           |                                         |
           |                                         |  
     ______V________    _                      ______V________      _
    |    Replica    |    |                    |    Replica    |      |
    |    (remote)   |    |                    |    (remote)   |      |
    |  nvmf target  |    |                    |  nvmf target  |      |
    |               |    |                    |               |      |
    |==== pool =====|    |                    |==== pool =====|      |
    |               |    | Mayastor           |               |      | Mayastor
    |    +----+     |    | Instance           |    +----+     |      | Instance
    |    |lvol|     |    |   "B"              |    |lvol|     |      |   "C"
    |    +----+     |    |                    |    +----+     |      | 
    |_______________|    |                    |_______________|      |
           |             |                           |               |
     ______V________     |                     ______V________       |
    |  base bdev    |    |                    |  base bdev    |      |
    |_______________|   _|                    |_______________|     _|
           |                                         |
           V                                         V
       DISK DEVICE       ] Node "B"              DISK DEVICE         ] Node "C"

If the StorageClass on which a volume is based specifies a replication factor of greater than one, then a synchronous mirroring scheme is employed to maintain multiple redundant data copies. For a replicated volume, creation and configuration of the volume's nexus requires additional orchestration steps. Prior to creating the nexus, not only must a local replica be created and exported as for the non-replicated case, but the requisite count of additional remote replicas required to meet the replication factor must be created and exported from Mayastor instances other than that hosting the nexus itself. The control plane core-agent component will select appropriate pool candidates, which includes ensuring sufficient available capacity and that no two replicas are sited on the same Mayastor instance (which would compromise availability during co-incident failures). Once suitable replicas have been successfully exported, the control plane completes the creation and configuration of the volume's nexus, with the replicas as its children. In contrast to their local counterparts, remote replicas are exported, and so connected to by the nexus, over NVMe-F using a user-mode initiator and target implementation from the SPDK.

Write I/O requests to the nexus are handled synchronously; the I/O is dispatched to all (healthy) children and only when completion is acknowledged by all is the I/O acknowledged to the calling initiator via the nexus front-end. Read I/O requests are similarly issued to all children, with just the first response returned to the caller.

Volume restore from an existing snapshot will create an exact replica of a storage volume captured at a specific point in time. They serve as an essential tool for data protection, recovery, and efficient management in Kubernetes environments. This article provides a step-by-step guide on how to create a volume restore.

Prerequisites

Step 1: Create a storage class

cat <<EOF | kubectl create -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: mayastor-1-restore
parameters:
  ioTimeout: "30"
  protocol: nvmf
  repl: "1"
  thin: "true"
provisioner: io.openebs.csi-mayastor
EOF

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: mayastor-1-restore
parameters:
  ioTimeout: "30"
  protocol: nvmf
  repl: "1"
  thin: "true"
provisioner: io.openebs.csi-mayastor

Note the name of the StorageClass, which, in this example, is mayastor-1-restore.

Step 2: Create a snapshot

Note the snapshot's name, for example, pvc-snap-1.

Create a volume restore of the existing snapshot

After creating a snapshot, you can create a PersistentVolumeClaim (PVC) from it to generate the volume restore. Use the following command:

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: restore-pvc //add a name for your new volume
spec:
  storageClassName: mayastor-1-restore //add your storage class name 
  dataSource:
    name: pvc-snap-1 //add your volumeSnapshot name
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
 EOF     

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: restore-pvc //add a name for your new volume
spec:
  storageClassName: mayastor-1-restore //add your storage class name 
  dataSource:
    name: pvc-snap-1 //add your volumeSnapshot name
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

By running this command, you create a new PVC named restore-pvc based on the specified snapshot. The restored volume will have the same data and configuration as the original volume had at the time of the snapshot.

What if I have no Disk Devices available that I can use for testing?

For basic test and evaluation purposes it may not always be practical or possible to allocate physical disk devices on a cluster to Mayastor for use within its pools. As a convenience, Mayastor supports two disk device type emulations for this purpose:

• Memory-Backed Disks ("RAM drive")

• File-Backed Disks

Memory-backed Disks are the most readily provisioned if node resources permit, since Mayastor will automatically create and configure them as it creates the corresponding pool. However they are the least durable option - since the data is held entirely within memory allocated to a Mayastor pod, should that pod be terminated and rescheduled by Kubernetes, that data will be lost. Therefore it is strongly recommended that this type of disk emulation be used only for short duration, simple testing. It must not be considered for production use.

File-backed disks, as their name suggests, store pool data within a file held on a file system which is accessible to the Mayastor pod hosting that pool. Their durability depends on how they are configured; specifically on which type of volume mount they are located. If located on a path which uses Kubernetes ephemeral storage (eg. EmptyDir), they may be no more persistent than a RAM drive would be. However, if placed on their own Persistent Volume (eg. a Kubernetes Host Path volume) then they may considered 'stable'. They are slightly less convenient to use than memory-backed disks, in that the backing files must be created by the user as a separate step preceding pool creation. However, file-backed disks can be significantly larger than RAM disks as they consume considerably less memory resource within the hosting Mayastor pod.

Using Memory-backed Disks

Creating a memory-backed disk emulation entails using the "malloc" uri scheme within the Mayastor pool resource definition.

apiVersion: "openebs.io/v1alpha1"
kind: DiskPool
metadata:
  name: mempool-1
  namespace: mayastor
spec:
  node: worker-node-1
  disks: ["malloc:///malloc0?size_mb=64"]

The example shown defines a pool named "mempool-1". The Mayastor pod hosted on "worker-node-1" automatically creates a 64MiB emulated disk for it to use, with the device identifier "malloc0" - provided that at least 64MiB of 2MiB-sized Huge Pages are available to that pod after the Mayastor container's own requirements have been satisfied.

The malloc:/// URI schema

The pool definition caccepts URIs matching the malloc:/// schema within its disks field for the purposes of provisioning memory-based disks. The general format is:

malloc:///malloc<DeviceId>?<parameters>

Where <DeviceId> is an integer value which uniquely identifies the device on that node, and where the parameter collection <parameters> may include the following:

Using File-backed Disks

Mayastor can use file-based disk emulation in place of physical pool disk devices, by employing the aio:/// URI schema within the pool's declaration in order to identify the location of the file resource.

apiVersion: "openebs.io/v1alpha1"
kind: DiskPool
metadata:
  name: filepool-1
  namespace: mayastor
spec:
  node: worker-node-1
  disks: ["aio:///var/tmp/disk1.img"]

apiVersion: "openebs.io/v1alpha1"
kind: DiskPool
metadata:
  name: filepool-1
  namespace: mayastor
spec:
  node: worker-node-1
  disks: ["aio:///tmp/disk1.img?blk_size=4096"]

The examples shown seek to create a pool using a file named "disk1.img", located in the /var/tmp directory of the Mayastor container's file system, as its member disk device. For this operation to succeed, the file must already exist on the specified path (which should be FULL path to the file) and this path must be accessible by the Mayastor pod instance running on the corresponding node.

The aio:/// schema requires no other parameters but optionally, "blk_size" may be specified. Block size accepts a value of either 512 or 4096, corresponding to the emulation of either a 512-byte or 4kB sector size device. If this parameter is omitted the device defaults to using a 512-byte sector size.

File-based disk devices are not over-provisioned; to create a 10GiB pool disk device requires that a 10GiB-sized backing file exist on a file system on an accessible path.

The preferred method of creating a backing file is to use the linux truncate command. The following example demonstrates the creation of a 1GiB-sized file named disk1.img within the directory /tmp.

truncate -s 1G /tmp/disk1.img

1. In order to start the upgrade process, the following previously deployed components have to be deleted.

• To delete the control-plane components, execute:

kubectl delete deploy core-agents -n mayastor
kubectl delete deploy csi-controller -n mayastor
kubectl delete deploy msp-operator -n mayastor
kubectl delete deploy rest -n mayastor
kubectl delete ds mayastor-csi -n mayastor

• Next, delete the associated RBAC operator. To do so, execute:

kubectl delete -f https://raw.githubusercontent.com/openebs/mayastor-control-plane/<version>/deploy/operator-rbac.yaml

helm template mayastor . -n mayastor --set etcd.persistence.storageClass="manual" --set loki-stack.loki.persistence.storageClassName="manual" --set etcd.initialClusterState=existing > helm_templates.yaml

• Next, update the helm_template.yaml file, add the following helm label to all the resources that are being created.

metadata: 
  annotations: 
    meta.helm.sh/release-name: $RELEASE_NAME 
    meta.helm.sh/release-namespace: $RELEASE_NAMESPACE 
  labels: 
    app.kubernetes.io/managed-by: Helm

• Copy the etcd and io-engine spec from the helm_templates.yaml and save it in two different files say, mayastor_2.0_etcd.yaml and mayastor_io_v2.0.yaml. Once done, remove the etcd and io-engine specs from helm_templates.yaml. These components need to be upgraded separately.

1. Install the new control-plane components using the helm-templates.yaml file.

kubectl apply -f helm_templates.yaml -n mayastor

• Verify the status of the pods. Upon successful deployment, all the pods will be in a running state.

kubectl get pods -n mayastor

NAME                                         READY   STATUS    RESTARTS   AGE
mayastor-65cxj                               1/1     Running   0          9m42s
mayastor-agent-core-7d7f59bbb8-nwptm         2/2     Running   0          104s
mayastor-api-rest-6d774fbdd8-hgrxj           1/1     Running   0          104s      mayastor-csi-controller-6469fdf8db-bgs2h     3/3     Running   0          104s
mayastor-csi-node-7zm2v                      2/2     Running   0          104s
mayastor-csi-node-gs76x                      2/2     Running   0          104s
mayastor-csi-node-mfqfq                      2/2     Running   0          104s
mayastor-etcd-0                              1/1     Running   0          13m
mayastor-etcd-1                              1/1     Running   0          13m
mayastor-etcd-2                              1/1     Running   0          13m
mayastor-loki-0                              1/1     Running   0          104s
mayastor-mwc9r                               1/1     Running   0          9m42s
mayastor-obs-callhome-588688bb4d-w9dl4       1/1     Running   0          104s
mayastor-operator-diskpool-8cd67554d-c4zpz   1/1     Running   0          104s
mayastor-promtail-66cj6                      1/1     Running   0          104s
mayastor-promtail-cx9m7                      1/1     Running   0          104s
mayastor-promtail-t789g                      1/1     Running   0          104s
mayastor-x8vtc                               1/1     Running   0          9m42s
nats-0                                       2/2     Running   0          13m
nats-1                                       2/2     Running   0          12m
nats-2                                       2/2     Running   0          12m

• Verify the etcd prefix and compat mode.

kubectl exec -it mayastor-etcd-0 -n mayastor -- bash
Defaulted container "etcd" out of: etcd, volume-permissions (init)
I have no name!@mayastor-etcd-0:/opt/bitnami/etcd$ export ETCDCTL_API=3
I have no name!@mayastor-etcd-0:/opt/bitnami/etcd$ etcdctl get --prefix ""

LINE NO.3 "mayastor_compat_v1":true - compat mode to look for. 
I have no name!@mayastor-etcd-0:/opt/bitnami/etcd$ etcdctl get --prefix ""
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/CoreRegistryConfig/db98f8bb-4afc-45d0-85b9-24c99cc443f2
{"id":"db98f8bb-4afc-45d0-85b9-24c99cc443f2","registration":"Automatic","mayastor_compat_v1":true}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/NexusSpec/069feb5e-ec65-4e97-b094-99262dfc9f44
uuid=8929e13f-99c0-4830-bcc2-d4b12a541b97"}},{"Replica":{"uuid":"9455811d-480e-4522-b94a-4352ba65cb73","share_uri":"nvmf://10.20.30.64:8420/nqn.2019-05.io.openebs:9455811d-480e-4522-b94a-4352ba65cb73?uuid=9455811d-480e-4522-b94a-4352ba65cb73"}}],"size":1073741824,"spec_status":{"Created":"Online"},"share":"nvmf","managed":true,"owner":"bf207797-b23d-447a-8d3f-98d378acfa8a","operation":null}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/NodeSpec/worker-0
{"id":"worker-0","endpoint":"10.20.30.56:10124","labels":{}}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/NodeSpec/worker-1
{"id":"worker-1","endpoint":"10.20.30.57:10124","labels":{}}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/NodeSpec/worker-2
{"id":"worker-2","endpoint":"10.20.30.64:10124","labels":{}}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/PoolSpec/pool-0
{"node":"worker-0","id":"pool-0","disks":["/dev/nvme0n1"],"status":{"Created":"Online"},"labels":{"openebs.io/created-by":"operator-diskpool"},"operation":null}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/PoolSpec/pool-1
{"node":"worker-1","id":"pool-1","disks":["/dev/nvme0n1"],"status":{"Created":"Online"},"labels":{"openebs.io/created-by":"operator-diskpool"},"operation":null}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/PoolSpec/pool-2
{"node":"worker-2","id":"pool-2","disks":["/dev/nvme0n1"],"status":{"Created":"Online"},"labels":{"openebs.io/created-by":"operator-diskpool"},"operation":null}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/ReplicaSpec/8929e13f-99c0-4830-bcc2-d4b12a541b97
{"name":"8929e13f-99c0-4830-bcc2-d4b12a541b97","uuid":"8929e13f-99c0-4830-bcc2-d4b12a541b97","size":1073741824,"pool":"pool-1","share":"nvmf","thin":false,"status":{"Created":"online"},"managed":true,"owners":{"volume":"bf207797-b23d-447a-8d3f-98d378acfa8a"},"operation":null}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/ReplicaSpec/9455811d-480e-4522-b94a-4352ba65cb73
{"name":"9455811d-480e-4522-b94a-4352ba65cb73","uuid":"9455811d-480e-4522-b94a-4352ba65cb73","size":1073741824,"pool":"pool-2","share":"nvmf","thin":false,"status":{"Created":"online"},"managed":true,"owners":{"volume":"bf207797-b23d-447a-8d3f-98d378acfa8a"},"operation":null}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/ReplicaSpec/f65d9888-7699-4c44-8ee2-f6aaa58dead0
{"name":"f65d9888-7699-4c44-8ee2-f6aaa58dead0","uuid":"f65d9888-7699-4c44-8ee2-f6aaa58dead0","size":1073741824,"pool":"pool-0","share":"none","thin":false,"status":{"Created":"online"},"managed":true,"owners":{"volume":"bf207797-b23d-447a-8d3f-98d378acfa8a"},"operation":null}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/StoreLeaseLock/CoreAgent/5e6787b9b88cdc5b
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/StoreLeaseOwner/CoreAgent
{"kind":"CoreAgent","lease_id":"5e6787b9b88cdc5b","instance_name":"mayastor-agent-core-7d7f59bbb8-nwptm"}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/VolumeSpec/bf207797-b23d-447a-8d3f-98d378acfa8a
{"uuid":"bf207797-b23d-447a-8d3f-98d378acfa8a","size":1073741824,"labels":{"local":"true"},"num_replicas":3,"status":{"Created":"Online"},"policy":{"self_heal":true},"topology":{"node":{"Explicit":{"allowed_nodes":["worker-1","worker-2","master","worker-0"],"preferred_nodes":["worker-2","master","worker-0","worker-1"]}},"pool":{"Labelled":{"exclusion":{},"inclusion":{"openebs.io/created-by":"operator-diskpool"}}}},"last_nexus_id":"069feb5e-ec65-4e97-b094-99262dfc9f44","operation":null,"thin":false,"target":{"node":"worker-0","nexus":"069feb5e-ec65-4e97-b094-99262dfc9f44","protocol":"nvmf","active":true,"config":{"controllerIdRange":{"start":1,"end":65519},"reservationKey":1,"reservationType":"ExclusiveAccess","preemptPolicy":"Holder"},"frontend":{"host_acl":[]}},"publish_context":null,"volume_group":null}
069feb5e-ec65-4e97-b094-99262dfc9f44
{"children":[{"healthy":true,"uuid":"f65d9888-7699-4c44-8ee2-f6aaa58dead0"},{"healthy":true,"uuid":"8929e13f-99c0-4830-bcc2-d4b12a541b97"},{"healthy":true,"uuid":"9455811d-480e-4522-b94a-4352ba65cb73"}],"clean_shutdown":false}
I have no name!@mayastor-etcd-0:/opt/bitnami/etcd$

• Verify if the DiskPools are online.

kubectl get dsp -n mayastor

NAME     NODE       STATE    POOL_STATUS   CAPACITY       USED         AVAILABLE
pool-0   worker-0   Online   Online        374710730752   3221225472   371489505280
pool-1   worker-1   Online   Online        374710730752   3221225472   371489505280
pool-2   worker-2   Online   Online        374710730752   3221225472   371489505280

• Next, verify the status of the volumes.

kubectl mayastor get volumes

ID                                    REPLICAS  TARGET-NODE  ACCESSIBILITY  STATUS SIZE        THIN-PROVISIONED
bf207797-b23d-447a-8d3f-98d378acfa8a  3         worker-0     nvmf           Online  1073741824  false

1. After upgrading control-plane components, the data-plane pods have to be upgraded. To do so, deploy the io-engine DaemonSet from Mayastor's new version.

Using the command given below, the data-plane pods (now io-engine pods) will be upgraded to Mayastor v2.0.

kubectl apply -f mayastor_io_v2.0.yaml -n mayastor

• Delete the previously deployed data-plane pods (mayastor-xxxxx). The data-plane pods need to be manually deleted as their update-strategy is set to delete. Upon successful deletion, the new io-engine pods will be up and running.

• NATS has been replaced by gRPC for Mayastor versions 2.0 or later. Hence, the NATS components (StatefulSets and services) have to be removed from the cluster.

kubectl delete sts nats -n mayastor
kubectl delete svc nats -n mayastor

1. After control-plane and io-engine, the etcd has to be upgraded. Before starting the etcd upgrade, label the etcd PV and PVCs with helm. Use the below example to create a labels.yaml file. This will be needed to make them helm compatible.

metadata:
  annotations:
    meta.helm.sh/release-name: mayastor
    meta.helm.sh/release-namespace: mayastor
  labels:
    app.kubernetes.io/managed-by: Helm

kubectl patch pvc <data-mayastor-etcd-x> --patch-file labels.yaml n mayastor 

kubectl patch pv <etcd-volume-x> --patch-file labels.yaml -n mayastor

• Next, deploy the etcd YAML. To deploy, execute:

kubectl apply -f mayastor_2.0_etcd.yaml -n mayastor

Now, verify the etcd space and compat mode, execute:

kubectl exec -it mayastor-etcd-0 -n mayastor -- bash
Defaulted container "etcd" out of: etcd, volume-permissions (init)
I have no name!@mayastor-etcd-0:/opt/bitnami/etcd$ export ETCDCTL_API=3
I have no name!@mayastor-etcd-0:/opt/bitnami/etcd$ etcdctl get --prefix ""

LINE NO.3 "mayastor_compat_v1":true - compat mode to look for. 
I have no name!@mayastor-etcd-0:/opt/bitnami/etcd$ etcdctl get --prefix ""
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/CoreRegistryConfig/db98f8bb-4afc-45d0-85b9-24c99cc443f2
{"id":"db98f8bb-4afc-45d0-85b9-24c99cc443f2","registration":"Automatic","mayastor_compat_v1":true}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/NexusSpec/069feb5e-ec65-4e97-b094-99262dfc9f44
uuid=8929e13f-99c0-4830-bcc2-d4b12a541b97"}},{"Replica":{"uuid":"9455811d-480e-4522-b94a-4352ba65cb73","share_uri":"nvmf://10.20.30.64:8420/nqn.2019-05.io.openebs:9455811d-480e-4522-b94a-4352ba65cb73?uuid=9455811d-480e-4522-b94a-4352ba65cb73"}}],"size":1073741824,"spec_status":{"Created":"Online"},"share":"nvmf","managed":true,"owner":"bf207797-b23d-447a-8d3f-98d378acfa8a","operation":null}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/NodeSpec/worker-0
{"id":"worker-0","endpoint":"10.20.30.56:10124","labels":{}}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/NodeSpec/worker-1
{"id":"worker-1","endpoint":"10.20.30.57:10124","labels":{}}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/NodeSpec/worker-2
{"id":"worker-2","endpoint":"10.20.30.64:10124","labels":{}}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/PoolSpec/pool-0
{"node":"worker-0","id":"pool-0","disks":["/dev/nvme0n1"],"status":{"Created":"Online"},"labels":{"openebs.io/created-by":"operator-diskpool"},"operation":null}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/PoolSpec/pool-1
{"node":"worker-1","id":"pool-1","disks":["/dev/nvme0n1"],"status":{"Created":"Online"},"labels":{"openebs.io/created-by":"operator-diskpool"},"operation":null}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/PoolSpec/pool-2
{"node":"worker-2","id":"pool-2","disks":["/dev/nvme0n1"],"status":{"Created":"Online"},"labels":{"openebs.io/created-by":"operator-diskpool"},"operation":null}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/ReplicaSpec/8929e13f-99c0-4830-bcc2-d4b12a541b97
{"name":"8929e13f-99c0-4830-bcc2-d4b12a541b97","uuid":"8929e13f-99c0-4830-bcc2-d4b12a541b97","size":1073741824,"pool":"pool-1","share":"nvmf","thin":false,"status":{"Created":"online"},"managed":true,"owners":{"volume":"bf207797-b23d-447a-8d3f-98d378acfa8a"},"operation":null}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/ReplicaSpec/9455811d-480e-4522-b94a-4352ba65cb73
{"name":"9455811d-480e-4522-b94a-4352ba65cb73","uuid":"9455811d-480e-4522-b94a-4352ba65cb73","size":1073741824,"pool":"pool-2","share":"nvmf","thin":false,"status":{"Created":"online"},"managed":true,"owners":{"volume":"bf207797-b23d-447a-8d3f-98d378acfa8a"},"operation":null}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/ReplicaSpec/f65d9888-7699-4c44-8ee2-f6aaa58dead0
{"name":"f65d9888-7699-4c44-8ee2-f6aaa58dead0","uuid":"f65d9888-7699-4c44-8ee2-f6aaa58dead0","size":1073741824,"pool":"pool-0","share":"none","thin":false,"status":{"Created":"online"},"managed":true,"owners":{"volume":"bf207797-b23d-447a-8d3f-98d378acfa8a"},"operation":null}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/StoreLeaseLock/CoreAgent/5e6787b9b88cdc5b
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/StoreLeaseOwner/CoreAgent
{"kind":"CoreAgent","lease_id":"5e6787b9b88cdc5b","instance_name":"mayastor-agent-core-7d7f59bbb8-nwptm"}
/openebs.io/mayastor/apis/v0/clusters/ce05eb25-50cc-400a-a57f-37e6a5ed9bef/namespaces/mayastor/VolumeSpec/bf207797-b23d-447a-8d3f-98d378acfa8a
{"uuid":"bf207797-b23d-447a-8d3f-98d378acfa8a","size":1073741824,"labels":{"local":"true"},"num_replicas":3,"status":{"Created":"Online"},"policy":{"self_heal":true},"topology":{"node":{"Explicit":{"allowed_nodes":["worker-1","worker-2","master","worker-0"],"preferred_nodes":["worker-2","master","worker-0","worker-1"]}},"pool":{"Labelled":{"exclusion":{},"inclusion":{"openebs.io/created-by":"operator-diskpool"}}}},"last_nexus_id":"069feb5e-ec65-4e97-b094-99262dfc9f44","operation":null,"thin":false,"target":{"node":"worker-0","nexus":"069feb5e-ec65-4e97-b094-99262dfc9f44","protocol":"nvmf","active":true,"config":{"controllerIdRange":{"start":1,"end":65519},"reservationKey":1,"reservationType":"ExclusiveAccess","preemptPolicy":"Holder"},"frontend":{"host_acl":[]}},"publish_context":null,"volume_group":null}
069feb5e-ec65-4e97-b094-99262dfc9f44
{"children":[{"healthy":true,"uuid":"f65d9888-7699-4c44-8ee2-f6aaa58dead0"},{"healthy":true,"uuid":"8929e13f-99c0-4830-bcc2-d4b12a541b97"},{"healthy":true,"uuid":"9455811d-480e-4522-b94a-4352ba65cb73"}],"clean_shutdown":false}
    I have no name!@mayastor-etcd-0:/opt/bitnami/etcd$

1. Once all the components have been upgraded, the HA module can now be enabled via the helm upgrade command.

helm upgrade --install mayastor . -n mayastor --set etcd.persistence.storageClass="manual" --set loki-stack.loki.persistence.storageClassName="manual" --set agents.ha.enabled="true"

Release "mayastor" does not exist. Installing it now.
NAME: mayastor
LAST DEPLOYED: Tue Apr 25 19:20:53 2023
NAMESPACE: mayastor
STATUS: deployed
REVISION: 1
NOTES:
OpenEBS Mayastor has been installed. Check its status by running:
$ kubectl get pods -n mayastor

1. This concludes the process of legacy upgrade. Run the below commands to verify the upgrade,

helm list -n mayastor

NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                           APP VERSION
mayastor        mayastor        1               2023-04-25 19:20:53.43928058 +0000 UTC  deployed        mayastor-2.1.0                  2.1.0

kubectl get pods -n mayastor

NAME                                         READY   STATUS    RESTARTS   AGE
mayastor-agent-core-7d7f59bbb8-nwptm         2/2     Running   0          34m
mayastor-agent-ha-node-fblrn                 1/1     Running   0          6m29s
mayastor-agent-ha-node-g6rf9                 1/1     Running   0          6m29s
mayastor-agent-ha-node-ktjvz                 1/1     Running   0          6m29s
mayastor-api-rest-6d774fbdd8-hgrxj           1/1     Running   0          34m
mayastor-csi-controller-6469fdf8db-bgs2h     3/3     Running   0          34m
mayastor-csi-node-7zm2v                      2/2     Running   0          34m
mayastor-csi-node-gs76x                      2/2     Running   0          34m
mayastor-csi-node-mfqfq                      2/2     Running   0          34m
mayastor-etcd-0                              1/1     Running   0          4m7s
mayastor-etcd-1                              1/1     Running   0          5m16s
mayastor-etcd-2                              1/1     Running   0          6m28s
mayastor-io-engine-6n6bh                     2/2     Running   0          25m
mayastor-io-engine-7gpsj                     2/2     Running   0          25m
mayastor-io-engine-95jjn                     2/2     Running   0          25m
mayastor-loki-0                              1/1     Running   0          34m
mayastor-obs-callhome-588688bb4d-w9dl4       1/1     Running   0          34m
mayastor-operator-diskpool-8cd67554d-c4zpz   1/1     Running   0          34m
mayastor-promtail-66cj6                      1/1     Running   0          34m
mayastor-promtail-cx9m7                      1/1     Running   0          34m
mayastor-promtail-t789g                      1/1     Running   0          34m
nats-0                                       2/2     Running   0          45m
nats-1                                       2/2     Running   0          45m
nats-2                                       2/2     Running   0          45m

kubectl mayastor get volumes

ID                                    REPLICAS  TARGET-NODE  ACCESSIBILITY  STATUS  SIZE        THIN-PROVISIONED
bf207797-b23d-447a-8d3f-98d378acfa8a  3         worker-0     nvmf           Online  1073741824  false

Worker Node OS

Cluster Orchestration

Kubernetes versions:

• v1.25.10

• v1.23.7

• v1.22.10

• v1.21.13

This documentation provides a comprehensive guide on migrating CStor application volumes to Mayastor. We utilize Velero for the backup and restoration process, enabling a seamless transition from a CStor cluster to Mayastor. This example specifically focuses on a GKE cluster.

Velero offers support for the backup and restoration of Kubernetes volumes attached to pods directly from the volume's file system. This is known as File System Backup (FSB) or Pod Volume Backup. The data movement is facilitated through the use of modules from free, open-source backup tools such as Restic (which is the tool of choice in this guide).

• For detailed instructions on Velero basic installation, visit https://velero.io/docs/v1.11/basic-install/.

Basics

When a Mayastor volume is provisioned based on a StorageClass which has a replication factor greater than one (set by its repl parameter), the control plane will attempt to maintain through a 'Kubernetes-like' reconciliation loop that number of identical copies of the volume's data "replicas" (a replica is a nexus target "child") at any point in time. When a volume is first provisioned the control plane will attempt to create the required number of replicas, whilst adhering to its internal heuristics for their location within the cluster (which will be discussed shortly). If it succeeds, the volume will become available and will bind with the PVC. If the control plane cannot identify a sufficient number of eligble Mayastor Pools in which to create required replicas at the time of provisioning, the operation will fail; the Mayastor Volume will not be created and the associated PVC will not be bound. Kubernetes will periodically re-try the volume creation and if at any time the appropriate number of pools can be selected, the volume provisioning should succeed.

Once a volume is processing I/O, each of its replicas will also receive I/O. Reads are round-robin distributed across replicas, whilst writes must be written to all. In a real world environment this is attended by the possibility that I/O to one or more replicas might fail at any time. Possible reasons include transient loss of network connectivity, node reboots, node or disk failure. If a volume's nexus (NVMe controller) encounters 'too many' failed I/Os for a replica, then that replica's child status will be marked Faulted and it will no longer receive I/O requests from the nexus. It will remain a member of the volume, whose departure from the desired state with respect to replica count will be reflected with a volume status of Degraded. How many I/O failures are considered "too many" in this context is outside the scope of this discussion.

The control plane will first 'retire' the old, faulted one which will then no longer be associated to the volume. Once retired, a replica will become available for garbage collection (deletion from the Mayastor Pool containing it), assuming that the nature of the failure was such that the pool itself is still viable (i.e. the underlying disk device is still accessible). Then it will attempt to restore the desired state (replica count) by creating a new replica, following its replica placement rules. If it succeeds, the nexus will "rebuild" that new replica - performing a full copy of all data from a healthy replica Online, i.e. the source. This process can proceed whilst the volume continues to process application I/Os although it will contend for disk throughput at both the source and destination disks.

If a nexus is cleanly restarted, i.e. the Mayastor pod hosting it restarts gracefully, with the assistance of the control plane it will 'recompose' itself; all of the previous healthy member replicas will be re-attached to it. If previously faulted replicas are available to be re-connected (Online), then the control plane will attempt to reuse and rebuild them directly, rather than seek replacements for them first. This edge-case therefore does not result in the retirement of the affected replicas; they are simply reused. If the rebuild fails then we follow the above process of removing a Faulted replica and adding a new one. On an unclean restart (i.e. the Mayastor pod hosting the nexus crashes or is forcefully deleted) only one healthy replica will be re-attached and all other replicas will eventually be rebuilt.

Once provisioned, the replica count of a volume can be changed using the kubectl-mayastor plugin scale subcommand. The value of the num_replicas field may be either increased or decreased by one and the control plane will attempt to satisfy the request by creating or destroying a replicas as appropriate, following the same replica placement rules as described herein. If the replica count is reduced, faulted replicas will be selected for removal in preference to healthy ones.

Replica Placement Heuristics

Accurate predictions of the behaviour of Mayastor with respect to replica placement and management of replica faults can be made by reference to these 'rules', which are a simplified representation of the actual logic:

• "Rule 1": A volume can only be provisioned if the replica count (and capacity) of its StorageClass can be satisfied at the time of creation

• "Rule 2": Every replica of a volume must be placed on a different Mayastor Node)

• "Rule 3": Children with the state Faulted are always selected for retirement in preference to those with state Online

N.B.: By application of the 2nd rule, replicas of the same volume cannot exist within different pools on the same Mayastor Node.

Example Scenarios

Scenario One

A cluster has two Mayastor nodes deployed, "Node-1" and "Node-2". Each Mayastor node hosts two Mayastor pools and currently, no Mayastor volumes have been defined. Node-1 hosts pools "Pool-1-A" and "Pool-1-B", whilst Node-2 hosts "Pool-2-A and "Pool-2-B". When a user creates a PVC from a StorageClass which defines a replica count of 2, the Mayastor control plane will seek to place one replica on each node (it 'follows' Rule 2). Since in this example it can find a suitable candidate pool with sufficient free capacity on each node, the volume is provisioned and becomes "healthy" (Rule 1). Pool-1-A is selected on Node-1, and Pool-2-A selected on Node-2 (all pools being of equal capacity and replica count, in this initial 'clean' state).

Sometime later, the physical disk of Pool-2-A encounters a hardware failure and goes offline. The volume is in use at the time, so its nexus (NVMe controller) starts to receive I/O errors for the replica hosted in that Pool. The associated replica's child from Pool-2-A enters the Faulted state and the volume state becomes Degraded (as seen through the kubectl-mayastor plugin).

Expected Behaviour: The volume will maintain read/write access for the application via the remaining healthy replica. The faulty replica from Pool-2-A will be removed from the Nexus thus changing the nexus state to Online as the remaining is healthy. A new replica is created on either Pool-2-A or Pool-2-B and added to the nexus. The new replica child is rebuilt and eventually the state of the volume returns to Online.

Scenario Two

A cluster has three Mayastor nodes deployed, "Node-1", "Node-2" and "Node-3". Each Mayastor node hosts one pool: "Pool-1" on Node-1, "Pool-2" on Node-2 and "Pool-3" on Node-3. No Mayastor volumes have yet been defined; the cluster is 'clean'. A user creates a PVC from a StorageClass which defines a replica count of 2. The control plane determines that it is possible to accommodate one replica within the available capacity of each of Pool-1 and Pool-2, and so the volume is created. An application is deployed on the cluster which uses the PVC, so the volume receives I/O.

Unfortunately, due to user error the SAN LUN which is used to persist Pool-2 becomes detached from Node-2, causing I/O failures in the replica which it hosts for the volume. As with scenario one, the volume state becomes Degraded and the faulted child's becomes Faulted.

Expected Behaviour: Since there is a Mayastor pool on Node-3 which has sufficient capacity to host a replacement replica, a new replica can be created (Rule 2: this 'third' incoming replica isn't located on either of the nodes that the two original ones are). The faulted replica in Pool-2 is retired from the nexus and a new replica is created on Pool-3 and added to the nexus. The new replica is rebuilt and eventually the state of the volume returns to Online.

Scenario Three

In the cluster from Scenario three, sometime after the Mayastor volume has returned to the Online state, a user scales up the volume, increasing the num_replicas value from 2 to 3. Before doing so they corrected the SAN misconfiguration and ensured that the pool on Node-2 was Online.

Expected Behaviour: The control plane will attempt to reconcile the difference in current (replicas = 2) and desired (replicas = 3) states. Since Node-2 no longer hosts a replica for the volume (the previously faulted replica was successfully retired and is no longer a member of the volume's nexus), the control plane will select it to host the new replica required (Rule 2 permits this). The volume state will become initially Degraded to reflect the difference in actual vs required redundant data copies but a rebuild of the new replica will be performed and eventually the volume state will be Online again.

Scenario Four

A cluster has three Mayastor nodes deployed; "Node-1", "Node-2" and "Node-3". Each Mayastor node hosts two Mayastor pools and currently, no Mayastor volumes have been defined. Node-1 hosts pools "Pool-1-A" and "Pool-1-B", whilst Node-2 hosts "Pool-2-A and "Pool-2-B" and Node-3 hosts "Pool-3-A" and "Pool-3-B". A single volume exists in the cluster, which has a replica count of 3. The volume's replicas are all healthy and are located on Pool-1-A, Pool-2-A and Pool-3-A. An application is using the volume, so all replicas are receiving I/O.

The host Node-3 goes down causing failure of all I/O sent to the replica it hosts from Pool-3-A.

Expected Behaviour: The volume will enter and remain in the Degraded state. The associated child from the replica from Pool-3-A will be in the state Faulted, as observed in the volume through the kubectl-mayastor plugin. Said replica will be removed from the Nexus thus changing the nexus state to Online as the other replicas are healthy. The replica will then be disowned from the volume (it won't be possible to delete it since the host is down). Since Rule 2 dictates that every replica of a volume must be placed on a different Mayastor Node no new replica can be created at this point and the volume remains Degraded indefinitely.

Scenario Five

Given the post-host failure situation of Scenario four, the user scales down the volume, reducing the value of num_replicas from 3 to 2.

Expected Behaviour: The control plane will reconcile the actual (replicas=3) vs desired (replicas=2) state of the volume. The volume state will become Online again.

Scenario Six

In scenario Five, after scaling down the Mayastor volume the user waits for the volume state to become Online again. The desired and actual replica count are now 2. The volume's replicas are located in pools on both Node-1 and Node-2. The Node-3 is now back up and its pools Pool-3-A and Pool-3-B are Online. The user then scales the volume again, increasing the num_replicas from 2 to 3 again.

Expected Behaviour: The volume's state will become Degraded, reflecting the difference in desired vs actual replica count. The control plane will select a pool on Node-3 as the location for the new replica required. Node-3 is therefore again a suitable candidate and has online pools with sufficient capacity.

Mayastor default information collection

By default, Mayastor collects basic information related to the number and scale of user-deployed instances. The collected data is anonymous and is encrypted at rest. This data is used to understand storage usage trends, which in turn helps maintainers prioritize their contributions to maximize the benefit to the community as a whole.

A summary of the information collected is given below:

Storage location of collected data

The collected information is stored on behalf of the OpenEBS project by DataCore Software Inc. in data centers located in Texas, USA.

Disable specific data collection

To disable collection of usage data or generation of events, the following Helm command, along with the flag, can either be executed during installation or can be re-executed post-installation.

Disable collection of usage data

To disable the collection of data metrics from the cluster, add the following flag to the Helm install command.

Disable generation of events data

When eventing is enabled, NATS pods are created to gather various events from the cluster, including statistical metrics such as pools created. To deactivate eventing within the cluster, include the following flag in the Helm installation command.

By following the given steps, you can successfully migrate etcd from one node to another during maintenance activities like node drain etc., ensuring the continuity and integrity of the etcd data.

Step 1: Draining the etcd Node

1. Assuming we have a three-node cluster with three etcd replicas, verify the etcd pods with the following commands:

Command to verify pods:

1. From etcd-0/1/2 we could see all the values are registered in database, once we migrated etcd to new node, all the key-value pairs should be available across all the pods. Run the following commands from any etcd pod.

Commands to get etcd data:

1. In this example, we drain the etcd node worker-0 and migrate it to the next available node (in this case, the worker-4 node), use the following command:

Command to drain the node:

Step 2: Migrating etcd to the New Node

1. After draining the worker-0 node, the etcd pod will be scheduled on the next available node, which is the worker-4 node.

2. The pod may end up in a CrashLoopBackOff status with specific errors in the logs.

3. When the pod is scheduled on the new node, it attempts to bootstrap the member again, but since the member is already registered in the cluster, it fails to start the etcd server with the error message member already bootstrapped.

4. To fix this issue, change the cluster's initial state from new to existing by editing the StatefulSet for etcd:

Command to check new etcd pod status

Command to edit the StatefulSet:

Step 3: Validating etcd Key-Value Pairs

Run the appropriate command from the migrated etcd pod to validate the key-value pairs and ensure they are the same as in the existing etcd. This step is crucial to avoid any data loss during the migration process.

By default, Mayastor allows the creation of three etcd members. If you wish to increase the number of etcd replicas, you will encounter an error. However, you can make the necessary configuration changes discussed in this guide to make it work.

Overview of StatefulSets

StatefulSets are Kubernetes resources designed for managing stateful applications. They provide stable network identities and persistent storage for pods. StatefulSets ensure ordered deployment and scaling, support persistent volume claims, and manage the state of applications. They are commonly used for databases, messaging systems, and distributed file systems. Here's how StatefulSets function:

• For a StatefulSet with N replicas, when pods are deployed, they are created sequentially in order from {0..N-1}.

• When pods are deleted, they are terminated in reverse order from {N-1..0}.

• Before a scaling operation is applied to a pod, all of its predecessors must be running and ready.

• Before a pod is terminated, all of its successors must be completely shut down.

• Mayastor uses etcd database for persisting configuration and state information. Etcd is setup as a Kubernetes StatefulSet when Mayastor is installed.

• From etcd-0/1/2, we can see that all the values are registered in the database. Once we scale up etcd with "n" replicas, all the key-value pairs should be available across all the pods.

To scale up the etcd members, the following steps can be performed:

1. Add a new etcd member

2. Add a peer URL

3. Create a PV (Persistent Volume)

4. Validate key-value pairs

Step 1: Adding a New etcd Member (Scaling Up etcd Replica)

To increase the number of replicas to 4, use the following kubectl scale command:

The new pod will be created on available nodes but will be in a pending state as there is no PV/PVC created to bind the volumes.

Step 2: Add a New Peer URL

Before creating a PV, we need to add the new peer URL (mayastor-etcd-3=http://mayastor-etcd-3.mayastor-etcd-headless.mayastor.svc.cluster.local:2380) and change the cluster's initial state from "new" to "existing" so that the new member will be added to the existing cluster when the pod comes up after creating the PV. Since the new pod is still in a pending state, the changes will not be applied to the other pods as they will be restarted in reverse order from {N-1..0}. It is expected that all of its predecessors must be running and ready.

Step 3: Create a Persistent Volume

Create a PV with the following YAML. Change the pod name/claim name based on the pod's unique identity.

Step 4: Validate Key-Value Pairs

Run the following command from the new etcd pod and ensure that the values are the same as those in etcd-0/1/2. Otherwise, it indicates a data loss issue.