This documentation provides a comprehensive guide on migrating CStor application volumes to Mayastor. We utilize Velero for the backup and restoration process, enabling a seamless transition from a CStor cluster to Mayastor. This example specifically focuses on a GKE cluster.
Velero offers support for the backup and restoration of Kubernetes volumes attached to pods directly from the volume's file system. This is known as File System Backup (FSB) or Pod Volume Backup. The data movement is facilitated through the use of modules from free, open-source backup tools such as Restic (which is the tool of choice in this guide).
For cloud providers, you can find the necessary plugins here.
For detailed Velero GKE configuration prerequisites, refer to this resource.
It's essential to note that Velero requires an object storage bucket for storing backups, and in our case, we use a Google Cloud Storage (GCS) bucket.
For detailed instructions on Velero basic installation, visit https://velero.io/docs/v1.11/basic-install/.
Backup from cStor
If you are deploying databases using operators, you need to find a way to actively modify the entire deployment through the operator. This ensures that you control and manage changes effectively within the operator-driven database deployment.
Step 1: Backup from cStor Cluster
Currently, we have a cStor cluster as the source, with a clustered MongoDB running as a StatefulSet using cStor volumes.
kubectl get pods
NAME READY STATUS RESTARTS AGE
mongo-client-758ddd54cc-h2gwl 1/1 Running 0 47m
mongod-0 1/1 Running 0 47m
mongod-1 1/1 Running 0 44m
mongod-2 1/1 Running 0 42m
ycsb-775fc86c4b-kj5vv 1/1 Running 0 47m
kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mongodb-persistent-storage-claim-mongod-0 Bound pvc-cb115a0b-07f4-4912-b686-e160e8a0690d 3Gi RWO cstor-csi-disk 54m
mongodb-persistent-storage-claim-mongod-1 Bound pvc-c9214764-7670-4cda-87e3-82f0bc59d8c7 3Gi RWO cstor-csi-disk 52m
mongodb-persistent-storage-claim-mongod-2 Bound pvc-fc1f7ed7-d99e-40c7-a9b7-8d6244403a3e 3Gi RWO cstor-csi-disk 50m
kubectl get cvc -n openebs
NAME CAPACITY STATUS AGE
pvc-c9214764-7670-4cda-87e3-82f0bc59d8c7 3Gi Bound 53m
pvc-cb115a0b-07f4-4912-b686-e160e8a0690d 3Gi Bound 55m
pvc-fc1f7ed7-d99e-40c7-a9b7-8d6244403a3e 3Gi Bound 50m
Step 2: Install Velero
For the prerequisites, refer to the overview section.
Verify the Velero namespace for Node Agent and Velero pods:
kubectl get pods -n velero
NAME READY STATUS RESTARTS AGE
node-agent-cwkrn 1/1 Running 0 43s
node-agent-qg6hd 1/1 Running 0 43s
node-agent-v6xbk 1/1 Running 0 43s
velero-56c45f5c64-4hzn7 1/1 Running 0 43s
Step 3: Data Validation
On the Primary Database (mongo-0) you can see some sample data.
You can also see the data available on the replicated secondary databases.
Step 4: Take Velero Backup
MongoDB uses replication, and data partitioning (sharding) for high availability and scalability. Taking a backup of the primary database is enough as the data gets replicated to the secondary databases. Restoring both primary and secondary at the same time can cause data corruption.
Velero supports two approaches for discovering pod volumes to be backed up using FSB:
Opt-in approach: Annotate pods containing volumes to be backed up.
Opt-out approach: Backup all pod volumes with the ability to opt-out specific volumes.
Opt-In for Primary MongoDB Pod:
To ensure that our primary MongoDB pod, which receives writes and replicates data to secondary pods, is included in the backup, we need to annotate it as follows:
Create a backup of the entire namespace. If any other applications run in the same namespace as MongoDB, we can exclude them from the backup using labels or flags from the Velero CLI:
Backup request "mongo-backup-13-09-23" submitted successfully.
Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup will continue in the background.
...........
Backup completed with status: Completed. You may check for more information using the commands `velero backup describe mongo-backup-13-09-23` and `velero backup logs mongo-backup-13-09-23`.
Backup Verification:
To check the status of the backup using the Velero CLI, you can use the following command. If the backup fails for any reason, you can inspect the logs with the velero backup logs command:
Before you begin, make sure you have the following:
Access to a Kubernetes cluster with Velero installed.
A backup of your Mongo database created using Velero.
Mayastor configured in your Kubernetes environment.
Step 1: Install Velero with GCP Provider on Destination (Mayastor Cluster)
Install Velero with the GCP provider, ensuring you use the same values for the BUCKET-NAME and SECRET-FILENAME placeholders that you used originally. These placeholders should be replaced with your specific values:
CustomResourceDefinition/backuprepositories.velero.io: attempting to create resource
CustomResourceDefinition/backuprepositories.velero.io: attempting to create resource client
CustomResourceDefinition/backuprepositories.velero.io: created
CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: attempting to create resource client
CustomResourceDefinition/backups.velero.io: created
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource client
CustomResourceDefinition/backupstoragelocations.velero.io: created
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource client
CustomResourceDefinition/deletebackuprequests.velero.io: created
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource client
CustomResourceDefinition/downloadrequests.velero.io: created
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumebackups.velero.io: created
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumerestores.velero.io: created
CustomResourceDefinition/restores.velero.io: attempting to create resource
CustomResourceDefinition/restores.velero.io: attempting to create resource client
CustomResourceDefinition/restores.velero.io: created
CustomResourceDefinition/schedules.velero.io: attempting to create resource
CustomResourceDefinition/schedules.velero.io: attempting to create resource client
CustomResourceDefinition/schedules.velero.io: created
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource client
CustomResourceDefinition/serverstatusrequests.velero.io: created
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource client
CustomResourceDefinition/volumesnapshotlocations.velero.io: created
Waiting for resources to be ready in cluster...
Namespace/velero: attempting to create resource
Namespace/velero: attempting to create resource client
Namespace/velero: created
ClusterRoleBinding/velero: attempting to create resource
ClusterRoleBinding/velero: attempting to create resource client
ClusterRoleBinding/velero: created
ServiceAccount/velero: attempting to create resource
ServiceAccount/velero: attempting to create resource client
ServiceAccount/velero: created
Secret/cloud-credentials: attempting to create resource
Secret/cloud-credentials: attempting to create resource client
Secret/cloud-credentials: created
BackupStorageLocation/default: attempting to create resource
BackupStorageLocation/default: attempting to create resource client
BackupStorageLocation/default: created
VolumeSnapshotLocation/default: attempting to create resource
VolumeSnapshotLocation/default: attempting to create resource client
VolumeSnapshotLocation/default: created
Deployment/velero: attempting to create resource
Deployment/velero: attempting to create resource client
Deployment/velero: created
DaemonSet/node-agent: attempting to create resource
DaemonSet/node-agent: attempting to create resource client
DaemonSet/node-agent: created
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.
thulasiraman_ilangovan@cloudshell:~$
Step 2: Verify Backup Availability
Check the availability of your previously-saved backups. If the credentials or bucket information doesn't match, you won't be able to see the backups:
Restore request "mongo-restore-13-09-23" submitted successfully.
Run `velero restore describe mongo-restore-13-09-23` or `velero restore logs mongo-restore-13-09-23` for more details.
Step 4: Check Restore Status
You can check the status of the restore process by using the velero get restore command.
velero get restore
When Velero performs a restore, it deploys an init container within the application pod, responsible for restoring the volume. Initially, the restore status will be InProgress.
Your storage class was originally set to cstor-csi-disk because you imported this PVC from a cStor volume, the status might temporarily stay as In Progress and your PVC will be in Pending status.
Step 5: Backup PVC and Change Storage Class
Retrieve the current configuration of the PVC which is in Pending status using the following command:
kubectl get pvc mongodb-persistent-storage-claim-mongod-0 -o yaml > pvc-mongo.yaml
Confirm that the PVC configuration has been saved by checking its existence with this command:
ls -lrt | grep pvc-mongo.yaml
Edit the pvc-mongo.yaml file to update its storage class. Below is the modified PVC configuration with mayastor-single-replica set as the new storage class:
The statefulset for Mongo will still have the cstor-csi-disk storage class at this point. This will be addressed in the further steps.
Once the PVC has been successfully deleted, you can recreate it using the updated configuration from the pvc-mongo.yaml file. Apply the new configuration with the following command:
kubectl apply -f pvc-mongo.yaml
Step 7: Check Velero init container
After recreating the PVC with Mayastor storageClass, you will observe the presence of a Velero initialization container within the application pod. This container is responsible for restoring the required volumes.
You can check the status of the restore operation by running the following command:
kubectl describe pod <enter_your_pod_name>
The output will display the pods' status, including the Velero initialization container. Initially, the status might show as "Init:0/1," indicating that the restore process is in progress.
You can track the progress of the restore by running:
velero get restore
NAME BACKUP STATUS STARTED COMPLETED ERRORS WARNINGS CREATED SELECTOR
mongo-restore-13-09-23 mongo-backup-13-09-23 Completed 2023-09-13 13:56:19 +0000 UTC 2023-09-13 14:06:09 +0000 UTC 0 4 2023-09-13 13:56:19 +0000 UTC <none>
You can then verify the data restoration by accessing your MongoDB instance. In the provided example, we used the "mongosh" shell to connect to the MongoDB instance and check the databases and their content. The data should reflect what was previously backed up from the cStor storage.
Due to the statefulset's configuration with three replicas, you will notice that the mongo-1 pod is created but remains in a Pending status. This behavior is expected as we have the storage class set to cStor in statefulset configuration.
Step 9: Capture the StatefulSet Configuration and Modify Storage Class
Capture the current configuration of the StatefulSet for MongoDB by running the following command:
kubectl get sts mongod -o yaml > sts-mongo-original.yaml
This command will save the existing StatefulSet configuration to a file named sts-mongo-original.yaml. Next, edit this YAML file to change the storage class to mayastor-single-replica.
Verify data replication on the secondary database to ensure synchronization.
root@mongod-1:/# mongosh mongodb://admin:admin@mongod-1.mongodb-service.default.svc.cluster.local:27017
Current Mongosh Log ID: 6501c744eb148521b3716af5
Connecting to: mongodb://<credentials>@mongod-1.mongodb-service.default.svc.cluster.local:27017/?directConnection=true&appName=mongosh+1.10.6
Using MongoDB: 7.0.1
Using Mongosh: 1.10.6
For mongosh info see: https://docs.mongodb.com/mongodb-shell/
------
The server generated these startup warnings when booting
2023-09-13T14:19:37.984+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
2023-09-13T14:19:38.679+00:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted
2023-09-13T14:19:38.679+00:00: You are running this process as the root user, which is not recommended
2023-09-13T14:19:38.679+00:00: vm.max_map_count is too low
------
rs0 [direct: secondary] test> use mydb
switched to db mydb
rs0 [direct: secondary] mydb> db.getMongo().setReadPref('secondary')
rs0 [direct: secondary] mydb> db.accounts.find()
[
{
_id: ObjectId("65019e2f183959fbdbd23f00"),
name: 'john',
total: '1058'
},
{
_id: ObjectId("65019e2f183959fbdbd23f01"),
name: 'jane',
total: '6283'
},
{
_id: ObjectId("65019e31183959fbdbd23f02"),
name: 'james',
total: '472'
}
]
rs0 [direct: secondary] mydb>