Links
Comment on page

Basic Architecture

The objective of this section is to provide the user and evaluator of Mayastor with a topological view of the gross anatomy of a Mayastor deployment. A description will be made of the expected pod inventory of a correctly deployed cluster, the roles and functions of the constituent pods and related Kubernetes resource types, and of the high level interactions between them and the orchestration thereof.
More detailed guides to Mayastor's components, their design and internal structure, and instructions for building Mayastor from source, are maintained within the project's GitHub repository.

Dramatis Personae

Name
Resource Type
Function
Frequency in Cluster
Control Plane
agent-core
Deployment
Principle control plane actor
Single
csi-controller
Deployment
Hosts Mayastor's CSI controller implementation and CSI provisioner side car
Single
api-rest
Pod
Hosts the public API REST server
Single
api-rest
Service
Exposes the REST API server
operator-diskpool
Deployment
Hosts DiskPool operator
Single
csi-node
DaemonSet
Hosts CSI Driver node plugin containers
All worker nodes
etcd
StatefulSet
Hosts etcd Server container
Configurable(Recommended: Three replicas)
etcd
Service
Exposes etcd DB endpoint
Single
etcd-headless
Service
Exposes etcd DB endpoint
Single
io-engine
DaemonSet
Hosts Mayastor I/O engine
User-selected nodes
DiskPool
CRD
Declares a Mayastor pool's desired state and reflects its current state
User-defined, one or many
Additional components
metrics-exporter-pool
Sidecar container (within io-engine DaemonSet)
Exports pool related metrics in Prometheus format
All worker nodes
pool-metrics-exporter
Service
Exposes exporter API endpoint to Prometheus
Single
promtail
DaemonSet
Scrapes logs of Mayastor-specific pods and exports them to Loki
All worker nodes
loki
StatefulSet
Stores the historical logs exported by promtail pods
Single
loki
Service
Exposes the Loki API endpoint via ClusterIP
Single

Component Roles

io-engine

The io-engine pod encapsulates Mayastor containers, which implement the I/O path from the block devices at the persistence layer, up to the relevant initiators on the worker nodes mounting volume claims. The Mayastor process running inside this container performs four major functions:
  • Creates and manages DiskPools hosted on that node.
  • Creates, exports, and manages volume controller objects hosted on that node.
  • Creates and exposes replicas from DiskPools hosted on that node over NVMe-TCP.
  • Provides a gRPC interface service to orchestrate the creation, deletion and management of the above objects, hosted on that node. Before the io-engine pod starts running, an init container attempts to verify connectivity to the agent-core in the namespace where Mayastor has been deployed. If a connection is established, the io-engine pod registers itself over gRPC to the agent-core. In this way, the agent-core maintains a registry of nodes and their supported api-versions. The scheduling of these pods is determined declaratively by using a DaemonSet specification. By default, a nodeSelector field is used within the pod spec to select all worker nodes to which the user has attached the label openebs.io/engine=mayastor as recipients of an io-engine pod. In this way, the node count and location are set appropriately to the hardware configuration of the worker nodes, and the capacity and performance demands of the cluster.

csi-node

The csi-node pods within a cluster implement the node plugin component of Mayastor's CSI driver. As such, their function is to orchestrate the mounting of Mayastor-provisioned volumes on worker nodes on which application pods consuming those volumes are scheduled. By default, a csi-node pod is scheduled on every node in the target cluster, as determined by a DaemonSet resource of the same name. Each of these pods encapsulates two containers, csi-node, and csi-driver-registrar. The node plugin does not need to run on every worker node within a cluster and this behavior can be modified, if desired, through the application of appropriate node labeling and the addition of a corresponding nodeSelector entry within the pod spec of the csi-node DaemonSet. However, it should be noted that if a node does not host a plugin pod, then it will not be possible to schedule an application pod on it, which is configured to mount Mayastor volumes.

etcd

etcd is a distributed reliable key-value store for the critical data of a distributed system. Mayastor uses etcd as a reliable persistent store for its configuration and state data.

Supportability:

The supportability tool is used to create support bundles (archive files) by interacting with multiple services present in the cluster where Mayastor is installed. These bundles contain information about Mayastor resources like volumes, pools and nodes, and can be used for debugging. The tool can collect the following information:
  • Topological information of Mayastor's resource(s) by interacting with the REST service
  • Historical logs by interacting with Loki. If Loki is unavailable, it interacts with the kube-apiserver to fetch logs.
  • Mayastor-specific Kubernetes resources by interacting with the kube-apiserver
  • Mayastor-specific information from etcd (internal) by interacting with the etcd server.

Loki:

Loki aggregates and centrally stores logs from all Mayastor containers which are deployed in the cluster.

Promtail:

Promtail is a log collector built specifically for Loki. It uses the configuration file for target discovery and includes analogous features for labeling, transforming, and filtering logs from containers before ingesting them to Loki.