This documentation aims to provide guidance on how to configure access to docker.mariadb.com in your MariaDB Enterprise Operator resources.

Customer credentials

MariaDB Corporation requires customers to authenticate when logging in to the . A Customer Download Token must be provided as the password. Customer Download Tokens are available through the MariaDB Customer Portal. To retrieve the customer download token for your account:

• Navigate to the Customer Download Token at the MariaDB Customer Portal.

• Log in using your MariaDB ID.

• Copy the Customer Download Token to use as the password when logging in to the MariaDB Enterprise Docker Registry.

Then, configure a Kubernetes kubernetes.io/dockerconfigjson Secret to authenticate:

kubectl create secret docker-registry mariadb-enterprise \
   --docker-server=docker.mariadb.com \
   --docker-username=<email> \
   --docker-password=<customer-download-token>

Openshift

If you are running in Openshift, it is recommended to use the global pull secret to configure customer credentials. The global pull secret is automatically used by all Pods in the cluster, without having to specify imagePullSecrets explicitly.

To configure the global pull secret, you can use the following commands:

• Extract your Openshift global pull secret:

oc extract secret/pull-secret -n openshift-config --confirm

• Login in the MariaDB registry providing the customer download token as password:

oc registry login \
  --registry="docker.mariadb.com" \
  --auth-basic="<email>:<customer-download-token>" \
  --to=.dockerconfigjson

• Update the global pull secret:

oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson

Alternatively, you can also create a dedicated Secret for authenticating:

oc create secret docker-registry mariadb-enterprise \
   --docker-server=docker.mariadb.com \
   --docker-username=<email> \
   --docker-password=<customer-download-token>

MariaDB

In order to configure access to docker.mariadb.com in your MariaDB resources, you can use the imagePullSecrets field to specify your customer credentials:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
  ...
  image: docker.mariadb.com/enterprise-server:11.4.4-2
  imagePullPolicy: IfNotPresent
  imagePullSecrets:
    - name: mariadb-enterprise

As a result, the Pods created as part of the reconciliation process will have the imagePullSecrets.

MaxScale

Similarly to MariaDB, you are able to configure access to docker.mariadb.com in your MaxScale resources:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MaxScale
metadata:
  name: maxscale
spec:
  ...
  image: docker.mariadb.com/maxscale-enterprise:25.01.1
  imagePullPolicy: IfNotPresent
  imagePullSecrets:
    - name: mariadb-enterprise

Backup, Restore and SqlJob

The batch Job resources will inherit the imagePullSecrets from the referred MariaDB, as they also make use of its image. However, you are also able to provide dedicated imagePullSecrets for these resources:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
  ...
  image: docker.mariadb.com/enterprise-server:11.4.4-2
  imagePullPolicy: IfNotPresent
  imagePullSecrets:
    - name: mariadb-enterprise

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Backup
metadata:
  name: backup
spec:
  ...
  mariaDbRef:
    name: mariadb
  imagePullSecrets:
    - name: backup-registry

When the resources from the previous examples are created, a Job with both mariadb-enterprise and backup-registry imagePullSecrets will be reconciled.

Certified images

All the Docker images used by this operator are based on Red Hat UBI and have been certified by Red Hat. The advantages of using UBI based images are:

• Immutability: UBI images are built to be secure and stable, reducing the risk of unintended changes or vulnerabilities due to mutable base layers.

• Small size: The UBI minimal and micro variants used by this operator are designed to be lightweight, containing only the essential packages. This can lead to smaller container image sizes, resulting in faster build times, reduced storage requirements, and quicker image pulls.

• Security and compliance: Regular CVE scanning and vulnerability patching help maintain compliance with industry standards and security best practices.

• Enterprise-grade support: UBI images are maintained and supported by Red Hat, ensuring timely security updates and long-term stability.

List of compatible images

MariaDB Enterprise Operator is compatible with the following Docker images:

Refer to the registry documentation to access docker.mariadb.com with your customer credentials.

Suspended state

When a resource is suspended, all operations performed by the operator are disabled, including but not limited to:

• Provisioning

• Upgrades

• Volume resize

• Galera cluster recovery

More specifically, the reconciliation loop of the operator is omitted, anything part of it will not happen while the resource is suspended. This could be useful in maintenance scenarios, where manual operations need to be performed, as it helps prevent conflicts with the operator.

Suspend a resource

Currently, only MariaDB and MaxScale resources support suspension. You can enable it by setting suspend=true:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  suspend: true

This results in the reconciliation loop being disabled and the status being marked as Suspended:

kubectl get mariadbs
NAME             READY   STATUS      PRIMARY           UPDATES                   AGE
mariadb-galera   True    Suspended   mariadb-galera-0  ReplicasFirstPrimaryLast  12m

To re-enable it, simply remove the suspend setting or set it to suspend=false.

The contains a number of sample manifests that aim to show the operator functionality in a practical way. Follow these instructions for getting started:

• Download the :

• Install the configuration shared by all the examples:

• Start deploying examples:

Some examples rely on external dependencies for specific tasks, make sure to install them when it applies:

• for metrics

• for TLS certificates

• for S3 object storage

It is recommended to complement the examples with the documentation to understand the full range of configuration options available.

If you are looking for production-grade examples, you can check the mariadb_galera_production.yaml and maxscale_galera_production.yaml examples.

MariaDB Enterprise Operator allows you to configure standalone MariaDB Enterprise Server instances. To achieve this, you can either omit the replicas field or set it to 1:

Whilst this can be useful for development and testing, it is not recommended for production use because of the following reasons:

• Single point of failure

• Upgrades require downtime

• Only vertical scaling is possible

For achieving high availability, we recommend deploying a Galera cluster. Refer to the and sections for more information.

This guide illustrates, step by step, how to migrate to 25.8.0 from previous versions.

• Uninstall you current mariadb-enterprise-operator for preventing conflicts:

helm uninstall mariadb-enterprise-operator

Alternatively, you may only downscale and delete the webhook configurations:

kubectl scale deployment mariadb-enterprise-operator --replicas=0
kubectl scale deployment mariadb-enterprise-operator-webhook --replicas=0
kubectl delete validatingwebhookconfiguration mariadb-enterprise-operator-webhook
kubectl delete mutatingwebhookconfiguration mariadb-enterprise-operator-webhook

• Upgrade mariadb-enterprise-operator-crds to 25.8.0:

helm repo update mariadb-enterprise-operator
helm upgrade --install mariadb-enterprise-operator-crds  mariadb-enterprise-operator/mariadb-enterprise-operator-crds --version 25.8.0

• The Galera data-plane must be updated to the 25.8.0 version.

If you want the operator to automatically update the data-plane (i.e. init and agent containers), you can set updateStrategy.autoUpdateDataPlane=true in your MariaDB resources:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  updateStrategy:
+   autoUpdateDataPlane: true

Alternatively, you can also do this manually:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
  galera:
    agent:
-      image: docker.mariadb.com/mariadb-enterprise-operator:1.0.0
+      image: docker.mariadb.com/mariadb-enterprise-operator:25.8.0
    initContainer:
-      image: docker.mariadb.com/mariadb-enterprise-operator:1.0.0
+      image: docker.mariadb.com/mariadb-enterprise-operator:25.8.0

• Upgrade mariadb-enterprise-operator to 25.8.0:

helm repo update mariadb-enterprise-operator
helm upgrade --install mariadb-enterprise-operator mariadb-enterprise-operator/mariadb-enterprise-operator --version 25.8.0 

• If you previously decided to downscale the operator, make sure you upscale it back:

kubectl scale deployment mariadb-enterprise-operator --replicas=1
kubectl scale deployment mariadb-enterprise-operator-webhook --replicas=1

• If you previously set updateStratety.autoUpdateDataPlane=true, you may consider reverting the changes once the upgrades have finished:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  updateStrategy:
+   autoUpdateDataPlane: false
-   autoUpdateDataPlane: true

In this guide, we will be migrating an external MariaDB into a new MariaDB instance running in Kubernetes and managed by MariaDB Enterprise Operator. We will be using for achieving this migration.

1. Take a logical backup of your external MariaDB using one of the commands below:

If you are currently using or migrating to a Galera instance, use the following command instead:

2. Ensure that your backup file matches the following format: backup.2024-08-26T12:24:34Z.sql. If the file name does not follow this format, it will be ignored by the operator.

3. Upload the backup file to one of the supported . We recommend using S3.

4. Create your MariaDB resource declaring that you want to and providing a that matches the backup:

5. If you are using Galera in your new instance, migrate your previous users and grants to use the User and Grant CRs. Refer to the for further detail.

MariaDB Enterprise Operator provides a seamless way to run and operate containerized versions of MariaDB Enterprise Server and MaxScale on Kubernetes, allowing you to leverage Kubernetes orchestration and automation capabilities. This document outlines the features and advantages of using Kubernetes and the MariaDB Enterprise Operator to streamline the deployment and management of MariaDB and MaxScale instances.

What is Kubernetes?

Kubernetes is more than just a container orchestrator; it is a comprehensive platform that provides APIs for managing both applications and the underlying infrastructure. It automates key aspects of container management, including deployment, scaling, and monitoring, while also handling essential infrastructure needs such as networking and storage. By unifying the management of applications and infrastructure, Kubernetes simplifies operations and improves efficiency in cloud-native environments.

Why Kubernetes?

Kubernetes brings several key benefits to the table when managing applications in a containerized environment:

• Standardization: Kubernetes relies on standard APIs for managing applications and infrastructure, making it easier to ensure uniformity across various environments. It acts as a common denominator across cloud providers and on-premises.

• Automation: Kubernetes APIs encapsulate operational best practises, minimizing the need for manual intervention and improving the efficiency of operations.

• Cost Effectiveness: Having an standarized way to manage infrastructure across cloud providers and automation to streamline operations, Kubernetes helps reducing the infrastructure and operational costs.

What is a Kubernetes Operator?

Kubernetes has been designed with flexibility in mind, allowing developers to extend its capabilities through custom resources and operators.

In particular, MariaDB Enterprise Operator, watches the desired state defined by users via MariaDB and MaxScale resources, and takes actions to ensure that the actual state of the system matches the desired state. This includes managing compute, storage and network resources, as well as the full lifecycle of the MariaDB and MaxScale instances. Whenever the desired state changes or the underlying infrastructure is modified, the Operator takes the necessary actions to reconcile the actual state with the desired state.

Operational expertise is baked into the MariaDB and MaxScale APIs and seamlessly managed by the Operator. This includes automated backups, restores, upgrades, monitoring, and other critical lifecycle tasks, ensuring reliability in Day 2 operations.

MariaDB Enterprise Operator Features

• Provision and Configure MariaDB and MaxScale Declaratively: Define MariaDB Enterprise Server and MaxScale clusters in YAML manifests and deploy them with ease in Kubernetes.

• High Availability with Galera: Ensure availability with MariaDB Enterprise Cluster, providing synchronous multi-master replication.

• Query and Connection-Based Routing with MaxScale: MaxScale provides query routing and connection load balancing for improved application performance.

• Cluster-Aware Rolling Updates: Perform rolling updates on MariaDB and MaxScale clusters, ensuring zero-downtime upgrades with no disruptions to your applications.

• Flexible Storage Configuration and Volume Expansion: Easily configure storage for MariaDB instances, including the ability to expand volumes as needed.

• Physical Backups based on and . By leveraging the feature, backups are taken without long read locks or service interruptions.

• Logical Backups based on .

• Backup Management: Take, restore, and schedule backups with multiple storage types supported: S3, PVCs, Kubernetes volumes and VolumeSnapshots..

• Policy-Driven Backup Retention: Implement backup retention policies with bzip2 and gzip compression.

• Target Recovery Time: Restore your database to the closest available backup based on a specified recovery time.

• Bootstrap New Instances: Initialize new MariaDB instances from backups, S3, PVCs or VolumeSnapshots to quickly spin up new clusters.

• TLS Certificate Management: Issue, configure, and rotate TLS certificates and Certificate Authorities (CAs) for secure connections.

• Advanced TLS Support: customize certificate lifetime, private key algorithm and TLS version.

• Native Integration with cert-manager: Leverage , the de-facto standard for managing certificates in Kubernetes, to enable issuance with private CAs, public CAs and HashiCorp Vault.

• Prometheus Metrics: Expose metrics using the MariaDB and MaxScale Prometheus exporters.

• Native Integration with prometheus-operator: Leverage to scrape metrics from MariaDB and MaxScale instances.

• Declarative User and Database Management: Manage users, grants, and logical databases in a declarative manner using Kubernetes resources.

• Secure, immutable and lightweight images based on Red Hat UBI, available for multiple architectires (amd64, arm64 and ppc64le).

• by Red Hat.

In this guide, we will be migrating from the to the without downtime. This guide assumes:

• version of the MariaDB Community Operator is installed in the cluster.

• MariaDB community resources will be migrated to its counterpart MariaDB enterprise resource. In this case, we will be using 11.4.4 version, which is supported in both community and enterprise versions. Check the supported and migrate to a counterpart community version first if needed.

• MaxScale resources cannot be migrated in a similar way, they need to be recreated. To avoid downtime, temporarily point your applications to MariaDB directly during the migration.

1. Install the Enterprise CRDs as described in the .

2. Get the and grant execute permissions:

3. Migrate MariaDB resources using the migration script. Make sure you set <mariadb-name> with the name of the MariaDB resource to be migrated and <operator-version> with the version of the Enterprise operator you will be installing:

4. Update the apiVersion of the rest of CRs to enterprise.mariadb.com/v1alpha1.

5. Uninstall the Community operator:

6. If your MariaDB Community had Galera enabled, delete the <mariadb-name> Role, as it will be specyfing the Community CRDs:

7. Install the Enterprise operator as described in the . This will trigger a rolling upgrade, make sure it finishes successfully before proceeding with the next step.

8. Delete the finalizers and uninstall the Community CRDs:

9. Run mariadb-upgrade in all Pods. Make sure you set <mariadb-name> with the name of the MariaDB resource:

10. Restart the Enterprise operator:

By leveraging the automation provided by MariaDB Enterprise Operator, you can declaratively manage large fleets of databases using CRs. This also covers day two operations, such as upgrades, which can be risky when rolling out updates to thousands of instances simultaneously.

To mitigate this, and to give you full control on the upgrade process, you are able to choose between multiple update strategies described in the following sections.

Update strategies

In order to provide you with flexibility for updating MariaDB reliably, this operator supports multiple update strategies:

• ReplicasFirstPrimaryLast: Roll out replica Pods one by one, wait for each of them to become ready, and then proceed with the primary Pod.

• RollingUpdate: Utilize the rolling update strategy from Kubernetes.

• OnDelete: Updates are performed manually by deleting Pods.

• Never: Pause updates.

Configuration

The update strategy can be configured in the updateStrategy field of the MariaDB resource:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
  updateStrategy:
    type: ReplicasFirstPrimaryLast

It defaults to ReplicasFirstPrimaryLast if not provided.

Trigger updates

Updates are not limited to updating the image field in the MariaDB resource, an update will be triggered whenever any field of the Pod template is changed. This translates into making changes to MariaDB fields that map directly or indirectly to the Pod template, for instance, the CPU and memory resources:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
  ...
- image: docker.mariadb.com/enterprise-server:10.6.18-14.2
+ image: docker.mariadb.com/enterprise-server:10.6.19-15.1
  resources:
    requests:
      cpu: 200m
      memory: 128Mi
    limits:
-     memory: 1Gi
+     memory: 2Gi

Once the update is triggered, the operator manages it differently based on the selected update strategy.

ReplicasFirstPrimaryLast

This role-aware update strategy consists in rolling out the replica Pods one by one first, waiting for each of them become ready (i.e. readiness probe passed), and then proceed with the primary Pod. This is the default update strategy, as it can potentially meet various reliability requirements and minimize the risks associated with updates:

• Write operations won't be affected until all the replica Pods have been rolled out. If something goes wrong in the update, such as an update to an incompatible MariaDB version, this is detected early when the replicas are being rolled out and the update operation will be paused at that point.

• Read operations impact is minimized by only rolling one replica Pod at a time.

• Waiting for every Pod to be synced minimizes the impact in the clustering protocols and the network.

RollingUpdate

This strategy leverages the rolling update strategy from the StatefulSet resource, which, unlike ReplicasFirstPrimaryLast, does not take into account the role of the Pods(primary or replica). Instead, it rolls out the Pods one by one, from the highest to the lowest StatefulSet index.

You are able to pass extra parameters to this strategy via the rollingUpdate object:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
  ...
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1

OnDelete

This strategy aims to provide a method to update MariaDB resources manually by allowing the user to restart the Pods individually. This way, the user has full control over the update process and can decide which Pods are rolled out at any given time.

Whenever an update is triggered, the MariaDB will be marked as pending to update:

kubectl get mariadbs
NAME             READY   STATUS           PRIMARY            UPDATES    AGE
mariadb-galera   True    Pending update   mariadb-galera-0   OnDelete   5m17s

From this point, you are able to delete the Pods to trigger the update, which will result the MariaDB marked as updating:

kubectl get mariadbs
NAME             READY   STATUS         PRIMARY            UPDATES    AGE
mariadb-galera   True    Updating       mariadb-galera-0   OnDelete   9m50s

Once all the Pods have been rolled out, the MariaDB resource will be back to a ready state:

NAME             READY   STATUS         PRIMARY            UPDATES    AGE
mariadb-galera   True    Running        mariadb-galera-0   OnDelete   12m

Never

The operator will not perform updates on the StatefulSet whenever this update strategy is configured. This could be useful in multiple scenarios:

• Progressive fleet upgrades: If you're managing large fleets of databases, you likely prefer to roll out updates progressively rather than simultaneously across all instances.

• Operator upgrades: When upgrading the operator, changes to the StatefulSet or the Pod template may occur from one version to another, which could trigger a rolling update of your MariaDB instances.

Data-plane updates

Galera relies on data-plane containers that run alongside MariaDB to implement provisioning and high availability operations on the cluster. These containers use the mariadb-enterprise-operator image, which can be automatically updated by the operator based on its image version:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  updateStrategy:
    autoUpdateDataPlane: true

By default, updateStrategy.autoUpdateDataPlane is false, which means that no automatic upgrades will be performed, but you can opt-in/opt-out from this feature at any point in time by updating this field. For instance, you may want to selectively enable updateStrategy.autoUpdateDataPlane in a subset of your MariaDB instances after the operator has been upgraded to a newer version, and then disable it once the upgrades are completed.

It is important to note that this feature is fully compatible with the Never strategy: no upgrades will happen when updateStrategy.autoUpdateDataPlane=true and updateStrategy.type=Never.

In this guide, we will be migrating existing MariaDB Galera and MaxScale instances to TLS without downtime.

1. Ensure that MariaDB has TLS enabled and not enforced. Set the following options if needed:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  tls:
+   enabled: true
+   required: false
+   galeraSSTEnabled: false
+   galeraServerSSLMode: PROVIDER
+   galeraClientSSLMode: DISABLED

By setting these options, the operator will issue and configure certificates for MariaDB, but TLS will not be enforced in the connections i.e. both TLS and non-TLS connections will be accepted. TLS enforcement will be optionally configured at the end of the migration process.

This will trigger a rolling upgrade, make sure it finishes successfully before proceeding with the next step. Refer to the updates documentation for further information about update strategies.

2. If you are currently using MaxScale, it is important to note that, unlike MariaDB, it does not support TLS and non-TLS connections simultaneously (see limitations). For this reason, you must temporarily point your applications to MariaDB during the migration process. You can achieve this by configuring your application to use the . At the end of the MariaDB migration process, the MaxScale instance will need to be recreated in order to use TLS, and then you will be able to point your application back to MaxScale. Ensure that all applications are pointing to MariaDB before moving on to the next step.

3. MariaDB is now accepting TLS connections. The next step is migrating your applications to use TLS by pointing them to MariaDB securely. Ensure that all applications are connecting to MariaDB via TLS before proceeding to the next step.

4. If you are currently using MaxScale, and you are planning to connect via TLS through it, you should now delete your MaxScale instance. If needed, keep a copy of the MaxScale manifest, as we will need to recreate it with TLS enabled in further steps:

kubectl get mxs maxscale-galera -o yaml > maxscale-galera.yaml
kubectl delete mxs maxscale-galera

It is very important that you wait until your old MaxScale instance is fully terminated to make sure that the old configuration is cleaned up by the operator.

5. For enhanced security, it is recommended to enforce TLS in all MariaDB connections by setting the following options. This will trigger a rolling upgrade, make sure it finishes successfully before proceeding with the next step:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  tls:
+   required: true
+   galeraServerSSLMode: SERVER_X509

6. For improved security, you can optionally configure TLS for Galera SSTs by following the steps below:

• Get the migration script and grant execute permissions:

curl -sLO https://operator.mariadb.com/scripts/migrate_galera_ssl.sh
chmod +x migrate_galera_ssl.sh

• Run the migration script. Make sure you set <mariadb-name> with the name of the MariaDB resource:

./migrate_galera_ssl.sh <mariadb-name>

• Set the following option to enable TLS for Galera SSTs:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  tls:
+   galeraSSTEnabled: true
+   galeraClientSSLMode: VERIFY_IDENTITY

This will trigger a rolling upgrade, make sure it finishes successfully before proceeding with the next step

7. As mentioned in step 4, recreate your MaxScale instance with tls.enabled=true if needed:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MaxScale
metadata:
  name: maxscale-galera
spec:
+ tls:
+   enabled: true

8. MaxScale is now accepting TLS connections. Next, you need to migrate your applications to use TLS by pointing them back to MaxScale securely. You have done this previously for MariaDB, you just need to update your application configuration to use the MaxScale Service and its CA bundle.

This operator gives you flexibility to define the storage that will back the /var/lib/mysql data directory mounted by MariaDB.

Configuration

The simplest way to configure storage for your MariaDB is:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  ...
  storage:
    size: 1Gi

This will make use of the default StorageClass available in your cluster, but you can also provide a different one:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  ...
  storage:
    size: 1Gi
    storageClassName: gp3

Under the scenes, the operator is configuring the StatefulSet's volumeClaimTemplate property, which you are also able to provide yourself:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  ...
  storage:
    size: 1Gi
    storageClassName: gp3
    volumeClaimTemplate:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
      storageClassName: gp3

Volume resize

It is possible to resize your storage after having provisioned a MariaDB. We need to distinguish between:

• PVCs already in use.

• StatefulSet storage size, which will be used when provisioning new replicas.

It is important to note that, for the first case, your StorageClass must support volume expansion by declaring the allowVolumeExpansion = true. In such case, it will be safe to expand the storage by increasing the size and setting resizeInUseVolumes = true:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  ...
  storage:
    size: 2Gi
    resizeInUseVolumes: true
    waitForVolumeResize: true

Depending on your storage provider, this operation might take a while, and you can decide to wait for this operation before the MariaDB becomes ready by setting waitForVolumeResize = true. Operations such as Galera cluster recovery and will not be performed if the MariaDB resource is not ready.

Ephemeral storage

Provisioning standalone MariaDB instances with ephemeral storage can be done by setting ephemeral = true:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  ...
  storage:
    ephemeral: true

This may be useful for multiple use cases, like provisioning ephemeral MariaDBs for the integration tests of your CI.

This documentation shows how to configure metadata in the MariaDB Enterprise Operator CRs.

Children object metadata

MariaDB and MaxScale resources allow you to propagate metadata to all the children objects by specifying the inheritMetadata field:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  inheritMetadata:
    labels:
      database.myorg.io: mariadb
    annotations:
      database.myorg.io: mariadb

This means that all the reconciled objects will inherit these labels and annotations. For instance, see the Services and Pods:

apiVersion: v1
kind: Service
metadata:
  annotations:
    database.myorg.io: mariadb
  labels:
    database.myorg.io: mariadb
  name: mariadb-galera-primary

apiVersion: v1
kind: Pod
metadata:
  annotations:
    database.myorg.io: mariadb
  labels:
    database.myorg.io: mariadb
  name: mariadb-galera-0

Pod metadata

You have the ability to provide dedicated metadata for Pods by specifying the podMetadata field in any CR that reconciles a Pod, for instance: MariaDB, MaxScale, Backup, Restore and SqlJobs:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Backup
metadata:
  name: backup
spec:
  inheritMetadata:
    labels:
      sidecar.istio.io/inject: "true"
    annotations:
      database.myorg.io: mariadb
  podMetadata:
    labels:
      sidecar.istio.io/inject: "false"

It is important to note that the podMetadata field supersedes the inheritMetadata field, therefore the labels and annotations provided in the former will override the ones in the latter.

Service metadata

Provision dedicated metadata for Services in the MariaDB resources can be done via the service, primaryService and secondaryService fields:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  service:
    type: LoadBalancer
    metadata:
      annotations:
        metallb.universe.tf/loadBalancerIPs: 172.18.0.150

  primaryService:
    type: LoadBalancer
    metadata:
      annotations:
        metallb.universe.tf/loadBalancerIPs: 172.18.0.160

  secondaryService:
    type: LoadBalancer
    metadata:
      annotations:
        metallb.universe.tf/loadBalancerIPs: 172.18.0.161

In the case of MaxScale, you can also do this via the kubernetesService field.

Refer to the to know more about the Service fields and MaxScale.

PVC metadata

Both MariaDB and MaxScale allow you to define a volumeClaimTemplate to be used by the underlying StatefulSet. You may also define metadata for it:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  storage:
    size: 1Gi
    volumeClaimTemplate:
      metadata:
        annotations:
          database.myorg.io: mariadb
        labels:
          database.myorg.io: mariadb
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi

Use cases

Being able to provide metadata allows you to integrate with other CNCF landscape projects:

Metallb

If you run on bare metal and you use Metallb for managing the LoadBalancer objects, you can declare its IPs via annotations:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  service:
    type: LoadBalancer
    metadata:
      annotations:
        metallb.universe.tf/loadBalancerIPs: 172.18.0.150

Istio

Istio injects the data-plane container to all Pods, but you might want to opt-out of this feature in some cases:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Backup
metadata:
  name: backup
spec:
  podMetadata:
    labels:
      sidecar.istio.io/inject: "false"

For instance, you probably don't want to inject the Istio sidecar to Backup Pods, as it will prevent the Jobs from finishing and therefore your backup process will hang.

This documentation provides guidance on installing the MariaDB Enterprise Operator operator in OpenShift. This operator has been certified by Red Hat and it is available in the OpenShift console.

Operators are deployed into OpenShift with the Operator Lifecycle Manager (OLM), which facilitates the installation, updates, and overall management of their lifecycle.

Prerequisites

Configure your customer credentials as described in the documentation to be able to pull images.

The recommended way to configure credentials is to use the global pull secret provided by OpenShift, as described in this section. Alternatively, the operator bundle has a mariadb-enterprise imagePullSecret configured by default. This means that you can configure a Secret named mariadb-enterprise in same namespace where the operator will be installed in order to pull images from the MariaDB Enterprise registry.

PackageManifest

You can install the certified operator in OpenShift clusters that have the mariadb-enterprise-operator packagemanifest available. In order to check this, run the following command:

oc get packagemanifests -n openshift-marketplace mariadb-enterprise-operator

NAME                          CATALOG                 AGE
mariadb-enterprise-operator   Certified Operators     21h

SecurityContextConstraints

Both the operator and the operand Pods run with the restricted-v2 SecurityContextConstraint, the most restrictive SCC in OpenShift in terms of container permissions. This implies that OpenShift automatically assigns a SecurityContext for the Pods with minimum permissions, for example:

securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL
  runAsNonRoot: true
  runAsUser: 1000650000

You can read more about SecurityContextConstraints in the OpenShift documentation.

Installation in all namespaces

To install the operator watching resources on all namespaces, you need to create a Subscription object for mariadb-enterprise-operator using the stable channel in the openshift-operators namespace:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: mariadb-enterprise-operator
  namespace: openshift-operators
spec:
  channel: stable
  installPlanApproval: Automatic
  name: mariadb-enterprise-operator
  source: certified-operators
  sourceNamespace: openshift-marketplace
  startingCSV: mariadb-enterprise-operator.v1.0.0

This will use the global-operators OperatorGroup that is created by default in the openshift-operators namespace. This OperatorGroup will watch all namespaces in the cluster, and the operator will be able to manage resources across all namespaces.

You can read more about OperatorGroups in the OpenShift documentation.

Installation in specific namespaces

In order to define which namespaces the operator will be watching, you need to create an OperatorGroup in the namespace where the operator will be installed:

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: mariadb-enterprise-operator
  namespace: my-namespace
spec:
  targetNamespaces:
  - my-namespace
  - my-other-namespace
  - my-other-other-namespace
  upgradeStrategy: Default

This OperatorGroup will watch the namespaces defined in the targetNamespaces field. The operator will be able to manage resources only in these namespaces.

Then, the operator can be installed by creating a Subscription object in the same namespace as the OperatorGroup:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: mariadb-enterprise-operator
  namespace: my-namespace
spec:
  channel: stable
  installPlanApproval: Automatic
  name: mariadb-enterprise-operator
  source: certified-operators
  sourceNamespace: openshift-marketplace
  startingCSV: mariadb-enterprise-operator.v1.0.0

Release channels

Currently, the only release channel available to install the operator is stable, which contains supported releases of the operator. This is controlled by the channel field in the Subscription object.

Updates

Updates are fully managed by OLM and controlled by the installPlanApproval field in the Subscription object. The default value is Automatic, which means that OLM will automatically update the operator to the latest version available in the channel. If you want to control the updates, you can set this field to Manual, and OLM will only update the operator when you approve the update.

Uninstalling

The first step for uninstalling the operator is to delete the Subscription object. This will not remove the operator, but it will stop OLM from managing the operator:

oc delete subscription mariadb-enterprise-operator

After that, you can uninstall the ClusterServiceVersion (CSV) object that was created by OLM. This will remove the operator from the cluster:

oc delete clusterserviceversion mariadb-enterprise-operator.v1.0.0

OpenShift console

As an alternative to create Subscription objects via the command line, you can install operators by using the OpenShift console. Go to the Operators > OperatorHub section and search by mariadb enterprise:

Select MariaDB Enterprise Operator, click on install, and you will be able to create a Subscription object via the UI.

Once deployed, the operator comes with example resources that can be deployed from the console directly. For instance, to create a MariaDB:

As you can see in the previous screenshot, the form view that the OpenShift console offers is limited, we recommend using the YAML view:

This guide aims to provide a quick way to get started with the MariaDB Enterprise Operator for Kubernetes. It will walk you through the process of deploying a MariaDB Enterprise Cluster and MaxScale via the MariaDB and MaxScale CRs (Custom Resources) respectively.

Before you begin, ensure you meet the following prerequisites:

• Configure your customer access for docker.mariadb.com

• Install the MariaDB Enterprise Operator

The first step will be configuring a Secret with the credentials used by the MariaDB CR:

apiVersion: v1
kind: Secret
metadata:
  name: mariadb
stringData:
  password: MariaDB11!

kubectl apply -f secret.yaml

Next, we will deploy a MariaDB Enterprise Cluster (Galera) using the following CR:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  rootPasswordSecretKeyRef:
    name: mariadb
    key: password
  imagePullSecrets:
  -  name: mariadb-enterprise
  maxScaleRef:
    name: maxscale-galera
  username: mariadb
  passwordSecretKeyRef:
    name: mariadb
    key: password
  database: mariadb
  storage:
    size: 1Gi
  replicas: 3
  galera:
    enabled: true

kubectl apply -f mariadb-galera.yaml

Let's break it down:

• rootPasswordSecretKeyRef: A reference to a Secret containing the root password.

• imagePullSecrets: The name of the Secret containing the customer credentials to pull the MariaDB Enterprise Server image.

• maxScaleRef: The name of the MaxScale CR that we will be creating right after.

• username, passwordSecretKeyRef and database: The initial user and database to create.

• storage: The size of the volume that will back the data directory.

• replicas: The number of MariaDB Enterprise Server instances to deploy.

• galera: Configuration for the Galera clustering.

After applying the CR, we can observe the MariaDB Pods being created:

❯ kubectl get pods
NAME                                                           READY   STATUS    RESTARTS      AGE
mariadb-galera-0                                               2/2     Running   0             101s
mariadb-galera-1                                               2/2     Running   0             101s
mariadb-galera-2                                               2/2     Running   0             101s

Now, let's deploy a MaxScale CR:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MaxScale
metadata:
  name: maxscale-galera
spec:
  imagePullSecrets:
   -  name: mariadb-enterprise
  mariaDbRef:
    name: mariadb-galera
  replicas: 2

kubectl apply -f maxscale-galera.yaml

Again, let's break it down:

• imagePullSecrets: The name of the Secret containing the customer credentials to pull the MaxScale image.

• mariaDbRef: A reference to the MariaDB CR that we want to connect to.

• replicas: The number of MaxScale instances to deploy.

After applying the CR, we can observe the MaxScale Pods being created, and that both the MariaDB and MaxScale CRs will become ready eventually:

❯ kubectl get pods
mariadb-galera-0                                               2/2     Running   0             10m
mariadb-galera-1                                               2/2     Running   0             10m
mariadb-galera-2                                               2/2     Running   0             10m
maxscale-galera-0                                              1/1     Running   0             81s
maxscale-galera-1                                              1/1     Running   0             81s

❯ kubectl get maxscale
NAME              READY   STATUS    PRIMARY            AGE
maxscale-galera   True    Running   mariadb-galera-0   65s

❯ kubectl get mariadb
NAME             READY   STATUS    PRIMARY            UPDATES                    AGE
mariadb-galera   True    Running   mariadb-galera-0   ReplicasFirstPrimaryLast   10m

To conclude, let's connect to the MariaDB Enterprise Cluster through MaxScale using the initial user and database we initially defined in the MariaDB CR:

❯ kubectl run mariadb-connect --rm -it --image=docker.mariadb.com/enterprise-server:11.4.4-2 -- bash -c "mariadb -u mariadb -p'MariaDB11!' --ssl=false -h maxscale-galera"
If you don't see a command prompt, try pressing enter.
MariaDB [(none)]> SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mariadb            |
+--------------------+
2 rows in set (0.001 sec)

Refer to the documentation, the API reference and the examples catalog for further detail.

This section provide guidance on how to run MariaDB and MaxScale in high availability mode. If you are looking to run the operator in HA as well, please refer to the .

Our recommended HA setup for production is:

• with at least 3 nodes. Always an odd number of nodes.

• with at least 2 nodes to load balance requests to the cluster.

• Use to avoid noisy neighbours.

• Define .

Refer to the following sections for further detail.

Kubernetes Services

In order to address nodes, MariaDB Enterprise Operator provides you with the following Kubernetes Services:

• <mariadb-name>: To be used for read requests. It will point to all nodes.

• <mariadb-name>-primary: To be used for write requests. It will point to a single node, the primary.

• <mariadb-name>-secondary: To be used for read requests. It will point to all nodes, except the primary.

Whenever the primary changes, either by the user or by the operator, both the <mariadb-name>-primary and <mariadb-name>-secondary Services will be automatically updated by the operator to address the right nodes.

The primary may be manually changed by the user at any point by updating the spec.galera.primary.podIndex field. Alternatively, automatic primary failover can be enabled by setting spec.galera.primary.automaticFailover, which will make the operator to switch primary whenever the primary Pod goes down.

MaxScale

While Kubernetes Services can be utilized to dynamically address primary and secondary instances, the most robust high availability configuration we recommend relies on MaxScale. Please refer to for further detail.

Pod Anti-Affinity

To achieve real high availability, we need to run each MariaDB Pod in different Kubernetes Nodes. This practice, known as anti-affinity, helps reducing the blast radius of Nodes being unavailable.

By default, anti-affinity is disabled, which means that multiple Pods may be scheduled in the same Node, something not desired in HA scenarios.

You can selectively enable anti-affinity in all the different Pods managed by the MariaDB resource:

Anti-affinity may also be enabled in the resources that have a reference to MariaDB, resulting in their Pods being scheduled in Nodes where MariaDB is not running. For instance, the Backup and Restore processes can run in different Nodes:

In the case of MaxScale, the Pods will also be placed in Nodes isolated in terms of compute, ensuring isolation not only among themselves but also from the MariaDB Pods. For example, if you run a MariaDB and MaxScale with 3 replicas each, you will need 6 Nodes in total:

Default anti-affinity rules generated by the operator might not satisfy your needs, but you can always define your own rules. For example, if you want the MaxScale Pods to be in different Nodes, but you want them to share Nodes with MariaDB:

Dedicated Nodes

If you want to avoid noisy neighbours running in the same Kubernetes Nodes as your MariaDB, you may consider using dedicated Nodes. For achieving this, you will need:

• Taint your Nodes and add the counterpart toleration in your Pods.

• Select the Nodes where Pods will be scheduled in via a nodeSelector.

• Add podAntiAffinity to your Pods as described in the section.

The previous steps can be achieved by setting these fields in the MariaDB resource:

Pod Disruption Budgets

By defining a PodDisruptionBudget, you are telling Kubernetes how many Pods your database tolerates to be down. This quite important for planned maintenance operations such as Node upgrades.

MariaDB Enterprise Operator creates a default PodDisruptionBudget if you are running in HA, but you are able to define your own by setting:

This documentation aims to provide guidance on various configuration aspects shared across many MariaDB Enterprise Operator CRs.

my.cnf

An inline can be provisioned in the MariaDB resource via the myCnf field:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
  ...
  myCnf: |
    [mariadb]
    bind-address=*
    default_storage_engine=InnoDB
    binlog_format=row
    innodb_autoinc_lock_mode=2
    innodb_buffer_pool_size=1024M
    max_allowed_packet=256M

In this field, you may provide any or supported by MariaDB.

Under the hood, the operator automatically creates a ConfigMap with the contents of the myCnf field, which will be mounted in the MariaDB instance. Alternatively, you can manage your own configuration using a pre-existing ConfigMap by linking it via myCnfConfigMapKeyRef. It is important to note that the key in this ConfigMap i.e. the config file name, must have a .cnf extension in order to be detected by MariaDB:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
  ...
  myCnfConfigMapKeyRef:
    name: mariadb
    key: mycnf

To ensure your configuration changes take effect, the operator triggers a MariaDB update whenever the myCnf field or the ConfigMap is updated. For the operator to detect changes in a ConfigMap, it must be labeled with enterprise.mariadb.com/watch. Refer to the external resources section for further detail.

Compute resources

CPU and memory resouces can be configured via the resources field in both the MariaDB and MaxScale CRs:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
  ...
  resources:
    requests:
      cpu: 1
      memory: 4Gi
    limits:
      memory: 4Gi

In the case of MariaDB, it is recommended to set the innodb_buffer_pool_size system variable to a value that is 70-80% of the available memory. This can be done via the myCnf field:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
  ...
  myCnf: |
    [mariadb]
    innodb_buffer_pool_size=3200M

Timezones

By default, MariaDB does not load timezone data on startup for performance reasons and defaults the timezone to SYSTEM, obtaining the timezone information from the environment where it runs. See the for further information.

You can explicitly configure a timezone in your MariaDB instance by setting the timeZone field:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  timeZone: "UTC"

This setting is immutable and implies loading the timezone data on startup.

In regards to Backup and SqlJob resources, which get reconciled into CronJobs, you can also define a timeZone associated with their cron expression:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Backup
metadata:
  name: backup-scheduled
spec:
  mariaDbRef:
    name: mariadb
  schedule:
    cron: "*/1 * * * *"
    suspend: false
  timeZone: "UTC"

If timeZone is not provided, the local timezone will be used, as described in the Kubernetes docs.

Passwords

Some CRs require passwords provided as Secret references to function properly. For instance, the root password for a MariaDB resource:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  rootPasswordSecretKeyRef:
    name: mariadb
    key: root-password

By default, fields like rootPasswordSecretKeyRef are optional and defaulted by the operator, resulting in random password generation if not provided:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  rootPasswordSecretKeyRef:
    name: mariadb
    key: root-password
    generate: true

You may choose to explicitly provide a Secret reference via rootPasswordSecretKeyRef and opt-out from random password generation by either not providing the generate field or setting it to false:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  rootPasswordSecretKeyRef:
    name: mariadb
    key: root-password
    generate: false

This way, we are telling the operator that we are expecting a Secret to be available eventually, enabling the use of GitOps tools to seed the password:

• sealed-secrets: The Secret is reconciled from a SealedSecret, which is decrypted by the sealed-secrets controller.

• external-secrets: The Secret is reconciled fom an ExternalSecret, which is read by the external-secrets controller from an external secrets source (Vault, AWS Secrets Manager ...).

External resources

Many CRs have a references to external resources (i.e. ConfigMap, Secret) not managed by the operator.

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
  ...
  myCnfConfigMapKeyRef:
    name: mariadb
    key: mycnf

These external resources should be labeled with enterprise.mariadb.com/watch so the operator can watch them and perform reconciliations based on their changes. For example, see the my.cnf ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: mariadb
  labels:
    enterprise.mariadb.com/watch: ""
data:
  mycnf: |
    [mariadb]
    bind-address=*
    default_storage_engine=InnoDB
    binlog_format=row
    innodb_autoinc_lock_mode=2
    innodb_buffer_pool_size=1024M
    max_allowed_packet=256M

Probes

Kubernetes probes serve as an inversion of control mechanism, enabling the application to communicate its health status to Kubernetes. This enables Kubernetes to take appropriate actions when the application is unhealthy, such as restarting or stop sending traffic to Pods.

Fine tunning of probes for databases running in Kubernetes is critical, you may do so by tweaking the following fields:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  # Tune your liveness probe accordingly to avoid Pod restarts.
  livenessProbe:
    periodSeconds: 10
    timeoutSeconds: 5

  # Tune your readiness probe accordingly to prevent disruptions in network traffic.
  readinessProbe:
    periodSeconds: 10
    timeoutSeconds: 5

  # Tune your startup probe accordingly to ensure that the SST completes with a large amount of data.
  # failureThreshold × periodSeconds = 30 × 10 = 300s = 5m until the container gets restarted if unhealthy
  startupProbe:
    failureThreshold: 30
    periodSeconds: 10
    timeoutSeconds: 5

There isn't an universally correct default value for these thresholds, so we recommend determining your own based on factors like the compute resources, network, storage, and other aspects of the environment where your MariaDB and MaxScale instances are running.

MariaDB Operator Enterprise enables you to manage SQL resources declaratively through CRs. By SQL resources, we refer to users, grants, and databases that are typically created using SQL statements.

The key advantage of this approach is that, unlike executing SQL statements manually, which is a one-time operation, declaring a SQL resource via a CR ensures that the resource is periodically reconciled by the operator. This provides a guarantee that the resource will be recreated if it gets manually deleted. Additionally, it prevents state drifts, as the operator will regularly update the resource according to the CR specification.

User CR

By creating this resource, you are declaring an intent to create an user in the referred MariaDB instance, just like a statement would do:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: User
metadata:
  name: bob
spec:
  mariaDbRef:
    name: mariadb
  passwordSecretKeyRef:
    name: bob-password
    key: password
  maxUserConnections: 20
  host: "%"
  cleanupPolicy: Delete

In the example above, a user named bob identified by the password available in the bob-password Secret will be created in the mariadb instance.

Refer to the API reference for more detailed information about every field.

Custom name

By default, the CR name is used to create the user in the database, but you can specify a different one providing the name field under spec:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: User
metadata:
  name: user
spec:
  name: user-custom

Grant CR

By creating this resource, you are declaring an intent to grant permissions to a given user in the referred MariaDB instance, just like a statement would do.

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Grant
metadata:
  name: grant-bob
spec:
  mariaDbRef:
    name: mariadb
  privileges:
    - "SELECT"
    - "INSERT"
    - "UPDATE"
  database: "*"
  table: "*"
  username: bob
  grantOption: true
  host: "%"

You may provide any set of .

Refer to the API reference for more detailed information about every field.

Database CR

By creating this resource, you are declaring an intent to create a logical database in the referred MariaDB instance, just like a statement would do:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Database
metadata:
  name: wordpress
spec:
  mariaDbRef:
    name: mariadb
  characterSet: utf8
  collate: utf8_general_ci

Refer to the API reference for more detailed information about every field.

Custom name

By default, the CR name is used to create the user in the database, but you can specify a different one providing the name field under spec:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Database
metadata:
  name: database
spec:
  name: database-custom

Initial User, Grant and Database

If you only need one user to interact with a single logical database, you can use of the MariaDB resource to configure it, instead of creating the User, Grant and Database resources separately:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
  username: bob
  passwordSecretKeyRef:
    name: bob-password
    key: password
  database: wordpress

Behind the scenes, the operator will be creating an User resource with ALL PRIVILEGES in the initial Database.

Authentication plugins

Passwords can be supplied using the passwordSecretKeyRef field in the User CR. This is a reference to a Secret that contains a password in plain text.

Alternatively, you can use to avoid passing passwords in plain text and provide the password in a hashed format instead. This doesn't affect the end user experience, as they will still need to provide the password in plain text to authenticate.

Password hash

Provide the password hashed using the function:

apiVersion: v1
kind: Secret
metadata:
  name: mariadb-auth
stringData:
  passwordHash: "*57685B4F0FF9D049082E296E2C39354B7A98774E"
---
apiVersion: enterprise.mariadb.com/v1alpha1
kind: User
metadata:
  name: user-password-hash
spec:
  mariaDbRef:
    name: mariadb
  passwordHashSecretKeyRef:
    name: mariadb-auth
    key: passwordHash
  host: "%"

The password hash can be obtained by executing SELECT PASSWORD('<password>'); in an existing MariaDB installation.

Password plugin

Provide the password hashed using any of the available , for example mysql_native_password:

apiVersion: v1
kind: Secret
metadata:
  name: mariadb-auth
stringData:
  passwordHash: "*57685B4F0FF9D049082E296E2C39354B7A98774E"
  nativePasswordPlugin: mysql_native_password
---
apiVersion: enterprise.mariadb.com/v1alpha1
kind: User
metadata:
  name: user-password-plugin
spec:
  mariaDbRef:
    name: mariadb
  passwordPlugin:
    pluginNameSecretKeyRef:
        name: mariadb-auth
        key: nativePasswordPlugin
    pluginArgSecretKeyRef:
        name: mariadb-auth
        key: passwordHash
  host: "%"

The plugin name should be available in a Secret referenced by pluginNameSecretKeyRef and the argument passed to it in pluginArgSecretKeyRef. The argument is the hashed password in most cases, refer to the for further detail.

Configure reconciliation

As we previously mentioned, SQL resources are periodically reconciled by the operator into SQL statements. You are able to configure the reconciliation interval using the following fields:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: User
metadata:
  name: user
spec:
  requeueInterval: 30s
  retryInterval: 5s

If the SQL statement executed by the operator is successful, it will schedule the next reconciliation cycle using the requeueInterval. If the statement encounters an error, the operator will use the retryInterval instead.

Cleanup policy

Whenever you delete a SQL resource, the operator will also delete the associated resource in the database. This is the default behaviour, that can also be achieved by setting cleanupPolicy=Delete:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: User
metadata:
  name: user
spec:
  cleanupPolicy: Delete

You can opt-out from this cleanup process using cleanupPolicy=Skip. Note that this resources will remain in the database.

What is a physical backup?

A physical backup is a snapshot of the entire data directory (/var/lib/mysql), including all data files. This type of backup captures the exact state of the database at a specific point in time, allowing for quick restoration in case of data loss or corruption.

Physical backups are the recommended method for backing up MariaDB databases, especially in production environments, as they are faster and more efficient than logical backups.

Backup strategies

Multiple strategies are available for performing physical backups, including:

• mariadb-backup: Taken using the enterprise version of mariadb-backup, specifically MariaDB Enterprise backup, which is available in the MariaDB enterprise images. The operator supports scheduling Jobs to perform backups using this utility.

• Kubernetes VolumeSnapshot: Leverage Kubernetes VolumeSnapshots to create snapshots of the persistent volumes used by the MariaDB Pods. This method relies on a compatible CSI (Container Storage Interface) driver that supports volume snapshots. See the VolumeSnapshots section for more details.

In order to use VolumeSnapshots, you will need to provide a VolumeSnapshotClass that is compatible with your storage provider. The operator will use this class to create snapshots of the persistent volumes:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: PhysicalBackup
metadata:
  name: physicalbackup
spec:
  mariaDbRef:
    name: mariadb
  storage:
    volumeSnapshot:
      volumeSnapshotClassName: csi-hostpath-snapclass

For the rest of compatible backup storage types, the mariadb-backup CLI will be used to perform the backup. For instance, to use S3 as backup storage:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: PhysicalBackup
metadata:
  name: physicalbackup
spec:
  mariaDbRef:
    name: mariadb
  storage:
    s3:
      bucket: physicalbackups
      endpoint: minio.minio.svc.cluster.local:9000
      accessKeyIdSecretKeyRef:
        name: minio
        key: access-key-id
      secretAccessKeySecretKeyRef:
        name: minio
        key: secret-access-key
      tls:
        enabled: true
        caSecretKeyRef:
          name: minio-ca
          key: ca.crt

Storage types

Multiple storage types are supported for storing physical backups, including:

• S3 compatible storage: Store backups in a S3 compatible storage, such as AWS S3 or Minio.

• Persistent Volume Claims (PVC): Use any of the StorageClasses available in your Kubernetes cluster to create a PersistentVolumeClaim (PVC) for storing backups.

• Kubernetes Volumes: Store backups in any of the in-tree storage providers supported by Kubernetes out of the box, such as NFS.

• Kubernetes VolumeSnapshots: Use Kubernetes VolumeSnapshots to create snapshots of the persistent volumes used by the MariaDB Pods. This method relies on a compatible CSI (Container Storage Interface) driver that supports volume snapshots. See the VolumeSnapshots section for more details.

Scheduling

Physical backups can be scheduled using the spec.schedule field in the PhysicalBackup resource. The schedule is defined using a Cron format and allows you to specify how often backups should be taken:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: PhysicalBackup
metadata:
  name: physicalbackup
spec:
  mariaDbRef:
    name: mariadb
  schedule:
    cron: "*/1 * * * *"
    suspend: false
    immediate: true

If you want to immediatly trigger a backup after creating the PhysicalBackup resource, you can set the immediate field to true. This will create a backup immediately, regardless of the schedule.

If you want to suspend the schedule, you can set the suspend field to true. This will prevent any new backups from being created until the PhysicalBackup is resumed.

Compression

When using physical backups based on mariadb-backup, you are able to choose the compression algorithm used to compress the backup files. The available options are:

• bzip2: Good compression ratio, but slower compression/decompression speed compared to gzip.

• gzip: Good compression/decompression speed, but worse compression ratio compared to bzip2.

• none: No compression.

To specify the compression algorithm, you can use the compression field in the PhysicalBackup resource:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: PhysicalBackup
metadata:
  name: physicalbackup
spec:
  mariaDbRef:
    name: mariadb
  compression: bzip2

compression is defaulted to none by the operator.

Retention policy

You can define a retention policy both for backups based on mariadb-backup and for VolumeSnapshots. The retention policy allows you to specify how long backups should be retained before they are automatically deleted. This can be defined via the maxRetention field in the PhysicalBackup resource:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: PhysicalBackup
metadata:
  name: physicalbackup
spec:
  mariaDbRef:
    name: mariadb
  maxRetention: 720h # 30 days

When using physical backups based on mariadb-backup, the operator will automatically delete backups files in the specified storage older than the retention period.

When using VolumeSnapshots, the operator will automatically delete the VolumeSnapshot resources older than the retention period using the Kubernetes API.

Restoration

Physical backups can only be restored in brand new MariaDB instances without any existing data. This means that you cannot restore a physical backup into an existing MariaDB instance that already has data.

To perform a restoration, you can specify a PhysicalBackup as restoration source under the spec.bootstrapFrom field in the MariaDB resource:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  bootstrapFrom:
    backupRef:
      name: physicalbackup
      kind: PhysicalBackup

This will take into account the backup strategy and storage type used in the PhysicalBackup, and it will perform the restoration accordingly.

As an alternative, you can also provide a reference to an S3 bucket that was previously used to store the physical backup files:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  bootstrapFrom:
    s3:
      bucket: physicalbackups
      prefix: mariadb
      endpoint: minio.minio.svc.cluster.local:9000
      accessKeyIdSecretKeyRef:
        name: minio
        key: access-key-id
      secretAccessKeySecretKeyRef:
        name: minio
        key: secret-access-key
      tls:
        enabled: true
        caSecretKeyRef:
          name: minio-ca
          key: ca.crt
    backupContentType: Physical

It is important to note that the backupContentType field must be set to Physical when restoring from a physical backup. This ensures that the operator uses the correct restoration method.

To restore a VolumeSnapshot, you can provide a reference to a specific VolumeSnapshot resource in the spec.bootstrapFrom field:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  bootstrapFrom:
    volumeSnapshotRef:
      name: physicalbackup-20250611163352

Target recovery time

By default, the operator will match the closest backup available to the current time. You can specify a different target recovery time by using the targetRecoveryTime field in the PhysicalBackup resource. This lets you define the exact point in time you want to restore to:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  bootstrapFrom:
    targetRecoveryTime: 2025-06-17T08:07:00Z

Timeout

By default, both backups based on mariadb-backup and VolumeSnapshots will have a timeout of 1 hour. You can change this timeout by using the timeout field in the PhysicalBackup resource:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: PhysicalBackup
metadata:
  name: physicalbackup
spec:
  mariaDbRef:
    name: mariadb
  timeout: 2h

When timed out, the operator will delete the Jobs or VolumeSnapshots resources associated wit the PhysicalBackup resource. The operator will create new Jobs or VolumeSnapshots to retry the backup operation if the PhysicalBackup resource is still scheduled.

Extra options

When taking backups based on mariadb-backup, you can specify extra options to be passed to the mariadb-backup command using the args field in the PhysicalBackup resource:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: PhysicalBackup
metadata:
  name: physicalbackup
spec:
  mariaDbRef:
    name: mariadb
  args:
    - "--verbose"

Refer to the mariadb-backup documentation for a list of available options.

S3 credentials

Credentials for accessing an S3 compatible storage can be provided via the s3 key in the storage field of the PhysicalBackup resource. The credentials can be provided as a reference to a Kubernetes Secret:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: PhysicalBackup
metadata:
  name: physicalbackup
spec:
  mariaDbRef:
    name: mariadb
  storage:
    s3:
      bucket: physicalbackups
      endpoint: minio.minio.svc.cluster.local:9000
      accessKeyIdSecretKeyRef:
        name: minio
        key: access-key-id
      secretAccessKeySecretKeyRef:
        name: minio
        key: secret-access-key
      tls:
        enabled: true
        caSecretKeyRef:
          name: minio-ca
          key: ca.crt

Alternatively, if you are running in EKS, you can use dynamic credentials from an EKS Service Account using EKS Pod Identity or IRSA:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: mariadb-backup
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<<account_id>>:role/my-role-irsa

apiVersion: enterprise.mariadb.com/v1alpha1
kind: PhysicalBackup
metadata:
  name: physicalbackup
spec:
  mariaDbRef:
    name: mariadb
  serviceAccountName: mariadb-backup
  storage:
    s3:
      bucket: physicalbackups
      prefix: mariadb
      endpoint: s3.us-east-1.amazonaws.com
      region:  us-east-1
      tls:
        enabled: true

By leaving out the accessKeyIdSecretKeyRef and secretAccessKeySecretKeyRef credentials and pointing to the correct serviceAccountName, the backup Job will use the dynamic credentials from EKS.

Staging area

When using S3 storage for backups, a staging area is used for keeping the external backups while they are being processed. By default, this staging area is an emptyDir volume, which means that the backups are temporarily stored in the node's local storage where the PhysicalBackup Job is scheduled. In production environments, large backups may lead to issues if the node doesn't have sufficient space, potentially causing the backup/restore process to fail.

Additionally, when restoring these backups, the operator will pull the backup files from S3, uncompress them if needded, and restore them to each of the MariaDB Pods in the cluster individually. To save network bandwidth and compute resources, a staging area is used to keep the uncompressed backup files after they have been restored to the first MariaDB Pod. This allows the operator to restore the same backup to the rest of MariaDB Pods seamlessly, without needing to pull and uncompress the backup again.

To configure the staging area, you can use the stagingStorage field in the PhysicalBackup resource:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: PhysicalBackup
metadata:
  name: physicalbackup
spec:
  mariaDbRef:
    name: mariadb
  storage:
    s3:
      bucket: physicalbackups
      prefix: mariadb
      endpoint: minio.minio.svc.cluster.local:9000
      region:  us-east-1
      accessKeyIdSecretKeyRef:
        name: minio
        key: access-key-id
      secretAccessKeySecretKeyRef:
        name: minio
        key: secret-access-key
      tls:
        enabled: true
        caSecretKeyRef:
          name: minio-ca
          key: ca.crt
  stagingStorage:
    persistentVolumeClaim:
      resources:
        requests:
          storage: 1Gi
      accessModes:
        - ReadWriteOnce

Similarly, you may also use a staging area when bootstrapping from backup, in the MariaDB resource:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  mariaDbRef:
    name: mariadb
  bootstrapFrom:
    s3:
      bucket: physicalbackups
      prefix: mariadb
      endpoint: minio.minio.svc.cluster.local:9000
      accessKeyIdSecretKeyRef:
        name: minio
        key: access-key-id
      secretAccessKeySecretKeyRef:
        name: minio
        key: secret-access-key
      tls:
        enabled: true
        caSecretKeyRef:
          name: minio-ca
          key: ca.crt
    backupContentType: Physical
    stagingStorage:
      persistentVolumeClaim:
        resources:
          requests:
            storage: 1Gi
        accessModes:
          - ReadWriteOnce

In the examples above, a PVC with the default StorageClass will be provisioned to be used as staging area.

VolumeSnapshots

The operator is capable of creating VolumeSnapshot resources of the PVCs used by the MariaDB Pods. This allows you to create point-in-time snapshots of your data in a Kubernetes-native way, leveraging the capabilities of your storage provider.

Most of the fields described in this documentation apply to VolumeSnapshots, including scheduling, retention policy, and compression. The main difference with the mariadb-backup based backups is that the operator will not create a Job to perform the backup, but instead it will create a VolumeSnapshot resource directly.

In order to create consistent, point-in-time snapshots of the MariaDB data, the operator will perform the following steps:

1. Execute a BACKUP STAGE START statement followed by BACKUP STAGE BLOCK_COMMIT in one of the secondary Pods.

2. Create a VolumeSnapshot resource of the data PVC mounted by the MariaDB secondary Pod.

3. Wait until the VolumeSnapshot resource becomes ready. When timing out, the operator will delete the VolumeSnapshot resource and retry the operation.

4. Issue a BACKUP STAGE END statement.

This backup process is described in the MariaDB documentation and is designed to be non-blocking.

Non-blocking physical backups

Both for mariadb-backup and VolumeSnapshot backup strategies, the enterprise operator performs non-blocking physical backups by leveraging the BACKUP STAGE feature.. This implies that the backups are taken without long read locks, enabling consistent, production-grade backups with minimal impact on running workloads, ideal for high-availability and performance-sensitive environments.

Important considerations and limitations

Root credentials

When restoring a backup, the root credentials specified through the spec.rootPasswordSecretKeyRef field in the MariaDB resource must match the ones in the backup. These credentials are utilized by the liveness and readiness probes, and if they are invalid, the probes will fail, causing your MariaDB Pods to restart after the backup restoration.

Restore Job

When using backups based on mariadb-backup, restoring and uncompressing large backups can consume significant compute resources and may cause restoration Jobs to become stuck due to insufficient resources. To prevent this, you can define the compute resources allocated to the Job:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
  bootstrapFrom:
    restoreJob:
      resources:
        requests:
          cpu: 100m
          memory: 128Mi
        limits:
          memory: 1Gi

ReadWriteOncePod access mode partially supported

When using backups based on mariadb-backup, the data PVC used by the MariaDB Pod cannot use the ReadWriteOncePod access mode, as it needs to be mounted at the same time by both the MariaDB Pod and the PhysicalBackup Job. In this case, please use either the ReadWriteOnce or ReadWriteMany access modes instead.

Alternatively, if you want to keep using the ReadWriteOncePod access mode, you must use backups based on VolumeSnapshots, which do not require creating a Job to perform the backup and therefore avoid the volume sharing limitation.

PhysicalBackup Jobs scheduling

PhysicalBackup Jobs must mount the data PVC used by one of the secondary MariaDB Pods. To avoid scheduling issues caused by the commonly used ReadWriteOnce access mode, the operator schedules backup Jobs on the same node as MariaDB by default.

If you prefer to disable this behavior and allow Jobs to run on any node, you can set podAffinity=false:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: PhysicalBackup
metadata:
  name: physicalbackup
spec:
  mariaDbRef:
    name: mariadb
  podAffinity: false

This configuration may be suitable when using the ReadWriteMany access mode, which allows multiple Pods across different nodes to mount the volume simultaneously.

Troubleshooting

Custom columns are used to display the status of the PhysicalBackup resource:

kubectl get physicalbackups

NAME             COMPLETE   STATUS    MARIADB   LAST SCHEDULED   AGE
physicalbackup   True       Success   mariadb   17s              17s

To get a higher level of detail, you can also check the status field directly:

kubectl get physicalbackups physicalbackup -o json | jq -r '.status'

{
  "conditions": [
    {
      "lastTransitionTime": "2025-07-14T07:01:14Z",
      "message": "Success",
      "reason": "JobComplete",
      "status": "True",
      "type": "Complete"
    }
  ],
  "lastScheduleCheckTime": "2025-07-14T07:00:00Z",
  "lastScheduleTime": "2025-07-14T07:00:00Z",
  "nextScheduleTime": "2025-07-15T07:00:00Z"
}

You may also check the related events for the PhysicalBackup resource to see if there are any issues:

kubectl get events --field-selector involvedObject.name=physicalbackup

LAST SEEN   TYPE     REASON                  OBJECT                                 MESSAGE
116s        Normal   WaitForFirstConsumer    persistentvolumeclaim/physicalbackup   waiting for first consumer to be created before binding
116s        Normal   JobScheduled            physicalbackup/physicalbackup          Job physicalbackup-20250714140837 scheduled
116s        Normal   ExternalProvisioning    persistentvolumeclaim/physicalbackup   Waiting for a volume to be created either by the external provisioner 'rancher.io/local-path' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
116s        Normal   Provisioning            persistentvolumeclaim/physicalbackup   External provisioner is provisioning volume for claim "default/physicalbackup"
113s        Normal   ProvisioningSucceeded   persistentvolumeclaim/physicalbackup   Successfully provisioned volume pvc-7b7c71f9-ea7e-4950-b612-2d41d7ab35b7

Common errors

mariadb-backup log copy incomplete: consider increasing innodb_log_file_size

In some situations, when using the mariadb-backup strategy, you may encounter the following error in the backup Job logs:

mariadb [00] 2025-08-04 09:15:57 Was only able to copy log from 58087 to 59916, not 68968; try increasing
innodb_log_file_size
mariadb mariabackup: Stopping log copying thread.[00] 2025-08-04 09:15:57 Retrying read of log at LSN=59916

This can be addressed by increasing the innodb_log_file_size in the MariaDB configuration. You can do this by adding the following to your MariaDB resource:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
...
  myCnf: |
    [mariadb]
    innodb_log_file_size=200M

Refer to MDEV-36159 for further details on this issue.

MaxScale is a sophisticated database proxy, router, and load balancer designed specifically for and by MariaDB. It provides a range of features that ensure optimal high availability:

• Query-based routing: Transparently route write queries to the primary nodes and read queries to the replica nodes.

• Connection-based routing: Load balance connections between multiple servers.

• Automatic primary failover based on MariaDB internals.

• Replay pending transactions when a server goes down.

• Support for Galera and Replication.

To better understand what MaxScale is capable of you may check the and the .

MaxScale resources

Prior to configuring MaxScale within Kubernetes, it's essential to have a basic understanding of the resources managed through its API.

Servers

A server defines the backend database servers that MaxScale forwards traffic to. For more detailed information, please consult the .

Monitors

A monitor is an agent that queries the state of the servers and makes it available to the services in order to route traffic based on it. For more detailed information, please consult the .

Depending on which highly available configuration your servers have, you will need to choose betweeen the following modules:

• : Detects whether servers are part of the cluster, ensuring synchronization among them, and assigning primary and replica roles as needed.

• : Probes the state of the cluster, assigns roles to the servers, and executes failover, switchover, and rejoin operations as necessary.

Services

A service defines how the traffic is routed to the servers based on a routing algorithm that takes into account the state of the servers and its role. For more detailed information, please consult the .

Depending on your requirements to route traffic, you may choose between the following routers:

• : Route write queries to the primary server and read queries to the replica servers.

• : Load balance connections between multiple servers.

Listeners

A listener specifies a port where MaxScale listens for incoming connections. It is associated with a service that handles the requests received on that port. For more detailed information, please consult the .

MaxScale CR

The minimal spec you need to provision a MaxScale instance is just a reference to a MariaDB resource:

This will provision a new StatefulSet for running MaxScale and configure the servers specified by the MariaDB resource. Refer to the section if you want to manually configure the MariaDB servers.

The rest of the configuration uses reasonable set automatically by the operator. If you need a more fine grained configuration, you can provide this values yourself:

As you can see, the we previously mentioned have a counterpart resource in the MaxScale CR.

You also need to set a reference in the MariaDB resource to make it MaxScale-aware. This is explained in the section.

Refer to the for further detail.

MariaDB CR

You can set a spec.maxScaleRef in your MariaDB resource to make it MaxScale-aware. By doing so, the primary server reported by MaxScale will be used in MariaDB and the high availability tasks such the primary failover will be delegated to MaxScale:

Refer to the for further detail.

MaxScale embedded in MariaDB

To streamline the setup outlined in the and sections, you can provision a MaxScale to be used with MariaDB in just one resource:

This will automatically set the references between MariaDB and MaxScale and the rest of the fields.

It is important to note that, this is intended for simple use cases that only require a single replica and where no further modifications are done on the spec.maxscale field. If you need a more fine grained configuration and perform further updates to the MaxScale resource, please use a dedicated MaxScale as described in the section.

Refer to the for further detail.

Defaults

MariaDB Enterprise Operator aims to provide highly configurable CRs, but at the same time maximize its usability by providing reasonable defaults. In the case of MaxScale, the following defaulting logic is applied:

• spec.servers are inferred from spec.mariaDbRef.

• spec.monitor.module is inferred from the spec.mariaDbRef.

• spec.monitor.cooperativeMonitoring is set if is enabled.

• If spec.services is not provided, a readwritesplit service is configured on port 3306 by default.

Server configuration

As an alternative to provide a reference to a MariaDB via spec.mariaDbRef, you can also specify the servers manually:

As you could see, you can refer to in-cluser MariaDB servers by providing the DNS names of the MariaDB Pods as server addresses. In addition, you can also refer to external MariaDB instances running outside of the Kubernetes cluster where the operator was deployed:

Pointing to external MariaDBs has a some limitations: Since the operator doesn't have a reference to a MariaDB resource (spec.mariaDbRef), it will be unable to perform the following actions:

• Infer the monitor module (spec.monitor.module), so it will need to be provided by the user.

• Autogenerate authentication credentials (spec.auth), so they will need to be provided by the user. See section.

Server maintenance

You can put servers in maintenance mode by setting maintenance = true:

Maintenance mode prevents MaxScale from routing traffic to the server and also excludes it from being elected as the new primary during failover events.

Configuration

Similar to MariaDB, MaxScale allows you to provide global configuration parameters in a maxscale.conf file. You don't need to provide this config file directly, but instead you can use the spec.config.params to instruct the operator to create the maxscale.conf:

Both this global configuration and the resources created by the operator using the are stored under a volume provisioned by the spec.config.volumeClaimTemplate.

Refer to the for more details about the supported parameters.

Authentication

MaxScale requires authentication with differents levels of permissions for the following components/actors:

• consumed by MariaDB Enterprise Operator.

• Clients connecting to MaxScale.

• MaxScale connecting to MariaDB servers.

• MaxScale monitor connecting to MariaDB servers.

• MaxScale configuration syncer to connect to MariaDB servers. See section.

By default, the operator generates this credentials when spec.mariaDbRef is set and spec.auth.generate = true, but you are still able to provide your own:

As you could see, you are also able to limit the number of connections for each component/actor. Bear in mind that, when running in , you may need to increase this number, as more MaxScale instances implies more connections.

Kubernetes Services

To enable your applications to communicate with MaxScale, a Kubernetes Service is provisioned with all the ports specified in the MaxScale listeners. You have the flexibility to provide a template to customize this Service:

This results in the reconciliation of the following Service:

There is also another Kubernetes Service to access the GUI, please refer to the section for further detail.

Connection

You can leverage the Connection resource to automatically configure connection strings as Secret resources that your applications can mount:

Alternatively, you can also provide a connection template to your MaxScale resource:

Note that, the Connection uses the Service described in the section and you are able to specify which MaxScale service to connect to by providing the port (spec.port) of the corresponding MaxScale listener.

High availability

To synchronize the configuration state across multiple replicas, MaxScale stores the configuration externally in a MariaDB table and conducts periodic polling across all replicas. By default, the table mysql.maxscale_config is used, but this can be configured by the user as well as the synchronization interval.

Another crucial aspect to consider regarding HA is that only one monitor can be running at any given time to avoid conflicts. This can be achieved via cooperative locking, which can be configured by the user. Refer to for more information.

Multiple MaxScale replicas can be specified by providing the spec.replicas field. Note that, MaxScale exposes the , so you can scale/downscale it by running the following command:

Suspend resources

In order to enable this feature, you must set the --feature-maxscale-suspend feature flag:

Then you will be able to suspend any , for instance, you can suspend a monitor:

MaxScale GUI

MaxScale offers a great user interface that provides very useful information about the . You can enable it by providing the following configuration:

The GUI is exposed via a dedicated Kubernetes Service in the same port as the . Once you access, you will need to enter the credentials configured by the operator in a Secret. See the section for more details.

MaxScale API

MariaDB Enterprise Operator interacts with the to reconcile the specification provided by the user, considering both the MaxScale status retrieved from the API and the provided spec.

Troubleshooting

The operator tracks both the MaxScale status in regards to Kubernetes resources as well as the status of the resources. This information is available on the status field of the MaxScale resource, it may be very useful for debugging purposes:

Kubernetes events emitted by mariadb-enterprise-operator may also be very relevant for debugging. For instance, an event is emitted whenever the primary server changes:

The operator logs can also be a good source of information for troubleshooting. You can increase its verbosity and enable request logs by running:

Common errors

Permission denied writing /var/lib/maxscale

This error occurs when the user that runs the container does not have enough privileges to write in /var/lib/maxscale:

To mitigate this, by default, the operator sets the following securityContext in the MaxScale's StatefulSet:

This enables the CSIDriver and the kubelet to recursively set the ownership ofr the /var/lib/maxscale folder to the group 999, which is the one expected by MaxScale. It is important to note that not all the CSIDrivers implementations support this feature, see the for further information.

What is a logical backup?

A logical backup is a backup that contains the logical structure of the database, such as tables, indexes, and data, rather than the physical storage format. It is created using mariadb-dump, which generates SQL statements that can be used to recreate the database schema and populate it with data.

Logical backups serve not just as a source of restoration, but also enable data mobility between MariaDB instances. These backups are called "logical" because they are independent from the MariaDB topology, as they only contain DDLs and INSERT statements to populate data.

Although logical backups are a great fit for data mobility and migrations, they are not as efficient as physical backups for large databases. For this reason, physical backups are the recommended method for backing up MariaDB databases, especially in production environments.

Storage types

Currently, the following storage types are supported:

• S3 compatible storage: Store backups in a S3 compatible storage, such as AWS S3 or Minio.

• PVCs: Use the available StorageClasses in your Kubernetes cluster to provision a PVC dedicated to store the backup files.

• Kubernetes volumes: Use any of the volume types supported natively by Kubernetes.

Our recommendation is to store the backups externally in a S3 compatible storage.

Backup CR

You can take a one-time backup of your MariaDB instance by declaring the following resource:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Backup
metadata:
  name: backup
spec:
  mariaDbRef:
    name: mariadb
  storage:
    persistentVolumeClaim:
      resources:
        requests:
          storage: 100Mi
      accessModes:
        - ReadWriteOnce

This will use the default StorageClass to provision a PVC that would hold the backup files, but ideally you should use a S3 compatible storage:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Backup
metadata:
  name: backup
spec:
  mariaDbRef:
    name: mariadb
  storage:
    s3:
      bucket: backups
      prefix: mariadb
      endpoint: minio.minio.svc.cluster.local:9000
      region:  us-east-1
      accessKeyIdSecretKeyRef:
        name: minio
        key: access-key-id
      secretAccessKeySecretKeyRef:
        name: minio
        key: secret-access-key
      tls:
        enabled: true
        caSecretKeyRef:
          name: minio-ca
          key: tls.crt

By providing the authentication details and the TLS configuration via references to Secret keys, this example will store the backups in a local Minio instance.

Alternatively you can use dynamic credentials from an EKS Service Account using EKS Pod Identity or IRSA:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: mariadb-backup
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<<account_id>>:role/my-role-irsa

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Backup
metadata:
  name: backup
spec:
  mariaDbRef:
    name: mariadb
  serviceAccountName: mariadb-backup
  storage:
    s3:
      bucket: backups
      prefix: mariadb
      endpoint: s3.us-east-1.amazonaws.com
      region:  us-east-1
      tls:
        enabled: true

By leaving out the accessKeyIdSecretKeyRef and secretAccessKeySecretKeyRef credentials and pointing to the correct serviceAccountName, the backup Job will use the dynamic credentials from EKS.

Scheduling

To minimize the Recovery Point Objective (RPO) and mitigate the risk of data loss, it is recommended to perform backups regularly. You can do so by providing a spec.schedule in your Backup resource:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Backup
metadata:
  name: backup
spec:
  mariaDbRef:
    name: mariadb
  schedule:
    cron: "*/1 * * * *"
    suspend: false

This resource gets reconciled into a CronJob that periodically takes the backups.

It is important to note that regularly scheduled Backups complement very well the target recovery time feature detailed below.

Retention policy

Given that the backups can consume a substantial amount of storage, it is crucial to define your retention policy by providing the spec.maxRetention field in your Backup resource:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Backup
metadata:
  name: backup
spec:
  mariaDbRef:
    name: mariadb
  maxRetention: 720h # 30 days

Compression

You are able to compress backups by providing the compression algorithm you want to use in the spec.compression field:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Backup
metadata:
  name: backup
spec:
  mariaDbRef:
    name: mariadb
  compression: gzip

Currently the following compression algorithms are supported:

• bzip2: Good compression ratio, but slower compression/decompression speed compared to gzip.

• gzip: Good compression/decompression speed, but worse compression ratio compared to bzip2.

• none: No compression.

compression is defaulted to none by the operator.

Restore CR

You can easily restore a Backup in your MariaDB instance by creating the following resource:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Restore
metadata:
  name: restore
spec:
  mariaDbRef:
    name: mariadb
  backupRef:
    name: backup

This will trigger a Job that will mount the same storage as the Backup and apply the dump to your MariaDB database.

Nevertheless, the Restore resource doesn't necessarily need to specify a spec.backupRef, you can point to other storage source that contains backup files, for example a S3 bucket:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Restore
metadata:
  name: restore
spec:
  mariaDbRef:
    name: mariadb
  s3:
    bucket: backups
    prefix: mariadb
    endpoint: minio.minio.svc.cluster.local:9000
    region:  us-east-1
    accessKeyIdSecretKeyRef:
      name: minio
      key: access-key-id
    secretAccessKeySecretKeyRef:
      name: minio
      key: secret-access-key
    tls:
      enabled: true
      caSecretKeyRef:
        name: minio-ca
        key: tls.crt

Target recovery time

If you have multiple backups available, specially after configuring a scheduled Backup, the operator is able to infer which backup to restore based on the spec.targetRecoveryTime field.

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Restore
metadata:
  name: restore
spec:
  mariaDbRef:
    name: mariadb
  backupRef:
    name: backup
  targetRecoveryTime: 2023-12-19T09:00:00Z

The operator will look for the closest backup available and utilize it to restore your MariaDB instance.

By default, spec.targetRecoveryTime will be set to the current time, which means that the latest available backup will be used.

Bootstrap new MariaDB instances

To minimize your Recovery Time Objective (RTO) and to switfly spin up new clusters from existing Backups, you can provide a Restore source directly in the MariaDB object via the spec.bootstrapFrom field:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-from-backup
spec:
  storage:
    size: 1Gi
  bootstrapFrom:
    backupRef:
      name: backup
    targetRecoveryTime: 2023-12-19T09:00:00Z

As in the Restore resource, you don't strictly need to specify a reference to a Backup, you can provide other storage types that contain backup files:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-from-backup
spec:
  storage:
    size: 1Gi
  bootstrapFrom:
    s3:
      bucket: backups
      prefix: mariadb
      endpoint: minio.minio.svc.cluster.local:9000
      accessKeyIdSecretKeyRef:
        name: minio
        key: access-key-id
      secretAccessKeySecretKeyRef:
        name: minio
        key: secret-access-key
      tls:
        enabled: true
        caSecretKeyRef:
          name: minio-ca
          key: tls.crt
    targetRecoveryTime: 2023-12-19T09:00:00Z

Under the hood, the operator creates a Restore object just after the MariaDB resource becomes ready. The advantage of using spec.bootstrapFrom over a standalone Restore is that the MariaDB is bootstrap-aware and this will allow the operator to hold primary switchover/failover operations until the restoration is finished.

Backup and restore specific databases

By default, all the logical databases are backed up when a Backup is created, but you may also select specific databases by providing the databases field:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Backup
metadata:
  name: backup
spec:
  mariaDbRef:
    name: mariadb
  databases:
    - db1
    - db2
    - db3

When it comes to restore, all the databases available in the backup will be restored, but you may also choose a single database to be restored via the database field available in the Restore resource:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Restore
metadata:
  name: restore
spec:
  mariaDbRef:
    name: mariadb
  backupRef:
    name: backup
  database: db1

There are a couple of points to consider here:

• The referred database (db1 in the example) must previously exist for the Restore to succeed.

• The mariadb CLI invoked by the operator under the hood only supports selecting a single database to restore via the --one-database option, restoration of multiple specific databases is not supported.

Extra options

Not all the flags supported by mariadb-dump and mariadb have their counterpart field in the Backup and Restore CRs respectively, but you may pass extra options by using the args field. For example, setting the --verbose flag can be helpful to track the progress of backup and restore operations:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Backup
metadata:
  name: backup
spec:
  mariaDbRef:
    name: mariadb
  args:
    - --verbose

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Restore
metadata:
  name: restore
spec:
  mariaDbRef:
    name: mariadb
  backupRef:
    name: backup
  args:
    - --verbose

Refer to the mariadb-dump and mariadb CLI options in the reference section.

Staging area

When using S3 storage for backups, a staging area is used for keeping the external backups while they are being processed. By default, this staging area is an emptyDir volume, which means that the backups are temporarily stored in the node's local storage where the Backup/Restore Job is scheduled. In production environments, large backups may lead to issues if the node doesn't have sufficient space, potentially causing the backup/restore process to fail.

To overcome this limitation, you are able to define your own staging area by setting the stagingStorage field to both the Backup and Restore CRs:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Backup
metadata:
  name: backup
spec:
  storage:
    s3:
      ...
  stagingStorage:
    persistentVolumeClaim:
      resources:
        requests:
          storage: 10Gi
      accessModes:
        - ReadWriteOnce

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Restore
metadata:
  name: restore
spec:
  s3:
    ...
  stagingStorage:
    persistentVolumeClaim:
      resources:
        requests:
          storage: 10Gi
      accessModes:
        - ReadWriteOnce

In the examples above, a PVC with the default StorageClass will be used as staging area. Refer to the API reference for more configuration options.

Similarly, you may also use a custom staging area when bootstrapping from backup:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
  bootstrapFrom:
    s3:
      ...
    stagingStorage:
      persistentVolumeClaim:
        resources:
          requests:
            storage: 10Gi
        accessModes:
          - ReadWriteOnce

Important considerations and limitations

Root credentials

When restoring a backup, the root credentials specified through the spec.rootPasswordSecretKeyRef field in the MariaDB resource must match the ones in the backup. These credentials are utilized by the liveness and readiness probes, and if they are invalid, the probes will fail, causing your MariaDB Pods to restart after the backup restoration.

Restore job

Restoring large backups can consume significant compute resources and may cause Restore Jobs to become stuck due to insufficient resources. To prevent this, you can define the compute resources allocated to the Job:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
  storage:
    size: 1Gi
  bootstrapFrom:
    restoreJob:
      args:
        - --verbose
      resources:
        requests:
          cpu: 100m
          memory: 128Mi
        limits:
          memory: 1Gi

Galera backup limitations

mysql.global_priv

Galera only replicates the tables with InnoDB engine, see the Galera docs.

Something that does not include mysql.global_priv, the table used to store users and grants, which uses the MyISAM engine. This basically means that a Galera instance with mysql.global_priv populated will not replicate this data to an empty Galera instance. However, DDL statements (CREATE USER, ALTER USER ...) will be replicated.

Taking this into account, if we think now about a restore scenario where:

• The backup file includes a DROP TABLE statement for the mysql.global_priv table.

• The backup has some INSERT statements for the mysql.global_priv table.

• The Galera cluster has 3 nodes: galera-0, galera-1 and galera-2.

• The backup is restored in galera-0.

This is what will happen under the scenes while restoring the backup:

• The DROP TABLE statement is a DDL so it will be executed in galera-0, galera-1 and galera-2.

• The INSERT statements are not DDLs, so they will only be applied to galera-0.

• This results in the galera-1 and galera-2 not having the mysql.global_priv table.

After the backup is fully restored, the liveness and readiness probes will kick in, they will succeed in galera-0, but they will fail in galera-1 and galera-2, as they rely in the root credentials available in mysql.global_priv, resulting in the galera-1 and galera-2 getting restarted.

To address this issue, when backing up MariaDB instances with Galera enabled, the mysql.global_priv table will be excluded from backups by using the --ignore-table option with mariadb-dump. This prevents the replication of the DROP TABLE statement for the mysql.global_priv table. You can opt-out from this feature by setting spec.ignoreGlobalPriv=false in the Backup resource.

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Backup
metadata:
  name: backup
spec:
  mariaDbRef:
    name: mariadb
  ignoreGlobalPriv: false

Also, to avoid situations where mysql.global_priv is unreplicated, all the entries in that table must be managed via DDLs. This is the recommended approach suggested in the Galera docs. There are a couple of ways that we can guarantee this:

• Use the rootPasswordSecretKeyRef, username and passwordSecretKeyRef fields of the MariaDB CR to create the root and initial user respectively. This fields will be translated into DDLs by the image entrypoint.

• Rely on the User and Grant CRs to create additional users and grants. Refer to the SQL resource documentation for further detail.

LOCK TABLES

Galera is not compatible with the LOCK TABLES statement:

• LOCK TABLES Limitations

For this reason, the operator automatically adds the --skip-add-locks option to the Backup to overcome this limitation.

Migrations using logical backups

Migrating an external MariaDB to a MariaDB running in Kubernetes

You can leverage logical backups to bring your external MariaDB data into a new MariaDB instance running in Kubernetes. Follow this runbook for doing so:

1. Take a logical backup of your external MariaDB using one of the commands below:

mariadb-dump --user=${MARIADB_USER} --password=${MARIADB_PASSWORD} --host=${MARIADB_HOST} --single-transaction --events --routines --all-databases > backup.2024-08-26T12:24:34Z.sql

mariadb-dump --user=${MARIADB_USER} --password=${MARIADB_PASSWORD} --host=${MARIADB_HOST} --single-transaction --events --routines --all-databases --skip-add-locks --ignore-table=mysql.global_priv > backup.2024-08-26T12:24:34Z.sql

1. Ensure that your backup file is named in the following format: backup.2024-08-26T12:24:34Z.sql. If the file name does not follow this format, it will be ignored by the operator.

2. Upload the backup file to one of the supported storage types. We recommend using S3.

3. Create your MariaDB resource declaring that you want to bootstrap from the previous backup and providing a root password Secret that matches the backup:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  rootPasswordSecretKeyRef:
    name: mariadb
    key: root-password
  replicas: 3
  galera:
    enabled: true
  storage:
    size: 1Gi
  bootstrapFrom:
    s3:
      bucket: backups
      prefix: mariadb
      endpoint: minio.minio.svc.cluster.local:9000
      accessKeyIdSecretKeyRef:
        name: minio
        key: access-key-id
      secretAccessKeySecretKeyRef:
        name: minio
        key: secret-access-key
      tls:
        enabled: true
        caSecretKeyRef:
          name: minio-ca
          key: tls.crt
    targetRecoveryTime: 2024-08-26T12:24:34Z

1. If you are using Galera in your new instance, migrate your previous users and grants to use the User and Grant CRs. Refer to the SQL resource documentation for further detail.

Migrating to a MariaDB with different topology

Database mobility between MariaDB instances with different topologies is possible with logical backups. However, there are a couple of technical details that you need to be aware of in the following scenarios:

Migrating between standalone and replicated MariaDBs

This should be fully compatible, no issues have been detected.

Migrating from standalone/replicated to Galera MariaDBs

There are a couple of limitations regarding the backups in Galera, please make sure you read the Galera backup limitations section before proceeding.

To overcome this limitations, the Backup in the standalone/replicated instance needs to be taken with spec.ignoreGlobalPriv=true. In the following example, we are backing up a standalone MariaDB (single instance):

apiVersion: enterprise.mariadb.com/v1alpha1
kind: Backup
metadata:
  name: backup-standalone
spec:
  mariaDbRef:
    name: mariadb-standalone
  ignoreGlobalPriv: true

Once the previous Backup is completed, we will be able bootstrap a new Galera instance from it:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
spec:
  replicas: 3
  galera:
    enabled: true
  storage:
    size: 1Gi
  bootstrapFrom:
    backupRef:
      name: backup-standalone