MariaDB Enterprise Kubernetes Operator provides a seamless way to run and operate containerized versions of MariaDB Enterprise Server and MaxScale on Kubernetes, allowing you to leverage Kubernetes orchestration and automation capabilities. This document outlines the features and advantages of using Kubernetes and the MariaDB Enterprise Kubernetes Operator to streamline the deployment and management of MariaDB and MaxScale instances.

What is Kubernetes?

Kubernetes is more than just a container orchestrator; it is a comprehensive platform that provides APIs for managing both applications and the underlying infrastructure. It automates key aspects of container management, including deployment, scaling, and monitoring, while also handling essential infrastructure needs such as networking and storage. By unifying the management of applications and infrastructure, Kubernetes simplifies operations and improves efficiency in cloud-native environments.

Why Kubernetes?

Kubernetes brings several key benefits to the table when managing applications in a containerized environment:

• Standardization: Kubernetes relies on standard APIs for managing applications and infrastructure, making it easier to ensure uniformity across various environments. It acts as a common denominator across cloud providers and on-premises.

• Automation: Kubernetes APIs encapsulate operational best practises, minimizing the need for manual intervention and improving the efficiency of operations.

• Cost Effectiveness: Having an standarized way to manage infrastructure across cloud providers and automation to streamline operations, Kubernetes helps reducing the infrastructure and operational costs.

What is a Kubernetes Operator?

Kubernetes has been designed with flexibility in mind, allowing developers to extend its capabilities through custom resources and operators.

In particular, MariaDB Enterprise Kubernetes Operator, watches the desired state defined by users via MariaDB and MaxScale resources, and takes actions to ensure that the actual state of the system matches the desired state. This includes managing compute, storage and network resources, as well as the full lifecycle of the MariaDB and MaxScale instances. Whenever the desired state changes or the underlying infrastructure is modified, the Operator takes the necessary actions to reconcile the actual state with the desired state.

Operational expertise is baked into the MariaDB and MaxScale APIs and seamlessly managed by the Operator. This includes automated backups, restores, upgrades, monitoring, and other critical lifecycle tasks, ensuring reliability in Day 2 operations.

MariaDB Enterprise Kubernetes Operator Features

• Provision and Configure MariaDB and MaxScale Declaratively: Define MariaDB Enterprise Server and MaxScale clusters in YAML manifests and deploy them with ease in Kubernetes.

• Multiple Topologies supported:

This documentation aims to provide guidance on how to configure access to docker.mariadb.com in your MariaDB Enterprise Kubernetes Operator resources.

Customer credentials

MariaDB Corporation requires customers to authenticate when logging in to the . A Customer Download Token must be provided as the password. Customer Download Tokens are available through the MariaDB Customer Portal. To retrieve the customer download token for your account:

Certified images

All the Docker images used by this operator are based on and have been . The advantages of using UBI based images are:

• Immutability: UBI images are built to be secure and stable, reducing the risk of unintended changes or vulnerabilities due to mutable base layers.

This operator allows you to configure standalone MariaDB Enterprise Server instances. To achieve this, you can either omit the replicas field or set it to 1:

apiVersion: enterprise.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
  rootPasswordSecretKeyRef:
    name: mariadb
    key: password

  replicas: 1

  port: 3306

  storage:
    size: 1Gi

  myCnf: |
    [mariadb]
    bind-address=*
    default_storage_engine=InnoDB
    binlog_format=row
    innodb_autoinc_lock_mode=2
    innodb_buffer_pool_size=800M
    max_allowed_packet=256M

  resources:
    requests:
      cpu: 500m
      memory: 1Gi
    limits:
      memory: 1Gi

  metrics:
    enabled: true

Whilst this can be useful for development and testing, it is not recommended for production use because of the following reasons:

• Single point of failure

• Upgrades require downtime

• Only vertical scaling is possible

For achieving high availability, we recommend deploying a highly available topology as described in the .

This guide aims to provide a quick way to get started with the MariaDB Enterprise Kubernetes Operator for Kubernetes. It will walk you through the process of deploying a MariaDB Enterprise Cluster and MaxScale via the MariaDB and MaxScale CRs () respectively.

Before you begin, ensure you meet the following prerequisites:

• Configure your

The first step will be configuring a Secret

blob-csi-driver (Azure Blob Storage)

This section outlines a recommended StorageClass configuration for the that resolves common mounting and list operation issues encountered in Kubernetes environments.

The following is recommended when working with Azure Blob Storage (ABS).

Next, when defining your PhysicalBackup resource, make sure to use the new StorageClass we created.

This guide illustrates, step by step, how to update to 25.8.0 from previous versions.

• Uninstall you current mariadb-enterprise-operator for preventing conflicts:

Alternatively, you may only downscale and delete the webhook configurations:

• Upgrade mariadb-enterprise-operator-crds to 25.8.0

Suspended state

When a resource is suspended, all operations performed by the operator are disabled, including but not limited to:

• Provisioning

• Upgrades

The contains a number of sample manifests that aim to show the operator functionality in a practical way. Follow these instructions for getting started:

• Download the :

• Install the configuration shared by all the examples:

• Start deploying examples:

This guide illustrates, step by step, how to update to 25.10.3 from previous versions. This guide only applies if you are updating from a version prior to 25.10.x, otherwise you may upgrade directly (see Helm and OpenShift docs)

• The Galera data-plane must be updated to the 25.10.3 version. You must set updateStrategy.autoUpdateDataPlane=true in your MariaDB resources before updating the operator. Then, once updated, the operator will also be updating the data-plane based on its version:

• Once set, you may proceed to update the operator. If you are using Helm:

Upgrade the mariadb-enterprise-operator-crds helm chart to 25.10.3:

Upgrade the mariadb-enterprise-operator helm chart to 25.10.3:

As part of the 25.10 LTS release, we have introduced support for LTS versions. Refer to the for sticking to LTS versions.

• If you are on OpenShift:

If you are on the stable channel using installPlanApproval=Automatic in your Subscription object, then the operator will be automatically updated. If you use installPlanApproval=Manual, you should have a new InstallPlan which needs to be approved to update the operator:

As part of the 25.10 LTS release, we have introduced new . Consider switching to the stable-v25.10 if you are willing to stay in the 25.10.x version:

• Consider reverting updateStrategy.autoUpdateDataPlane back to false in your MariaDB object to avoid unexpected updates:

In this guide, we will be migrating an external MariaDB into a new MariaDB instance running in Kubernetes and managed by MariaDB Enterprise Kubernetes Operator. We will be using logical backups for achieving this migration.

1. Take a logical backup of your external MariaDB using one of the commands below:

mariadb-dump --user=${MARIADB_USER} --password=${MARIADB_PASSWORD} --host=${MARIADB_HOST} --single-transaction --events --routines --all-databases > backup.2024-08-26T12:24:34Z.sql

If you are currently using or migrating to a Galera instance, use the following command instead:

mariadb-dump --user=${MARIADB_USER} --password=${MARIADB_PASSWORD} --host=${MARIADB_HOST} --single-transaction --events --routines --all-databases --skip-add-locks --ignore-table=mysql.global_priv > backup.2024-08-26T12:24:34Z.sql

2. Ensure that your backup file matches the following format: backup.2024-08-26T12:24:34Z.sql. If the file name does not follow this format, it will be ignored by the operator.

3. Upload the backup file to one of the supported . We recommend using S3.

4. Create your MariaDB resource declaring that you want to and providing a that matches the backup:

5. If you are using Galera in your new instance, migrate your previous users and grants to use the User and Grant CRs. Refer to the for further detail.

This documentation shows how to configure metadata in the MariaDB Enterprise Kubernetes Operator CRs.

Children object metadata

MariaDB and MaxScale resources allow you to propagate metadata to all the children objects by specifying the inheritMetadata field:

This means that all the reconciled objects will inherit these labels and annotations. For instance, see the Services

In order to effectively manage the full lifecycle of both and topologies, the operator relies on a set of components that run alonside the MariaDB instances and expose APIs for remote management. These components are collectively referred to as the "data-plane".

Components

The mariadb-enterprise-operator data-plane components are implemented as lightweight containers that run alongside the MariaDB instances within the same Pod. These components are available in the operator image. More preciselly, they are subcommands of the CLI shipped as binary inside the image.

This operator gives you flexibility to define the storage that will back the /var/lib/mysql data directory mounted by MariaDB.

Configuration

The simplest way to configure storage for your MariaDB is:

This will make use of the default StorageClass available in your cluster, but you can also provide a different one:

mariadb-enterprise-operator supports managing resources in external MariaDB instances i.e running outside of the Kubernetes cluster where the operator runs. This feature allows to manage users, privileges, databases, run SQL jobs declaratively and taking backups using the same CRs that you use to manage internal MariaDB instances.

ExternalMariaDB configuration

The ExternalMariaDB resource is similar to the internal MariaDB resource, but we need to provide a host

The following is a list of images that have plugins installed and available to use.

In this guide, we will be migrating existing MariaDB Galera and MaxScale instances to TLS without downtime.

1. Ensure that MariaDB has TLS enabled and not enforced. Set the following options if needed:

By setting these options, the operator will issue and configure certificates for MariaDB, but TLS will not be enforced in the connections i.e. both TLS and non-TLS connections will be accepted. TLS enforcement will be optionally configured at the end of the migration process.

This will trigger a rolling upgrade, make sure it finishes successfully before proceeding with the next step. Refer to the updates documentation for further information about update strategies.

2. If you are currently using MaxScale, it is important to note that, unlike MariaDB, it does not support TLS and non-TLS connections simultaneously (see ). For this reason, you must temporarily point your applications to MariaDB during the migration process. You can achieve this by configuring your application to use the . At the end of the MariaDB migration process, the MaxScale instance will need to be recreated in order to use TLS, and then you will be able to point your application back to MaxScale. Ensure that all applications are pointing to MariaDB before moving on to the next step.

3. MariaDB is now accepting TLS connections. The next step is by pointing them to MariaDB securely. Ensure that all applications are connecting to MariaDB via TLS before proceeding to the next step.

4. If you are currently using MaxScale, and you are planning to connect via TLS through it, you should now delete your MaxScale instance. If needed, keep a copy of the MaxScale manifest, as we will need to recreate it with TLS enabled in further steps:

It is very important that you wait until your old MaxScale instance is fully terminated to make sure that the old configuration is cleaned up by the operator.

5. For enhanced security, it is recommended to enforce TLS in all MariaDB connections by setting the following options. This will trigger a rolling upgrade, make sure it finishes successfully before proceeding with the next step:

6. For improved security, you can optionally configure TLS for Galera SSTs by following the steps below:

• Get the and grant execute permissions:

• Run the migration script. Make sure you set <mariadb-name> with the name of the MariaDB resource:

• Set the following option to enable TLS for Galera SSTs:

This will trigger a rolling upgrade, make sure it finishes successfully before proceeding with the next step

7. As mentioned in step 4, recreate your MaxScale instance with tls.enabled=true if needed:

8. MaxScale is now accepting TLS connections. Next, you need to by pointing them back to MaxScale securely. You have done this previously for MariaDB, you just need to update your application configuration to use the and its CA bundle.

In this guide, we will be migrating from the MariaDB Community Operator to the MariaDB Enterprise Kubernetes Operator without downtime. This guide assumes:

• 0.37.1 version of the MariaDB Community Operator is installed in the cluster.

• MariaDB community resources will be migrated to its counterpart MariaDB enterprise resource. In this case, we will be using 11.4.4 version, which is supported in both community and enterprise versions. Check the supported MariaDB Enterprise images and migrate to a counterpart community version first if needed.

• MaxScale resources cannot be migrated in a similar way, they need to be recreated. To avoid downtime, temporarily point your applications to MariaDB directly during the migration.

1. Install the Enterprise CRDs as described in the .

2. Get the and grant execute permissions:

3. Migrate MariaDB resources using the migration script. Make sure you set <mariadb-name> with the name of the MariaDB resource to be migrated and <operator-version> with the version of the Enterprise operator you will be installing:

4. Update the apiVersion of the rest of CRs to enterprise.mariadb.com/v1alpha1.

5. Uninstall the Community operator:

6. If your MariaDB Community had Galera enabled, delete the <mariadb-name> Role, as it will be specyfing the Community CRDs:

7. Install the Enterprise operator as described in the . This will trigger a rolling upgrade, make sure it finishes successfully before proceeding with the next step.

8. Delete the finalizers and uninstall the Community CRDs:

9. Run mariadb-upgrade in all Pods. Make sure you set <mariadb-name> with the name of the MariaDB resource:

10. Restart the Enterprise operator:

By leveraging the automation provided by MariaDB Enterprise Kubernetes Operator, you can declaratively manage large fleets of databases using CRs. This also covers day two operations, such as upgrades, which can be risky when rolling out updates to thousands of instances simultaneously.

To mitigate this, and to give you full control on the upgrade process, you are able to choose between multiple update strategies described in the following sections.

Update strategies

In order to provide you with flexibility for updating MariaDB reliably, this operator supports multiple update strategies:

• : Roll out replica Pods one by one, wait for each of them to become ready, and then proceed with the primary Pod.

• : Utilize the rolling update strategy from Kubernetes.

• : Updates are performed manually by deleting Pods.

• : Pause updates.

Configuration

The update strategy can be configured in the updateStrategy field of the MariaDB resource:

It defaults to ReplicasFirstPrimaryLast if not provided.

Trigger updates

Updates are not limited to updating the image field in the MariaDB resource, an update will be triggered whenever any field of the Pod template is changed. This translates into making changes to MariaDB fields that map directly or indirectly to the Pod template, for instance, the CPU and memory resources:

Once the update is triggered, the operator manages it differently based on the selected update strategy.

ReplicasFirstPrimaryLast

This role-aware update strategy consists in rolling out the replica Pods one by one first, waiting for each of them become ready (i.e. readiness probe passed), and then proceed with the primary Pod. This is the default update strategy, as it can potentially meet various reliability requirements and minimize the risks associated with updates:

• Write operations won't be affected until all the replica Pods have been rolled out. If something goes wrong in the update, such as an update to an incompatible MariaDB version, this is detected early when the replicas are being rolled out and the update operation will be paused at that point.

• Read operations impact is minimized by only rolling one replica Pod at a time.

• Waiting for every Pod to be synced minimizes the impact in the clustering protocols and the network.

RollingUpdate

This strategy leverages the rolling update strategy from the , which, unlike , does not take into account the role of the Pods(primary or replica). Instead, it rolls out the Pods one by one, from the highest to the lowest StatefulSet index.

You are able to pass extra parameters to this strategy via the rollingUpdate object:

OnDelete

This strategy aims to provide a method to update MariaDB resources manually by allowing the user to restart the Pods individually. This way, the user has full control over the update process and can decide which Pods are rolled out at any given time.

Whenever an , the MariaDB will be marked as pending to update:

From this point, you are able to delete the Pods to trigger the update, which will result the MariaDB marked as updating:

Once all the Pods have been rolled out, the MariaDB resource will be back to a ready state:

Never

The operator will not perform updates on the StatefulSet whenever this update strategy is configured. This could be useful in multiple scenarios:

• Progressive fleet upgrades: If you're managing large fleets of databases, you likely prefer to roll out updates progressively rather than simultaneously across all instances.

• Operator upgrades: When upgrading the operator, changes to the StatefulSet or the Pod template may occur from one version to another, which could trigger a rolling update of your MariaDB instances.

Data-plane updates

Highly available topologies rely on that run alongside MariaDB to enable the remote management of the database instances. These containers use the mariadb-enterprise-operator image, which can be automatically updated by the operator based on its image version:

By default, updateStrategy.autoUpdateDataPlane is false, which means that no automatic upgrades will be performed, but you can opt-in/opt-out from this feature at any point in time by updating this field. For instance, you may want to selectively enable updateStrategy.autoUpdateDataPlane in a subset of your MariaDB instances after the operator has been upgraded to a newer version, and then disable it once the upgrades are completed.

It is important to note that this feature is fully compatible with the strategy: no upgrades will happen when updateStrategy.autoUpdateDataPlane=true and updateStrategy.type=Never.

This documentation provides guidance on installing the MariaDB Enterprise Kubernetes Operator operator in OpenShift. This operator has been certified by Red Hat and it is available in the OpenShift console.

Operators are deployed into OpenShift with the Operator Lifecycle Manager (OLM), which facilitates the installation, updates, and overall management of their lifecycle.

Prerequisites

Configure your customer credentials as described in the documentation to be able to pull images.

The recommended way to configure credentials is to use the global pull secret provided by OpenShift, as described in this section. Alternatively, the operator bundle has a mariadb-enterprise imagePullSecret configured by default. This means that you can configure a Secret named mariadb-enterprise in same namespace where the operator will be installed in order to pull images from the MariaDB Enterprise registry.

PackageManifest

You can install the certified operator in OpenShift clusters that have the mariadb-enterprise-operator packagemanifest available. In order to check this, run the following command:

SecurityContextConstraints

Both the operator and the operand Pods run with the restricted-v2 SecurityContextConstraint, the most restrictive SCC in OpenShift in terms of container permissions. This implies that OpenShift automatically assigns a SecurityContext for the Pods with minimum permissions, for example:

You can read more about .

Installation in all namespaces

To install the operator watching resources on all namespaces, you need to create a Subscription object for mariadb-enterprise-operator using the stable channel in the openshift-operators namespace:

This will use the global-operators OperatorGroup that is created by default in the openshift-operators namespace. This OperatorGroup will watch all namespaces in the cluster, and the operator will be able to manage resources across all namespaces.

You can read more about .

Installation in specific namespaces

In order to define which namespaces the operator will be watching, you need to create an OperatorGroup in the namespace where the operator will be installed:

This OperatorGroup will watch the namespaces defined in the targetNamespaces field. The operator will be able to manage resources only in these namespaces.

Then, the operator can be installed by creating a Subscription object in the same namespace as the OperatorGroup:

Release channels

We maintain support across a variety of OpenShift channels to ensure compatibility with different release schedules and stability requirements. Below, you will find an overview of the specific OpenShift channels we support.

An example Subscription would look like this:

Updates

Updates are fully managed by OLM and controlled by the installPlanApproval field in the Subscription object. The default value is Automatic, which means that OLM will automatically update the operator to the latest version available in the channel. If you want to control the updates, you can set this field to Manual, and OLM will only update the operator when you approve the update.

Uninstalling

The first step for uninstalling the operator is to delete the Subscription object. This will not remove the operator, but it will stop OLM from managing the operator:

After that, you can uninstall the ClusterServiceVersion (CSV) object that was created by OLM. This will remove the operator from the cluster:

OpenShift console

As an alternative to create Subscription objects via the command line, you can install operators by using the OpenShift console. Go to the Operators > OperatorHub section and search by mariadb enterprise:

Select MariaDB Enterprise Kubernetes Operator, click on install, and you will be able to create a Subscription object via the UI.

Once deployed, the operator comes with example resources that can be deployed from the console directly. For instance, to create a MariaDB:

As you can see in the previous screenshot, the form view that the OpenShift console offers is limited, we recommend using the YAML view:

MariaDB Operator Enterprise enables you to manage SQL resources declaratively through CRs. By SQL resources, we refer to users, grants, and databases that are typically created using SQL statements.

The key advantage of this approach is that, unlike executing SQL statements manually, which is a one-time operation, declaring a SQL resource via a CR ensures that the resource is periodically reconciled by the operator. This provides a guarantee that the resource will be recreated if it gets manually deleted. Additionally, it prevents state drifts, as the operator will regularly update the resource according to the CR specification.

User CR

By creating this resource, you are declaring an intent to create an user in the referred MariaDB instance, just like a

This documentation aims to provide guidance on various configuration aspects shared across many MariaDB Enterprise Kubernetes Operator CRs.

my.cnf

An inline can be provisioned in the MariaDB resource via the myCnf field:

In this field, you may provide any or supported by MariaDB.

Under the hood, the operator automatically creates a ConfigMap with the contents of the myCnf field, which will be mounted in the MariaDB instance. Alternatively, you can manage your own configuration using a pre-existing ConfigMap by linking it via myCnfConfigMapKeyRef. It is important to note that the key in this ConfigMap i.e. the config file name, must have a .cnf extension in order to be detected by MariaDB:

To ensure your configuration changes take effect, the operator triggers a MariaDB update whenever the myCnf field or the ConfigMap is updated. For the operator to detect changes in a ConfigMap, it must be labeled with enterprise.mariadb.com/watch. Refer to the section for further detail.

Compute resources

CPU and memory resouces can be configured via the resources field in both the MariaDB and MaxScale CRs:

In the case of MariaDB, it is recommended to set the innodb_buffer_pool_size system variable to a value that is 70-80% of the available memory. This can be done via the :

Timezones

By default, MariaDB does not load timezone data on startup for performance reasons and defaults the timezone to SYSTEM, obtaining the timezone information from the environment where it runs. See the for further information.

You can explicitly configure a timezone in your MariaDB instance by setting the timeZone field:

This setting is immutable and implies loading the timezone data on startup.

In regards to Backup and SqlJob resources, which get reconciled into CronJobs, you can also define a timeZone associated with their cron expression:

If timeZone is not provided, the local timezone will be used, as described in the .

Passwords

Some CRs require passwords provided as Secret references to function properly. For instance, the root password for a MariaDB resource:

By default, fields like rootPasswordSecretKeyRef are optional and defaulted by the operator, resulting in random password generation if not provided:

You may choose to explicitly provide a Secret reference via rootPasswordSecretKeyRef and opt-out from random password generation by either not providing the generate field or setting it to false:

This way, we are telling the operator that we are expecting a Secret to be available eventually, enabling the use of GitOps tools to seed the password:

• : The Secret is reconciled from a SealedSecret, which is decrypted by the sealed-secrets controller.

• : The Secret is reconciled fom an ExternalSecret, which is read by the external-secrets controller from an external secrets source (Vault, AWS Secrets Manager ...).

External resources

Many CRs have a references to external resources (i.e. ConfigMap, Secret) not managed by the operator.

These external resources should be labeled with enterprise.mariadb.com/watch so the operator can watch them and perform reconciliations based on their changes. For example, see the my.cnf ConfigMap:

Probes

Kubernetes probes serve as an inversion of control mechanism, enabling the application to communicate its health status to Kubernetes. This enables Kubernetes to take appropriate actions when the application is unhealthy, such as restarting or stop sending traffic to Pods.

Fine tunning of probes for databases running in Kubernetes is critical, you may do so by tweaking the following fields:

There isn't an universally correct default value for these thresholds, so we recommend determining your own based on factors like the compute resources, network, storage, and other aspects of the environment where your MariaDB and MaxScale instances are running.

The Hashicorp Key Management Plugin is used to implement encryption using keys stored in the Hashicorp Vault KMS.

Configuring TDE in MariaDB Using Hashicorp Key Management Plugin

Transparent Data Encryption (TDE) can be configured in MariaDB leveraging the Hashicorp Key Management Plugin.

Requirements

• Running and accessible Vault KMS setup with a valid SSL certificate.

• Vault is unsealed and you've logged in to it with vault login $AUTH_TOKEN, where $AUTH_TOKEN is an authentication token given to you by an administrator

• openssl for generating secrets

Steps

1. Creating A New Key-Value Store In Vault. Create a new key-value store and take note of the path. In our example we will use mariadb.

2. Adding necessary secrets. We will put 2 secrets with ids 1 and 2. 2 will be used for temporary files, while 1 will be used for everything else. It is not neccessary to create 2 of them and in that case, temporary files will use 1

Day-2 Operations

Rotating Secrets

1. Put A New Secret In Vault. After logging in to vault, you can run again:

   This will start re-encrypting data.

2. Monitor Re-Encryption.

   If you check the encrpytion status again:

   You should see CURRENT_KEY_VERSION column start getting updated to point to the new key version.

Rotating Token

Make sure when rotating the token, to do so in advance of the token expiring.

1. Acquire a new token and update the secret.

2. Restart MariaDB Pods. MariaDB will continue using the old token until the Pods are restarted. You can add the following annotation to the Pods in order to trigger an update, see the for further detail:

Known Issues/Limitations

Vault Not Being Accessible Will Result In MariaDB Not Working

As MariaDB uses Vault to fetch it's decryption key, in case that Vault becomes unavailable, it will result in MariaDB not being able to fetch the decryption key and hence stop working. While the Hashicorp plugin has a configurable cache, that should be set and will result in MariaDB still working for a few seconds to minutes, depending on configuration, the cache is not reliable as it's ephemeral and short lived.

Deleting The Decryption Key Will Make Your Data Inaccessible.

It is recommended to back up the decryption key so accidental deletions will not result in issues.

Decryption Key Must Be Hexadecimal

Use the following to generate correct decryption keys.

Rotating The Decryption Key Before A Previous Re-Encryption Has Finished, Will Result In Data Corruption.

To check the re-encryption progress, you can run:

Look for the CURRENT_KEY_VERSION and make sure they are in sync with the latest version you have in Vault.

The MariaDB pam plugin facilitates user authentication by interfacing with the Pluggable Authentication Modules (PAM) framework, enabling diverse and centralized authentication schemes.

Currently the enterprise operator utilizes this plugin to provide support for:

• LDAP based authentication

LDAP

This guide outlines the process of configuring MariaDB to authenticate users against an LDAP or Active Directory service. The integration is achieved by using MariaDB's Pluggable Authentication Module (PAM) plugin, which delegates authentication requests to the underlying Linux PAM framework.

How Does It Work?

To enable LDAP authentication for MariaDB through PAM, several components work in tandem:

• PAM (Pluggable Authentication Modules): A framework used by Linux and other UNIX-like systems to consolidate authentication tasks. Applications like MariaDB can use PAM to authenticate users without needing to understand the underlying authentication mechanism. Operations such as system login, screen unlocking, and sudo access commonly use PAM.

• nss-pam-ldapd: This is the software package that provides the necessary bridge between PAM and an LDAP server. It includes the core components required for authentication.

• pam_ldap.so: A specific PAM module, provided by the nss-pam-ldapd package. This module is the "plug-in" that the PAM framework loads to handle authentication requests destined for an LDAP server.

What is needed for LDAP Auth?

nslcd is configured with 2 files. nslcd.conf which tells the daemon about the LDAP server and nsswitch.conf, determine the sources from which to obtain name-service information.

nslcd can be configured to run as a specific user based on the uid and gid props specified in the config file, however that user should have sufficient permissions to read/write to /var/run/nslcd, should own both nslcd.conf and nsswitch.conf and they should not be too open (0600).

Both of these configuration files will be attached later on in the example given.

nslcd.conf

The /etc/nslcd.conf is the configuration file for LDAP nameservice daemon.

In a production environment it is recommended to use LDAPS (LDAP secure), which uses traditional TLS encryption to secure data in transit. To do so, you need to add the following to your nslcd.conf file:

nsswitch.conf

The Name Service Switch (NSS) configuration file, located at /etc/nsswitch.conf. It is used by the GNU C Library and certain other applications to determine the sources from which to obtain name-service information in a range of categories, and in what order. Each category of information is identified by a database name.

Installing The PAM Plugin

The pam plugin is not enabled by default (even though it is installed). To enable it, you should add the following lines to your MariaDB Custom Resource:

See below for a complete example.

Combining It All Together

Fistly, we need to create our ConfigMaps and Secrets, that will store the nsswitch.conf, nslcd.conf and the mariadb pam module.

mariadb-nss-config.yaml:

kubectl apply -f mariadb-nss-config.yaml

Now that our configuration is done, we need to create the MariaDB custom resource along with needed configurations.

mariadb.yaml:

kubectl apply -f mariadb.yaml

And in the end we need to create our user in the database, which must have the same name as a user in ldap server. In the example below that's ldap-user. We also create mariadb-ldap secret, which holds the name of the plugin we are using as well as the module we need to load.

mariadb-user.yaml:

kubectl apply -f mariadb-user.yaml

After a few seconds, the user should have been created by the operator. To verify that all is working as expected, modify the <password> field below and run:

You should see something along the lines of:

LDAPS

If you followed the instructions for setting up a basic MariaDB instance with ldap, you need to fetch the public certificate that your LDAP server is set up with and add it to a called mariadb-ldap-tls.

If you have the certificate locally in a file called tls.crt you can run:

Known Issues

Slow Start On KIND

This may be a problem with the maximum number of file-handles a process can allocate. Some systems have this value set to really high, which causes an issue. To remedy this, you need to delete your kind cluster and run:

At this point, the problem should be fixed.

For more information, check .

What is a logical backup?

A logical backup is a backup that contains the logical structure of the database, such as tables, indexes, and data, rather than the physical storage format. It is created using mariadb-dump, which generates SQL statements that can be used to recreate the database schema and populate it with data.

Logical backups serve not just as a source of restoration, but also enable data mobility between MariaDB instances. These backups are called "logical" because they are independent from the MariaDB topology, as they only contain DDLs and INSERT statements to populate data.

Although logical backups are a great fit for data mobility and migrations, they are not as efficient as for large databases. For this reason, physical backups are the recommended method for backing up MariaDB databases, especially in production environments.

Storage types

Currently, the following storage types are supported:

• S3 compatible storage: Store backups in a S3 compatible storage, such as or .

• PVCs: Use the available in your Kubernetes cluster to provision a PVC dedicated to store the backup files.

• Kubernetes volumes: Use any of the supported natively by Kubernetes.

Our recommendation is to store the backups externally in a S3 compatible storage.

Backup CR

You can take a one-time backup of your MariaDB instance by declaring the following resource:

This will use the default StorageClass to provision a PVC that would hold the backup files, but ideally you should use a S3 compatible storage:

By providing the authentication details and the TLS configuration via references to Secret keys, this example will store the backups in a local Minio instance.

Alternatively you can use dynamic credentials from an EKS Service Account using EKS Pod Identity or IRSA:

By leaving out the accessKeyIdSecretKeyRef and secretAccessKeySecretKeyRef credentials and pointing to the correct serviceAccountName, the backup Job will use the dynamic credentials from EKS.

Scheduling

To minimize the Recovery Point Objective (RPO) and mitigate the risk of data loss, it is recommended to perform backups regularly. You can do so by providing a spec.schedule in your Backup resource:

This resource gets reconciled into a CronJob that periodically takes the backups.

It is important to note that regularly scheduled Backups complement very well the feature detailed below.

Retention policy

Given that the backups can consume a substantial amount of storage, it is crucial to define your retention policy by providing the spec.maxRetention field in your Backup resource:

Compression

You are able to compress backups by providing the compression algorithm you want to use in the spec.compression field:

Currently the following compression algorithms are supported:

• bzip2: Good compression ratio, but slower compression/decompression speed compared to gzip.

• gzip: Good compression/decompression speed, but worse compression ratio compared to bzip2.

• none: No compression.

compression is defaulted to none by the operator.

Restore CR

You can easily restore a Backup in your MariaDB instance by creating the following resource:

This will trigger a Job that will mount the same storage as the Backup and apply the dump to your MariaDB database.

Nevertheless, the Restore resource doesn't necessarily need to specify a spec.backupRef, you can point to other storage source that contains backup files, for example a S3 bucket:

Target recovery time

If you have multiple backups available, specially after configuring a , the operator is able to infer which backup to restore based on the spec.targetRecoveryTime field.

The operator will look for the closest backup available and utilize it to restore your MariaDB instance.

By default, spec.targetRecoveryTime will be set to the current time, which means that the latest available backup will be used.

Bootstrap new MariaDB instances

To minimize your Recovery Time Objective (RTO) and to switfly spin up new clusters from existing Backups, you can provide a Restore source directly in the MariaDB object via the spec.bootstrapFrom field:

As in the Restore resource, you don't strictly need to specify a reference to a Backup, you can provide other storage types that contain backup files:

Under the hood, the operator creates a Restore object just after the MariaDB resource becomes ready. The advantage of using spec.bootstrapFrom over a standalone Restore is that the MariaDB is bootstrap-aware and this will allow the operator to hold primary switchover/failover operations until the restoration is finished.

Backup and restore specific databases

By default, all the logical databases are backed up when a Backup is created, but you may also select specific databases by providing the databases field:

When it comes to restore, all the databases available in the backup will be restored, but you may also choose a single database to be restored via the database field available in the Restore resource:

There are a couple of points to consider here:

• The referred database (db1 in the example) must previously exist for the Restore to succeed.

• The mariadb CLI invoked by the operator under the hood only supports selecting a single database to restore via the option, restoration of multiple specific databases is not supported.

Extra options

Not all the flags supported by mariadb-dump and mariadb have their counterpart field in the Backup and Restore CRs respectively, but you may pass extra options by using the args field. For example, setting the --verbose flag can be helpful to track the progress of backup and restore operations:

Refer to the mariadb-dump and mariadb CLI options in the section.

Staging area

When using S3 storage for backups, a staging area is used for keeping the external backups while they are being processed. By default, this staging area is an emptyDir volume, which means that the backups are temporarily stored in the node's local storage where the Backup/Restore Job is scheduled. In production environments, large backups may lead to issues if the node doesn't have sufficient space, potentially causing the backup/restore process to fail.

To overcome this limitation, you are able to define your own staging area by setting the stagingStorage field to both the Backup and Restore CRs:

In the examples above, a PVC with the default StorageClass will be used as staging area. Refer to the for more configuration options.

Similarly, you may also use a custom staging area when :

Important considerations and limitations

Root credentials

When restoring a backup, the root credentials specified through the spec.rootPasswordSecretKeyRef field in the MariaDB resource must match the ones in the backup. These credentials are utilized by the liveness and readiness probes, and if they are invalid, the probes will fail, causing your MariaDB Pods to restart after the backup restoration.

Restore job

Restoring large backups can consume significant compute resources and may cause Restore Jobs to become stuck due to insufficient resources. To prevent this, you can define the compute resources allocated to the Job:

Galera backup limitations

mysql.global_priv

Galera only replicates the tables with InnoDB engine, see the .

Something that does not include mysql.global_priv, the table used to store users and grants, which uses the MyISAM engine. This basically means that a Galera instance with mysql.global_priv populated will not replicate this data to an empty Galera instance. However, DDL statements (CREATE USER, ALTER USER ...) will be replicated.

Taking this into account, if we think now about a restore scenario where:

• The backup file includes a DROP TABLE statement for the mysql.global_priv table.

• The backup has some INSERT statements for the mysql.global_priv table.

• The Galera cluster has 3 nodes: galera-0,

This is what will happen under the scenes while restoring the backup:

• The DROP TABLE statement is a DDL so it will be executed in galera-0, galera-1 and galera-2.

• The INSERT statements are not DDLs, so they will only be applied to galera-0.

After the backup is fully restored, the liveness and readiness probes will kick in, they will succeed in galera-0, but they will fail in galera-1 and galera-2, as they rely in the root credentials available in mysql.global_priv, resulting in the galera-1 and galera-2 getting restarted.

To address this issue, when backing up MariaDB instances with Galera enabled, the mysql.global_priv table will be excluded from backups by using the --ignore-table option with mariadb-dump. This prevents the replication of the DROP TABLE statement for the mysql.global_priv table. You can opt-out from this feature by setting spec.ignoreGlobalPriv=false in the Backup resource.

Also, to avoid situations where mysql.global_priv is unreplicated, all the entries in that table must be managed via DDLs. This is the recommended approach suggested in the . There are a couple of ways that we can guarantee this:

• Use the rootPasswordSecretKeyRef, username and passwordSecretKeyRef fields of the MariaDB CR to create the root and initial user respectively. This fields will be translated into DDLs by the image entrypoint.

• Rely on the and CRs to create additional users and grants. Refer to the for further detail.

LOCK TABLES

Galera is not compatible with the LOCK TABLES statement:

For this reason, the operator automatically adds the --skip-add-locks option to the Backup to overcome this limitation.

Migrations using logical backups

Migrating an external MariaDB to a MariaDB running in Kubernetes

You can leverage logical backups to bring your external MariaDB data into a new MariaDB instance running in Kubernetes. Follow this runbook for doing so:

1. Take a logical backup of your external MariaDB using one of the commands below:

1. Ensure that your backup file is named in the following format: backup.2024-08-26T12:24:34Z.sql. If the file name does not follow this format, it will be ignored by the operator.

2. Upload the backup file to one of the supported . We recommend using S3.

3. Create your MariaDB resource declaring that you want to and providing a that matches the backup:

1. If you are using Galera in your new instance, migrate your previous users and grants to use the User and Grant CRs. Refer to the for further detail.

Migrating to a MariaDB with different topology

Database mobility between MariaDB instances with different topologies is possible with logical backups. However, there are a couple of technical details that you need to be aware of in the following scenarios:

Migrating between standalone and replicated MariaDBs

This should be fully compatible, no issues have been detected.

Migrating from standalone/replicated to Galera MariaDBs

There are a couple of limitations regarding the backups in Galera, please make sure you read the section before proceeding.

To overcome this limitations, the Backup in the standalone/replicated instance needs to be taken with spec.ignoreGlobalPriv=true. In the following example, we are backing up a standalone MariaDB (single instance):

Once the previous Backup is completed, we will be able bootstrap a new Galera instance from it:

Reference

Troubleshooting

Galera Pods restarting after bootstrapping from a backup

Please make sure you understand the .

After doing so, ensure that your backup does not contain a DROP TABLE mysql.global_priv; statement, as it will make your liveness and readiness probes to fail after the backup restoration.

This section provides guidance on how to configure high availability in MariaDB and MaxScale instances. If you are looking for an HA setup for the operator, please refer to the .

Our recommended setup for production is:

• Use a for MariaDB:

  • with a primary node and at least 2 replicas.