Rebuild a Replica Node with MariaDB Enterprise Server

Overview

Starting with MariaDB MaxScale 22.08, MariaDB Monitor (mariadbmon) can rebuild a replica server using MariaDB Enterprise Backup (mariadb-backup) to clone the contents of another server:

  • The rebuild operation is a monitor module command that can be launched with MaxCtrl

  • The rebuild operation replaces the contents of a database server with the contents of another server

  • MariaDB Monitor performs this operation by running MariaDB Enterprise Backup on both the source and target servers using SSH

Compatibility

MariaDB Enterprise Server

  • MariaDB Enterprise Server 10.3

  • MariaDB Enterprise Server 10.4

  • MariaDB Enterprise Server 10.5

  • MariaDB Enterprise Server 10.6

MariaDB MaxScale

  • MariaDB MaxScale 22.08

  • MariaDB MaxScale 23.02

Use Cases

  • Rebuild a primary or replica node that has crashed or failed

  • Rebuild a replica node that has diverged from the primary server

  • Build a new replica node to add to the topology

Example Usage

To rebuild a replica node, execute maxctrl call command mariadbmon async-rebuild-server:

maxctrl call command mariadbmon async-rebuild-server MONITOR_NAME TARGET_NODE [ SOURCE_NODE ]

  • Replace MONITOR_NAME with the name of the monitor.

  • Replace TARGET_NODE with the name of the target node.

  • Optionally, replace SOURCE_NODE with the name of the source node. If the source node is not specified, then a source node is auto-selected. When auto-selecting, the monitor prefers to pick an up-to-date replica server.

The source server is effectively cloned and all data on the target server is lost, but the target server's configuration files are not affected.

MariaDB Monitor calls MariaDB Enterprise Backup with the --safe-slave-backup option, which means that the replica node stops replicating until the backup is complete.

The operation does not launch if the target server is already replicating or if the source server is not a primary or replica node.

Install Dependencies

The following tools must be installed on the source and target nodes:

  • MariaDB Enterprise Backup: Backups and restores MariaDB Server contents.

  • pigz: Compresses and decompresses the backup stream.

  • socat: Streams data from one machine to another.

MaxScale Configuration

The following MaxScale parameters are used to configure the functionality:

System Configuration

MaxScale must be able to run commands as ssh_user using sudo on both the source and target servers.

The specified user account can be given access to use sudo by adding a sudoers configuration file in the /etc/sudoers.d directory:

REBUILD_USER ALL= NOPASSWD: /bin/systemctl stop mariadb
REBUILD_USER ALL= NOPASSWD: /bin/systemctl start mariadb
REBUILD_USER ALL= NOPASSWD: /usr/sbin/lsof
REBUILD_USER ALL= NOPASSWD: /bin/kill
REBUILD_USER ALL= NOPASSWD: /usr/bin/mariabackup
REBUILD_USER ALL= NOPASSWD: /bin/mbstream
REBUILD_USER ALL= NOPASSWD: /bin/du
REBUILD_USER ALL= NOPASSWD: /bin/rm -rf /var/lib/mysql/*
REBUILD_USER ALL= NOPASSWD: /bin/chown -R mysql\:mysql /var/lib/mysql/*
REBUILD_USER ALL= NOPASSWD: /bin/cat

Replace REBUILD_USER with the user account specified in ssh_user.

The paths may need to be adapted depending on the environment.

Authentication

MaxScale uses MariaDB Monitor's user parameter to run MariaDB Enterprise Backup. The user account might need additional privileges to perform the backup. MariaDB Enterprise Backup connects on the backup node via Unix socket, so you might need to create a separate USER@'localhost' user account if your user has host-based privileges.

For additional information about the required privileges, see "Creating the Backup User".