Scale-Out with MariaDB Xpand

MariaDB Xpand can be scaled out, which increases the number of Xpand nodes in the cluster.

The system variables, system tables, and ALTER CLUSTER statements referenced on this page are only available on the Xpand nodes. When using the Xpand Storage Engine topology, you need to connect to an Xpand node to use these features.

Use Cases

On occasion, you may need to expand your deployment's capacity:

  • To prepare for a peak event (i.e., Cyber-Monday).

  • To improve performance.

Note

Your MariaDB Xpand is licensed for a maximum number of cores per node as well as a maximum number of nodes for the deployment. Contact MariaDB Corporation Sales to if you need an expanded license.

Scale-Out

  1. Provision the node and deploy Xpand in the desired topology on each host, using the same version and installation configuration.

    • If zones are in use, ensure that the target configuration has an equal number of node in each zone.

    • If you are using encrypted connections, ensure that certificates in keys are present in the same location with the correct permissions.

  2. Add nodes to your deployment.

    If zones are in use, the newly added nodes must be assigned to their respective zone using ALTER CLUSTER ZONE. Clusters that are only partially configured for zones are not fault tolerant in the face of a zone loss.

  3. Verify successful node addition:

    • Run this query from a SQL prompt. New node(s) are automatically assigned consecutive node number(s).

      SELECT * FROM system.nodeinfo ORDER BY instance_id;
      
    • You can also use the clx Command-Line Administration Tool. View your deployment's status by providing the following at a bash prompt

      $ /opt/clustrix/bin/clx status
      

      You should see that all nodes appear OK on the display. If zones are in use, all nodes should have a non-zero zone assigned. If zones are not in use, all nodes will be assigned to zone 0.

      You may notice that the data distribution among your nodes is not yet balanced. Be patient. It will be, soon.

      If the node you are trying to add does not appear in the list above, see section below on Errors during Scale-Out.

  4. Monitor Rebalancer Activity

    Your new node(s) have been successfully added to your Xpand deployment but they do not yet contain data. The Rebalancer will now automatically work in the background to move data onto the new node(s).

Errors During Scale-Out

As part of adding nodes to your deployment, Xpand performs some checks to ensure the nodes have the same configuration. This section describes errors that can be encountered with ALTER CLUSTER ADD and how to resolve those issues.

SQL Errors

The following are errors you may encounter at the SQL prompt:

26633

ERROR 1 (HY000): [26633] Bad parameter.: Host "xxx" already in cluster

This error occurs if you attempt to add an node that is already part of the deployment.

55300

ERROR 1 (HY000): [55300] Multiple nodes cannot be added when an existing addition is pending

This error occurs when:

  • An node addition could be in a pending state due to:

  • An node is still in the process of being added

  • An node running a different version of Xpand was added to the deployment

  • An node with a different multi-port setting was added to the deployment

PENDING_INVITES

List of nodes with pending invitations:

SELECT * FROM system.pending_invites;

The deployment periodically attempts to send invitations to nodes in system.pending_invites. For each invitation that is sent, there will be entries in clustrix.log:

sending invitation response(no error) to "10.2.13.68:24378"

Note

If the same node is in system.pending_invites and system.problem_nodes, you may want to remove the node from system.pending_invites before resolving the issue with system.problem_nodes. Doing this will prevent the node from being automatically added to the deployment once the problem is resolved, causing a premature group change. Instead, you may prefer to complete the node addition during off-peak hours.

Remove a pending node addition from PENDING_INVITES

DELETE FROM system.pending_invites;

PROBLEM_NODES

Query system.problem_nodes to see why an node could not be added

SELECT * FROM system.problem_nodes;

Here is the list of reasons provided in system.problem_nodes and how to resolve those issues:

Reason

Resolution

Software binaries differ

Make sure all nodes are running the same version of Xpand

Multiport settings mismatched

  • Network Security with Xpand contains information relative to enabling Multiport ports.