Scale-Out with MariaDB Xpand 6.1

Step 1: Check Nodes per Zone

Ensure that the target configuration has an equal number of nodes in each zone.

Step 2: Connect

Connect to a node that is already in the Xpand cluster.

For additional information, see "Connect and Query".

Step 3: Add Nodes to the Cluster

Add the new nodes to the cluster by executing the ALTER CLUSTER ADD statement and specifying the IP addresses of the new nodes:

ALTER CLUSTER ADD "192.0.2.1", "192.0.2.3", "192.0.2.3";

Step 4: Add Nodes to Zones

Add the new nodes to their respective zones by executing the ALTER CLUSTER ZONE statement:

ALTER CLUSTER nodeid [, nodeid] ... ZONE zone;

All nodes must have a valid zone (non-zero) assigned. Clusters that are only partially configured for zones are not fault tolerant in the face of a zone loss.

Step 5: Verify Node States

While new nodes are added as part of an online group change, the rest of the cluster is online and available during the duration of the group change. The Rebalancer copies data to the new nodes, and the new nodes are fully able to service transactions, queries, and hold data. However, the new nodes are marked with the "late" sub-state and do not participate in some internal operations (such as the sequence manager and lock manager). The lockman_max_locks and lockman_max_transaction_locks system variables apply to non-late nodes only, and late nodes do not manage any locks.

Check the state of the new nodes using one of the following methods:

• Join the system.nodeinfo and system.membership system tables:

  SELECT
    n.nodeid, n.hostname, n.zone,
    m.status, m.substate, m.late
  FROM
    system.nodeinfo n,
    system.membership m
  WHERE m.nid = n.nodeid;
  

• Execute the clx status sub-command:

  $ /opt/clustrix/bin/clx status
  

Regardless of the method used, the output should show the following:

• The new node(s) should automatically be assigned consecutive node number(s).

• Nodes that are late should appear with a "late" sub-state.

• All nodes should be listed as OK.

• Since zones are in use, all nodes should have a valid zone (non-zero) assigned.

If the node you are trying to add does not appear in the list above, see "SQL Errors During Scale-Out".

The data distribution might not be balanced among your nodes. We will check the data distribution in the next step.

Step 6: Monitor the Rebalancer

Your new node(s) have been successfully added to your Xpand cluster, but they do not yet contain data. The Rebalancer will now automatically work in the background to move data onto the new node(s).

For information about how to monitor the Rebalancer, see "Manage the Rebalancer for MariaDB Xpand".

Step 7: Perform Full Group Change

When you want to perform a full group change, run ALTER CLUSTER REFORM:

ALTER CLUSTER REFORM;

This action can be deferred to non-peak hours. In the meantime, if your cluster has an HA configuration in place prior to scaling out, with the requisite number of nodes, zones, and protected data, your cluster will continue to be fully protected and fault tolerant.

Step 1: Connect

Connect to a node that is already in the Xpand cluster.

For additional information, see "Connect and Query".

Step 2: Add Nodes to the Cluster

Add the new nodes to the cluster by executing the ALTER CLUSTER ADD statement and specifying the IP addresses of the new nodes:

ALTER CLUSTER ADD "192.0.2.1", "192.0.2.3", "192.0.2.3";

By default, Xpand will add the nodes with an online group change. However, adding nodes can cause a full group change in the following scenarios:

• ALTER CLUSTER ADD .. COORDINATE is executed

• The number of nodes being added is greater than or equal to the number of nodes currently in the cluster

In the second scenario listed above, the full group change can be avoided by adding nodes in smaller batches.

Step 3: Verify Node States

While new nodes are added as part of an online group change, the rest of the cluster is online and available during the duration of the group change. The Rebalancer copies data to the new nodes, and the new nodes are fully able to service transactions, queries, and hold data. However, the new nodes are marked with the "late" sub-state and do not participate in some internal operations (such as the sequence manager and lock manager). The lockman_max_locks and lockman_max_transaction_locks system variables apply to non-late nodes only, and late nodes do not manage any locks.

Check the state of the new nodes using one of the following methods:

• Join the system.nodeinfo and system.membership system tables:

  SELECT
    n.nodeid, n.hostname, n.zone,
    m.status, m.substate, m.late
  FROM
    system.nodeinfo n,
    system.membership m
  WHERE m.nid = n.nodeid;
  

• Execute the clx status sub-command:

  $ /opt/clustrix/bin/clx status
  

Regardless of the method used, the output should show the following:

• The new node(s) should automatically be assigned consecutive node number(s).

• Nodes that are late should appear with a "late" sub-state.

• All nodes should be listed as OK.

• Since zones are not in use, all nodes should be assigned to zone 0, which means that no zone is assigned.

If the node you are trying to add does not appear in the list above, see "SQL Errors During Scale-Out".

The data distribution might not be balanced among your nodes. We will check the data distribution in the next step.

Step 4: Monitor the Rebalancer

Your new node(s) have been successfully added to your Xpand cluster, but they do not yet contain data. The Rebalancer will now automatically work in the background to move data onto the new node(s).

For information about how to monitor the Rebalancer, see "Manage the Rebalancer for MariaDB Xpand".

26633

ERROR 1 (HY000): [26633] Bad parameter.: Host "xxx" already in cluster

This error occurs if you attempt to add an node that is already part of the cluster.

55300

ERROR 1 (HY000): [55300] Multiple nodes cannot be added when an existing addition is pending

This error occurs when a node addition is in a pending state due to:

• An node is still in the process of being added

• An node running a different version of Xpand was added to the cluster

• An node with a different multi-port setting was added to the cluster

PENDING_INVITES

The system.pending_invites system table contains a list of nodes with pending invitations:

SELECT * FROM system.pending_invites;

The cluster periodically attempts to send invitations to nodes in system.pending_invites. For each invitation that is sent, there will be entries in clustrix.log:

sending invitation response(no error) to "10.2.13.68:24378"

Remove a pending node addition from PENDING_INVITES

DELETE FROM system.pending_invites;

PROBLEM_NODES

The system.problem_nodes system table shows why a node could not be added:

SELECT * FROM system.problem_nodes;

Here is the list of reasons provided in system.problem_nodes and how to resolve those issues: