ALTER CLUSTER SOFTFAIL

Overview

Modifies the Xpand deployment, configures the Rebalancer to move data off of the specified Xpand Nodes so that they can be removed from the deployment.

When using the Xpand Storage Engine topology, the details described here only apply when you connect to the Xpand nodes.

DETAILS

ALTER CLUSTER SOFTFAIL node-id[, node-id...]

This statement softfails Xpand Nodes. It configures the Rebalancer to begin moving replicas off of the designated nodes. Once the rebalancer process finishes, run the ALTER CLUSTER REFORM statement, which initiates a group change to remove the nodes.

Note

It is recommended that you run this operation on a node other than the one you want to remove.

Before softfailing Xpand Nodes, consider the following:

  • Xpand requires a minimum of three Xpand Nodes to support production systems.

  • When using zones, Xpand requires a minimum of three zones.

  • When using zones, Xpand requires an equal number of nodes per zone.

  • The target deployment must have sufficient disk space.

Softfailing Nodes

To softfail nodes:

  1. Connect to an existing Xpand Node with the socket file used by the MariaDB Xpand service:

    $ mariadb --socket /data/clustrix/mysql.sock
    
  2. Determine the node ID for the nodes you want to softfail:

    SELECT * FROM system.nodeinfo ORDER BY nodeid;
    
  3. Issue an ALTER CLUSTER SOFTFAIL statement to initiate the softfail.

    ALTER CLUSTER SOFTFAIL 1, 2, 3, 4;
    
  4. Monitor softfail process:

    Verify that the specified nodes are marked for removal:

    SELECT * FROM system.softfailed_nodes;
    

    Check containers marked to be moved as part of the softfail operation. When the query returns 0 rows, the data migration is complete:

    SELECT * FROM system.softfailing_containers;
    

    Check softfailed nodes that are ready for removal:

    SELECT * FROM system.softfailed_nodes
    WHERE nodeid NOT IN (
       SELECT DISTINCT nodeid FROM system.softfailing_containers
    );
    
  5. When the softfail process is complete, you can remove the nodes from the deployment using an ALTER CLUSTER REFORM statement:

    ALTER CLUSTER REFORM;
    

There is brief interruption of service while the Xpand node or nodes are removed from the deployment.