ALTER CLUSTER ZONE

Overview

Modifies an Xpand deployment, grouping the given nodes into zones.

When using the Xpand Storage Engine topology, the details described here only apply when you connect to the Xpand nodes.

DETAILS

ALTER CLUSTER node-id[, node-id...] ZONE zone

Xpand allows you to group nodes into zones to improve fault tolerance, where a zone can be availability zones within the same AWS Region, different server racks, or separate servers in different data centers. Once you have determined your target zone configuration, use ALTER CLUSTER ZONE to assign the given nodes to a zone.

When Xpand is configured for zones, the Rebalancer becomes zone-aware and ensures replicas are placed in different zones. Additionally, Paxos acceptors are distributed across multiple zones. The result is that Xpand is fault tolerant in the face of unexpected zone failure, regardless of how many nodes are in the zone.

After all nodes are assigned to a zone, verify that there is an equal number of nodes in each zone and that no nodes are assigned to zone 0. Xpand supports configuring a minimum of 3 and a maximum of 5 zones.

To list Xpand Nodes by Zone:

SELECT * FROM system.nodeinfo ORDER BY zone;

Changes to zone configurations do not take effect until a group change occurs or an ALTER CLUSTER REFORM statement is run. Reforming the deployment causes a group change and an interruption in service.

If you no longer wish to use zones, assign all nodes to zone 0 using ALTER CLUSTER ZONE followed by ALTER CLUSTER REFORM.

MariaDB recommends allocating enough disk space so that, if a zone should fail, there is sufficient space remaining to re-protect.

Prerequisites

Use of zones with Xpand has the following prerequisites:

  • Xpand supports configurations with a minimum of 3 and a maximum of 5 zones.

  • There must be a sufficient number of zones to maintain a quorum of nodes following a failure

  • Zones must be in the same geographical region with <2ms network latency between each zone.

  • A deployment deployed across zones must have the same number of nodes in each zone.

Configuring Zones

To configure zones, use ALTER CLUSTER ZONE to assign nodes to a zone. Zone ids must be greater than 0 and there must be an equal number of nodes in each zone. For example, a 9 node deployment deployed across 3 zones should have 3 nodes each in zones 1, 2, and 3.

Xpand recommends provisioning enough free disk space to re-protect the data after a zone failure.

Note

Xpand currently supports a minimum of 3 and a maximum of 5 zones.

Creating Deployments in Multiple Zones

To create a new deployment, follow these steps:

  1. Deploy a sufficient number of Xpand nodes to ensure that the target configuration has an equal number of nodes in each zone.

    If you are using encrypted connections, ensure that certificates in keys are present in the same location with the correct permissions.

  2. Connect to an existing Xpand Node with the socket file used by the MariaDB Xpand service:

    $ mariadb --socket /data/clustrix/mysql.sock
    
  3. Use ALTER CLUSTER ADD to build a deployment.

    ALTER CLUSTER ADD 192.0.2.1, 192.0.2.2, 192.0.2.3,
       192.0.2.4, 192.0.2.5, 192.0.2.6, 192.0.2.7,
       192.0.2.8, 192.0.2.9;
    
  4. Query the system.nodeinfo table to determine the node ID's for each node:

    SELECT * FROM system.nodeinfo;
    
  5. Use ALTER CLUSTER ZONE to assign nodes to zones.

    ALTER CLUSTER 1, 2, 3 ZONE 1;
    ALTER CLUSTER 4, 5, 6 ZONE 2;
    ALTER CLUSTER 7, 8, 9 ZONE 3;
    
  6. Reform the deployment to initiate a group change, applying the changes:

    ALTER CLUSTER REFORM;
    
  7. Again, query for node information, ensure that none of the Xpand Nodes is in zone 0:

    SELECT * FROM system.nodeinfo;
    

Migrating an Existing Cluster to Multiple Zones

If you would like to migrate your existing deployment, the high-level steps are:

  1. Temporarily disable the Rebalancer.

  2. Provision nodes in new zones and install Xpand on each node.

  3. Use ALTER CLUSTER ADD procedure to add the new nodes to the deployment.

  4. Configure zones using ALTER CLUSTER ZONE.

  5. Use ALTER CLUSTER SOFTFAIL procedure to remove surplus nodes from the deployment.

  6. Re-enable the Rebalancer.

Once this procedure is complete, you should have the same number of nodes in each zone and all nodes should have a non-zero zone assigned. Because the scale-out and scale-in operations can be time consuming, it is often simpler to migrate to a new deployment when moving from default to zone configurations.

Fault Tolerance and Zones

The following examples illustrate how zones work with the default MAX_FAILURES = 1.

There must be the same number of nodes in each zone. MariaDB recommends using a 9 node deployment across 3 zones.

Xpand in 3 Zones

This deployment can tolerate one node failure:

1 Node Failure in Zoned Cluster

Or, one zone failure:

1 Full Zone Failure

There must be the same number of nodes in each zone. The following configuration is not supported:

Unequal nodes per Zone

If you require a larger deployment, add the same number of nodes to each zone:

Expanded Zones

See MAX_FAILURES for additional information.