ALTER CLUSTER ZONE
Modifies an Xpand deployment, grouping the given nodes into zones.
When using the Xpand Storage Engine topology, the details described here only apply when you connect to the Xpand nodes.
ALTER CLUSTER node-id[, node-id...] ZONE zone
Xpand allows you to group nodes into zones to improve fault tolerance, where a zone can be availability zones within the same AWS Region, different server racks, or separate servers in different data centers. Once you have determined your target zone configuration, use
ALTER CLUSTER ZONE to assign the given nodes to a zone.
When Xpand is configured for zones, the Rebalancer becomes zone-aware and ensures replicas are placed in different zones. Additionally, Paxos acceptors are distributed across multiple zones. The result is that Xpand is fault tolerant in the face of unexpected zone failure, regardless of how many nodes are in the zone.
After all nodes are assigned to a zone, verify that there is an equal number of nodes in each zone and that no nodes are assigned to zone
0. Xpand supports configuring a minimum of 3 and a maximum of 5 zones.
To list Xpand Nodes by Zone:
SELECT * FROM system.nodeinfo ORDER BY zone;
Changes to zone configurations do not take effect until a group change occurs or an ALTER CLUSTER REFORM statement is run. Reforming the deployment causes a group change and an interruption in service.
If you no longer wish to use zones, assign all nodes to zone
ALTER CLUSTER ZONE followed by ALTER CLUSTER REFORM.
MariaDB recommends allocating enough disk space so that, if a zone should fail, there is sufficient space remaining to re-protect.
Use of zones with Xpand has the following prerequisites:
Xpand supports configurations with a minimum of 3 and a maximum of 5 zones.
There must be a sufficient number of zones to maintain a quorum of nodes following a failure
Zones must be in the same geographical region with <2ms network latency between each zone.
A deployment deployed across zones must have the same number of nodes in each zone.
To configure zones, use
ALTER CLUSTER ZONE to assign nodes to a zone. Zone ids must be greater than 0 and there must be an equal number of nodes in each zone. For example, a 9 node deployment deployed across 3 zones should have 3 nodes each in zones 1, 2, and 3.
Xpand recommends provisioning enough free disk space to re-protect the data after a zone failure.
Xpand currently supports a minimum of 3 and a maximum of 5 zones.
Creating Deployments in Multiple Zones
To create a new deployment, follow these steps:
Deploy a sufficient number of Xpand nodes to ensure that the target configuration has an equal number of nodes in each zone.
If you are using encrypted connections, ensure that certificates in keys are present in the same location with the correct permissions.
Connect to an existing Xpand Node with the socket file used by the MariaDB Xpand service:
$ mariadb --socket /data/clustrix/mysql.sock
Use ALTER CLUSTER ADD to build a deployment.
ALTER CLUSTER ADD 192.0.2.1, 192.0.2.2, 192.0.2.3, 192.0.2.4, 192.0.2.5, 192.0.2.6, 192.0.2.7, 192.0.2.8, 192.0.2.9;
system.nodeinfotable to determine the node ID's for each node:
SELECT * FROM system.nodeinfo;
Use ALTER CLUSTER ZONE to assign nodes to zones.
ALTER CLUSTER 1, 2, 3 ZONE 1; ALTER CLUSTER 4, 5, 6 ZONE 2; ALTER CLUSTER 7, 8, 9 ZONE 3;
Reform the deployment to initiate a group change, applying the changes:
ALTER CLUSTER REFORM;
Again, query for node information, ensure that none of the Xpand Nodes is in zone
SELECT * FROM system.nodeinfo;
Migrating an Existing Cluster to Multiple Zones
If you would like to migrate your existing deployment, the high-level steps are:
Temporarily disable the Rebalancer.
Provision nodes in new zones and install Xpand on each node.
Use ALTER CLUSTER ADD procedure to add the new nodes to the deployment.
Configure zones using ALTER CLUSTER ZONE.
Use ALTER CLUSTER SOFTFAIL procedure to remove surplus nodes from the deployment.
Re-enable the Rebalancer.
Once this procedure is complete, you should have the same number of nodes in each zone and all nodes should have a non-zero zone assigned. Because the scale-out and scale-in operations can be time consuming, it is often simpler to migrate to a new deployment when moving from default to zone configurations.
Fault Tolerance and Zones
The following examples illustrate how zones work with the default
MAX_FAILURES = 1.
There must be the same number of nodes in each zone. MariaDB recommends using a 9 node deployment across 3 zones.
This deployment can tolerate one node failure:
Or, one zone failure:
There must be the same number of nodes in each zone. The following configuration is not supported:
If you require a larger deployment, add the same number of nodes to each zone:
See MAX_FAILURES for additional information.