Failover With Multiple MaxScales
Compare the ways of coordinating failover between multiple MariaDB MaxScale instances. Walk through how an active/passive pair can diverge a cluster and lose transactions, and how cooperative locking
When two or more MaxScale instances monitor the same replication cluster, they must agree on which server is the primary. If they disagree, each instance routes writes to a different server. Replication then either breaks or, worse, silently accepts both write streams, and the cluster contents diverge. Divergence is not something MaxScale can undo: recovery means rebuilding a server by hand, and the transactions written to the losing server are lost.
This page compares the three ways of coordinating failover across MaxScale instances, walks through the failure mode of each, and covers the tuning that cooperative locking needs to be safe.
The examples use three servers, server1 to server3, with server1 as the initial primary, and two MaxScale instances, MaxScale A and MaxScale B. Both instances run MariaDB Monitor over the same three servers with auto_failover and auto_rejoin enabled.
This page assumes you are familiar with GTID-based replication, automatic failover, and MariaDB Monitor.
Comparing the Coordination Modes
Active/passive (passive)
Not at all. You choose which instance is allowed to fail over.
No
No
Not applicable
cooperative_monitoring_locks=majority_of_running
Locks on a majority of the servers each instance can currently reach
Yes
No
2
cooperative_monitoring_locks=majority_of_all
Locks on a majority of all configured servers
Yes
Yes, with semisynchronous replication
3
The short version: majority_of_running is a strict improvement on active/passive and any deployment using auto_failover and auto_rejoin with more than one MaxScale should prefer it. Choose majority_of_all when a network partition is a realistic risk, and pair it with semisynchronous replication. See Choosing a Mode.
Active/Passive Configuration
In an active/passive deployment, the global passive setting is false on one instance and true on every other instance. A passive MaxScale still monitors the cluster and still routes queries; it only refrains from performing failover, switchover, and rejoin. The administrator decides which instance is the active one.
[maxscale]
passive=false[maxscale]
passive=trueNothing in this arrangement makes the two instances agree on which server is the primary. Each one reaches its own conclusion from what it can see, and a failover is exactly the moment when the two views come apart.
How an Active/Passive Pair Diverges a Cluster
In the initial state both instances see server1 as the primary and route writes there.
Initial state: both instances agree that server1 is the primary.
Step 1 — server1 goes down. Both instances lose their connections to it.
Step 2 — the active MaxScale starts a failover. MaxScale B waits until server1 has been down for failcount monitor intervals, then begins promoting server2. Promotion is not instant: the new primary has to process its relay log first.
The active instance has started a failover, which takes some time to complete.
Step 3 — server1 comes back up while the failover is still running. When there are contending primary candidates, MariaDB Monitor keeps the server that was the primary before, so MaxScale A picks server1 again and resumes writing to it. MaxScale A has no way of knowing a failover is in progress, because nothing in an active/passive setup communicates that. MaxScale B does not notice server1 returning either, because it only reconnects to it once the failover finishes.
The cluster has diverged: writes through MaxScale A land on the old primary, writes through MaxScale B on the new one.
Step 4 — the failover completes. MaxScale B redirects server3 to replicate from server2 and marks server2 as the primary. Writes through MaxScale B now go to server2 while writes through MaxScale A still go to server1.
Step 5 — the old primary is demoted. MaxScale B connects to server1 and rejoins it to the cluster. With enforce_read_only_servers=true it also sets read_only on it, which makes server1 an invalid primary candidate from MaxScale A's point of view. Only then does MaxScale A switch to server2.
Aftermath: both instances agree again, but server1 holds transactions server2 never saw.
The end result is that server1 contains transactions that are not present in server2. Those transactions were acknowledged to the client and are now effectively lost, and recovery is a manual process.
The window in which this happens is the failover itself, so it is not rare: a primary that restarts, or a primary whose network drops out and returns, is enough to trigger it. Do not run more than one MaxScale over the same cluster with auto_failover unless the instances coordinate through cooperative locking.
Cooperative Locking With majority_of_running
Cooperative monitoring makes the MaxScale instances agree on both questions — who performs cluster operations and which server is the primary — by coordinating through the database itself rather than directly with each other.
Set the same monitor configuration on every instance:
How Cooperative Locking Works
Coordination uses GET_LOCK(), a user-level advisory lock, to take exclusive ownership of a server:
Each monitor tries to acquire a lock named
maxscale_mariadbmonitoron every server it can reach. The instance that holds a majority of these locks is the primary monitor; any other instance is a secondary monitor. Only the primary monitor performs failover, switchover, or rejoin.The primary monitor also takes a second lock,
maxscale_mariadbmonitor_master, on the server it has selected as the primary. This lock is how it publishes that decision.A secondary monitor only accepts a server as the primary if that server carries the
maxscale_mariadbmonitor_masterlock, held by the same connection that holds the server'smaxscale_mariadbmonitorlock. If no server carries the lock, a secondary monitor marks no server as [Master] and writes through it fail rather than landing on a stale primary. Replicas keep their [Slave] status under the defaultslave_conditions, so reads continue to work.When a MaxScale loses lock majority it releases every lock it holds, including the master lock, so that another instance can take over.
With majority_of_running, majority is counted over the servers the instance can currently reach and lock. In a three-server cluster with all three running, that is two locks.
cooperative_monitoring_locks is independent of passive. If passive=true, cluster operations stay disabled even when the monitor holds the locks. Do not mix the two: set passive=false or leave it unset.
The Same Failover, With Cooperative Locking
Take the sequence from the previous section again, with MaxScale B as the primary monitor holding the locks.
Step 1 — server1 goes down. Its locks disappear with it, including the master lock that marked it as the primary.
Step 2 — MaxScale B starts the failover after failcount monitor intervals and begins promoting server2.
Step 3 — server1 comes back up while the failover is still running. This time MaxScale A does not resume writing to it. The restart cleared the master lock, so from MaxScale A's point of view no server is the primary. Connections through MaxScale A can only read.
No server carries the master lock, so the old primary takes no writes and nothing diverges.
Step 4 — the failover completes. MaxScale B redirects server3, marks server2 as the primary, and takes the master lock on it. Writes through MaxScale B resume.
Step 5 — MaxScale A follows. It sees the master lock on server2 and starts routing writes there. server1 is rejoined as a replica.
Both instances end up on the new primary, and no acknowledged transaction was lost.
The cost of this protection is a short read-only window: between the moment the primary is lost and the moment the new primary is marked, no instance accepts writes. That is the trade for not diverging.
Network Partitions and Split Brain
A network partition happens when the network splits into subnetworks, whether from an outage or a degraded link. To the MaxScale instances it looks like connection errors and timeouts, which is indistinguishable from servers going down.
Start from a healthy cluster where both instances route writes to server1, then partition it so that MaxScale A, server1, and server2 end up on one side and MaxScale B and server3 on the other.
The partition: MaxScale A keeps the old primary, MaxScale B is left with a single replica.
What happens next depends entirely on the mode.
Active/Passive During a Partition
If the active instance lands in the minority partition, it promotes what it can see. MaxScale B promotes server3 and writes there, while the passive MaxScale A keeps writing to the old primary server1. The cluster has diverged, exactly as in the failover case.
When connectivity returns, MaxScale B tries to rejoin server1 and server2 under server3. Either they refuse to join because their GTID positions are incompatible, or — the worse outcome — they join successfully and the divergence becomes silent. Transactions are lost either way.
Active/passive after the partition heals: two write streams, one of which has to be discarded.
majority_of_running During a Partition
majority_of_running does not help here, because each instance counts majority only over the servers it can see, and each side of a partition can reach a local majority.
MaxScale A acquires the locks on server1 and server2, sees two running servers out of two, and considers itself the primary monitor. It still sees the old primary, so it keeps writing to server1. MaxScale B does the same on its side: it locks server3, counts one running server out of one, declares itself the primary monitor too, promotes server3, and starts writing there. Both instances believe they own the cluster.
Once connectivity returns, one instance ends up with a genuine lock majority and picks either server1 or server3 as the primary. The other side's writes then have to be reconciled, and the rejoin fails or silently diverges exactly as in the active/passive case.
majority_of_running protects against divergence when servers fail, not when the network splits. Two instances can each claim a local majority and both act as the primary monitor.
majority_of_all During a Partition
majority_of_all counts majority over all configured servers rather than only the reachable ones. In a three-server cluster that is always two locks, whether or not the third server is up. Only one side of a partition can reach that count, so only one side can act.
Old Primary in the Majority Partition
MaxScale A holds locks on two of three servers and therefore has majority. It keeps server1 as the primary and keeps writing there, safely. MaxScale B holds one lock out of the two it needs, so it releases its locks, marks no server as [Master], and allows only reads on server3.
Only the majority side is writable, so there is only ever one write stream.
When connectivity returns, MaxScale B sees the master lock on server1 and starts accepting writes again. Nothing diverged, because the writes that would have diverged were refused.
Old Primary in the Minority Partition
Now suppose the partition leaves server1 alone with MaxScale A while MaxScale B keeps server2 and server3.
MaxScale A briefly continues writing to server1 — it takes a monitor tick or two to establish that it can no longer reach a majority. Once it does, it releases its locks and stops marking any server as [Master], so writes through MaxScale A start failing.
On the other side, MaxScale B holds two of three locks, so it is the primary monitor. After failcount monitor intervals it promotes server2 and begins accepting writes there.
The minority side goes read-only; the majority side promotes a new primary.
When connectivity returns, MaxScale A sees the master lock on server2 and starts writing there.
That leaves one question. If writes were briefly allowed on server1, and server2 was promoted without those writes ever reaching it, how is consistency preserved?
It is preserved only if the cluster uses semisynchronous replication configured so that the primary never falls back to asynchronous replication — no transaction may commit without an acknowledgment from at least one other server. There is no infinite setting for rpl_semi_sync_master_timeout, so set it to its maximum value. At lower values (the default is 10 seconds), the primary reverts to asynchronous replication when the timeout expires, and transactions can then commit on the minority partition and are lost when the partition heals. Set up semisynchronous replication before relying on majority_of_all — see Failure-tolerant replication and failover.
Choosing a Mode
Do not use active/passive with
auto_failoverand more than one MaxScale. Any of the alternatives is safer, and switching costs nothing but a configuration change.Use
majority_of_runningas the default choice. It avoids divergence whenever the trouble is servers failing rather than the network splitting, and it works with as few as two servers, since majority is counted over what is running.Use
majority_of_allwhen a network partition is a realistic risk, such as MaxScale instances and servers spread across datacenters. It needs at least three servers to survive one server going down, and it needs semisynchronous replication to make its guarantee real. It also stops the cluster when too many servers are down at once: with three configured servers, two locks are always required, so the cluster goes read-only as soon as fewer than two servers are reachable, even though the surviving server could still serve traffic.
To check which instance is the primary monitor, run maxctrl show monitors and read the primary field. Per-server lock state is in the server-specific lock_held field.
Tuning failcount for Stale Locks
Cooperative locking works because the locks vanish when their holder does. That is immediate for a clean shutdown, where the monitor closes its connections and MariaDB Server releases the locks. It is not immediate when a MaxScale disappears into a network outage: its connections merely look idle, and the locks stay held until MariaDB Server closes them.
To bound that, the monitor sets the session wait_timeout on every connection where it holds a lock:
The value is rounded up to whole seconds, clamped to the range 5 to 28800 seconds, and logged when MaxScale starts.
A stale lock is a problem if the surviving MaxScale reaches the point of starting a failover while a vanished instance's locks are still held. To rule that out, the failover delay has to outlast wait_timeout. Since the monitor waits failcount * monitor_interval before failing over, that means failcount must be at least 1 + (2 * backend_timeout) / monitor_interval. Adding one monitor interval of margin for the tick that detects the situation gives the value to configure:
monitor_interval
backend_timeout
Resulting wait_timeout
Smallest safe failcount
Failover starts after
2s (default)
3s (default)
8s
5 (the default)
10s
5s
10s
25s
6
30s
The default settings are already safe. Check the arithmetic again whenever you raise backend_timeout or lower monitor_interval or failcount.
Do not confuse this with the worst-case failover delay estimate, (monitor_interval + backend_timeout) * failcount, on the failcount reference. That formula answers "how long before a failover starts?" for a given failcount. The formula here answers "how small can failcount be and still be safe?" when cooperative locking is enabled.
backend_connect_timeout is deprecated and is now an alias of backend_timeout. Use backend_timeout in new configurations.
See Also
Automatic Failover With MariaDB MonitorFailure-tolerant replication and failoverMariaDB MonitorThis page is licensed: CC BY-SA / Gnu FDL
Last updated
Was this helpful?