MariaDB Galera Cluster - Known Limitations

This article contains information on known problems and limitations of MariaDB Galera Cluster.

Limitations from codership.com:

• Currently replication works only with the InnoDB storage engine. Any writes to tables of other types, including system (mysql.*) tables are not replicated (this limitation excludes DDL statements which implicitly modify mysql.* tables — those are replicated).

• LOCK/UNLOCK TABLES cannot be supported in master-master replication. Lock functions (GET_LOCK(), RELEASE_LOCK(),…) are not supported.

• DELETE operations are unsupported on tables without a primary key. Also, rows in tables without a primary key may appear in a different order on different nodes. Don't use tables without a primary key.

• The Query log cannot be directed to a table. If you enable query logging, you must forward the log to a file: log_output=FILE

• XA transactions can not be supported.

• Transaction size. While Galera does not explicitly limit the transaction size, a writeset is processed as a single memory-resident buffer and as a result, extremely large transactions (e.g. LOAD DATA) may adversely affect node performance. To avoid that wsrep_max_ws_rows and wsrep_max_ws_size variables limit transaction rows to 128K and transaction size to 1Gb by default. If necessary user may want to increase those limits. Future versions will add support for transaction fragmentation.

Limitations from other sources:

• The Query cache should not be used with Galera Cluster (source: Codership Google Group)

Other observations, in no particular order:

• If you are using mysqldump for state transfer, and it failed for whatever reason (e.g. you do not have the database account it attempts to connect with, or it does not have necessary permissions), you will see an SQL SYNTAX error in the server error log. Don't let it fool you, this is just a fancy way to deliver a message (the pseudo-statement inside of the bogus SQL will actually contain the error message).

• MDEV-400: If you are running a debug server, and a node was not able to connect to the cluster on startup, it might crash with a weird error 'debugger aborting because missing DBUG_RETURN or DBUG_VOID_RETURN macro in function "?func"'. Do not pay attention to the crash, focus on the fact that it _failed to connect_, this is the problem you need to resolve.

• MDEV-421: Do not use binlog-do-db and binlog-ignore-db — it is only supported for DML, but not for DDL, so it will quickly create a discrepancy and replication will abort.

• MDEV-443: Do not use character_set_server utf16 or utf32 or ucs2, at least if you choose rsync as a state transfer method. The server will crash.

• Do not use transactions of any essential size. Just to insert 100K rows, the server might require additional 200-300 Mb. In a less fortunate scenario it can be 1.5 Gb for 500K rows, or 3.5 Gb for 1M rows. See MDEV-466 for some numbers (you'll see that it's closed, but it's not closed because it was fixed).

• Locking is lax when DDL is involved. For example, if your DML transaction uses a table, and a parallel DDL statement is started, in the normal MySQL setup it would have waited for the metadata lock, but in Galera context it will be executed right away. It happens even if you are running a single node, as long as you configured it as a cluster node. See also MDEV-468. This behavior might cause various side-effects, the consequences have not been investigated yet. Try to avoid such parallelism.

• Do not rely on auto-increment values to be sequential. Galera uses a mechanism based on autoincrement increment to produce unique non-conflicting sequences, so on every single node the sequence will have gaps. See http://codership.blogspot.com/2009/02/managing-auto-increments-with-multi.html

• Every now and then, a node might start producing 'Unknown command' errors on pretty much every statement except for SHOW and maybe some other service commands. It happens when a cluster is suspected to be split, and the node is in a smaller part — for example, during a network glitch, when nodes temporarily lose each other. The error is scary, but it does not mean the server got crazy, it only means it thinks it might be. It was also noticed to happen when a node transfers state to another node.

• After a temporary split, if the 'good' part of the cluster was still reachable and its state was modified, resynchronization occurs. As a part of it, nodes of the 'bad' part of the cluster drop all client connections. It might be quite unexpected, especially if the client was idle and did not even know anything wrong was happening. Please also note that after the connection to the isolated node is restored, if there is a flow on the node, it takes a long time for it to synchronize, during which the "good" node says that the cluster is already of the normal size and synced, while the rejoining node says it's only joined (but not synced). The connections keep getting 'unknown command'. It should pass eventually.

• While binlog_format is checked on startup and can only be ROW, it can be changed at runtime. Do NOT change binlog_format at runtime, it is likely not only cause replication failure, but make all other nodes crash.

• If you are using rsync for state transfer, and a node crashes before the state transfer is over, rsync process might hang forever, occupying the port and not allowing to restart the node. The problem will show up as 'port in use' in the server error log. Find the orphan rsync process and kill it manually.

• Performance: by design performance of the cluster cannot be higher than performance of the slowest node; however, even if you have only one node, its performance can be considerably lower comparing to running the same server in a standalone mode (without wsrep provider). It is particularly true for big enough transactions (even those which are well within current limitations on transaction size quoted above).

• Windows is not supported.