RQG Extensions for MariaDB Features
- Galera mode
- CheckFieldValue Validator
- MariadbGtidCrashSafety Reporter
- SlaveCrashRecovery Reporter
- BinlogConsistency Reporter
- CrashRecovery reporter
- LimitRowsExamined Transformer
- ShowExplain Validator
- Comparison of VIEW Algorithms
- Multiple redefining grammars
- New grammar keywords _basetable and _view
- See also:
Most of the components described below have been or will be merged into the Main RQG tree on Launchpad. Most recent changes, as well as patches which were not accepted to the main tree, can be found in our work tree.
While all described changes were made in order to test MariaDB code, many of them are applicable to other MySQL flavors as well.
A set of changes to support running RQG tests with multi-master Galera replication, implemented to test MariaDB Galera cluster.
The top-level script
runall-new.pl got a new option --galera, which takes a string value. The string can be a combination of 'm' or 's', where each symbol represents a Galera node. 'm' stands for 'master', and 's' stands for 'slave'.
The new module
GaleraMySQLd.pm implements cluster initialization and startup. It is only interested in the number of nodes.
Internals are adapted to the multi-master or mixed mode:
- instead of executing data generation on each started server, it is only executed on the first 'master';
- test flow is executed on each 'master' (but not on 'slaves');
- no result comparison is performed after each query, the flow runs independently, but at the end, if no other errors were encountered,
runall-new.plperforms dump comparison in the usual manner.
Also, a sample grammar and data template are provided in
conf/galera, they are slightly modified versions of conf/engines/engine_stress.yy and engine_stress.zz, adapted for Galera taking into account its limitations (e.g. no
GaleraMySQLd.pm has defaults for everything mandatory but
wsrep_provider. If you run the test with more than 1 node, you need to provide the value on the command line in the usual manner (
--mysqld=--wsrep-provider=...), otherwise the test fail with the corresponding error message.
So, if you provide
--galera=mms --mysqld=--wsrep-provider=<path to Galera library>
3 nodes will be started, and Galera replication will be set up between them; the data will be generated on the first node; the test flow will be executed on the first and second nodes; at the end of the test, data dump from each node will be taken and compared; node vardirs will be placed under
A grammar can set requirements on results of a query through a specifically formatted comment. If the validator finds a comment which matches the template, it performs the requested check. The validation is defined in the comment itself: it says which field in which row in the result set should be checked, and provides the condition (currently simple numeric comparisons:
It allows to do simple verification without implementing a special validator.
A query in the grammar file
SELECT COUNT(*), MAX(`pk`) FROM _table /* Validate 2 > 10 for row all*/;
The comment means "Check that the value in field 2 is greater than 10 for all rows" (for this query "all rows" is overkill, it could just as well be "row 1").
Assuming that there is a table with not more than 10 rows, the validator will produce the error:
ERROR: For query 'SELECT COUNT(*), MAX(`pk`) FROM `BB` /* Validate 2 > 10 for row 1*/' on row 1 result 10 does not meet the condition > 10 Full row:  : 1;  : 10;
The exit code for this failure is
Usage example: investigation of the sporadic failure MDEV-4578
- it restarts the slave with --skip-slave-start, and executes
CHANGE MASTER .. MASTER_USE_GTID=current_pos; STOP SLAVE;
So, regardless how the replication was set up initially, after the first crash/restart it will start using GTID for sure;
- while the slave is running, and also before a crash and after restart the reporter checks
gtid*variables and the contents of the system table
mysql.gtid_slave_posand verifies basic consistency of the data, e.g. watches that GTID seq_no constantly grows.
The reporter can be used to test crash-safety of replication.
It is a periodic reporter, every 30 seconds it kills the slave server using
SIGKILL, and immediately restarts it on the old data directory, with the same parameters as before. On server restart, the reporter checks that the server and the replication started all right.
The reporter itself does not check consistency of the data, but it can be used together with
It is supposed to be used with
runall-new.pl, so that the server is started without MTR involvement.
Usage example: testing of GTID in MariaDB 10.0
The reporter checks that the contents of the binary log correctly reflects the contents of the server.
After the main test flow is finished, the reporter creates a data dump of the server, stores its binary logs, shuts down the server, starts a new one with the same parameters, but on a clean data directory, replays the binary logs into it, creates a dump of the new server, and compares two dumps.
It is to be used with
The idea is very much the same as in the old Recovery reporter: crash the server at the end of the test, restart it and make sure it started all right, and the data is not corrupted. The main difference is that the old reporter restarts the server in a hardcoded pre-defined manner, which limits its use. Instead, CrashRecovery reporter starts the server with the same set of options as the initial one, which imitates a more realistic scenario, and also allows to use it on non-default InnoDB configurations.
It is to be used with
The transformer checks whether the original query already contains a
ROWS EXAMINED clause. If it does not, it adds the clause either after the
LIMIT clause, or at the end of the query. In any case (even if
ROWS EXAMINED was already there), the transformer returns the following sequence of statements:
- <the query with
- <a query which sums up status variables related to examined rows>
The result of the main query is checked to be a subset of the original query's result set. The sum of status variables is checked to be not greater than the limit provided in the
ROWS EXAMINED clause, plus a margin. The margin is configured in the transformer.
If the result of the transformed query appears not to be a subset of the original result set,
STATUS_LENGTH_MISMATCH is returned.
If the sum of status variables is greater than the maximum allowed value,
STATUS_REQUIREMENT_UNMET is returned.
Note: Status values
STATUS_REQUIREMENT_UNMET_SELECT were added to
The validator checks that the output of
SHOW EXPLAIN correlates with the output of traditional
EXPLAIN executed for the same query. It also tries to filter out known expected mismatches between the produced plans.
Comparison of VIEW Algorithms
RQG already provided
--views[=<view type>] option, which means that in addition to the normal data generation, views of the requested types will be added and used in the test flow. However, you could only create views of the same type on servers that you were comparing. Now there are
--views2 options, which work the same way as
--mysqld2, etc. The backward compatibility is preserved, so you can still use
--views which will be applied to both servers unless there is
--views2 to override it.
The change was made in
gentest.pl and both
Multiple redefining grammars
RQG allows to provide
--redefine=<grammar file> option to override or add rules to the main grammar. Sometimes you have a redefining grammar for a feature, and another redefining grammar for an engine, and yet another redefining grammar for a specific version, and combining them makes the number of grammars to grow uncontrollably. The simple change has been made to allow providing several redefining grammars at once, e.g.:
runall.pl ... --redefine=memory_engine_redefine.yy --redefine=binlog_format_stmt_redefine.yy --redefine=legacy_version_redefine.yy
Warning: It is better not to allow them to override each other, since the result can be unexpected (that is, they should define different sets of rules)
The change was made in
gentest.pl and both
New grammar keywords _basetable and _view
The RQG keyword _table means any table or view from the test dataset. Often it's good enough, but sometimes it badly increases the number of erroneous statements. Now tables and views can be differentiated in a grammar depending on what the goal is: _table can still be used for both tables and views, while _basetable will only pick tables, and _view will only pick views.
If you create the following rule in your grammar
alter: ALTER TABLE _table ENGINE=MyISAM;
and you are running the test with views, you will be getting lots of
ER_WRONG_OBJECT errors ("is not BASE TABLE"). It is hardly the goal, so you can improve the grammar by changing it to
alter: ALTER TABLE _basetable ENGINE=MyISAM;
This way the rule will only pick "real" tables. Of course, if you still want to pick views sometimes, as a sanity check that there is no crash or anything, you can make the rule a bit more complicated, but much better tuned than it was, e.g.
alter: ALTER TABLE alter_object ENGINE=MyISAM; alter_object: _basetable | _basetable | _basetable | _basetable | _basetable | _view ;