mysql-test overview
MariaDB uses mysql-test to test the functionality. It is all-in-one framework. It does unit, regression, and conformance testings. The framework was inherited from MySQL, but it is greatly enhanced, optimized, and extended in MariaDB.
The basics
At the core, it is very simple. The client program mysqltest
executes a test file and compares the produced output with the result file. If they match, the test is passed, otherwise it is failed. Tthis approach can be used to test any SQL statements, as well as other executables (with the exec
command).
The complete process of testing is governed and monitored by a mysql-test-run.pl driver script or mtr for short (for convenience, mtr
is created as a symbolic link to mysql-test-run.pl
). It is responsible for preparing the test environment, creating a list of all tests to run, running them, and producing the report at the end. It can run many tests in parallel, execute tests in the order that minimizes server restarts (as they are slow), run tests in a debugger or under valgrind or strace, and so on.
Test files are located in suites. A suite is a directory that contains test files, result files, and optional configuration files. The mtr loos for suites in the mysql-test/suite
directory, and in the mysql-test
subdirectories of plugins and storage engine directories. For example, these are valid suite paths:
mysql-test/suite/rpl
mysql-test/suite/handler
storage/example/mysql-test/demo
plugin/auth_pam/mysql-test/pam
In all cases, suite directory name is the suite name. A notable historical exception is the main suite, which is located directly in the mysql-test
Test files have .test
extension and can be placed directly in the suite directory (for example, mysql-test/suite/handler/interface.test
) or in the t
subdirectory (like mysql-test/suite/rpl/t/rpl_alter.test
or mysql-test/t/grant.test
). Similarly, result files have .result
extenstion and can be in the suite directory too, or in the r
subdirectory.
A test file can include other files (with the source
command). These include files can have any name and be placed anywhere, but customarily they have .inc
extension and located in the suite directory or in the inc
or include
subdirectory (for example, mysql-test/suite/handler/init.inc
or mysql-test/include/start_slave.inc
).
Other files that affect testing, while not being tests themselves, are:
disabled.def
suite.opt
- other
*.opt
files my.cnf
- other
*.cnf
files combinations
- other
*.combinations
files suite.pm
*.sh
files*.require
files*.rdiff
filesvalgrind.supp
See Auxiliary files for details.
Overlays
In addition to regular suite directories, mtr supports overlays. An overlay is a directory with the same name as an existing suite, but located in a storage engine or plugin directory. For example, a storage/myisam/mysql-test/rpl
could be a myisam overlay of the rpl suite in mysql-test/suite/rpl
. And plugin/daemon_example/mysql-test/demo
could be a daemon_example overlay of the demo suite in storage/example/mysql-test/demo
. As a special exception, an overlay of the main suite, should be called main
, as in storage/pbxt/mysql-test/main
.
An overlay is like a second transparent layer in a graphics editor. It can obscure, extend, or modify the background image. Also, one can notice that an overlay is very close to a UnionFS, but implemented in perl inside mtr.
An overlay can replace almost any file in the overlayed suite, or add new files. For example, if some overlay of the main suite contains include/have_innodb.inc
file, than all tests that include it will see and use the overlayed version. Or, it can create t/create.opt
file (even though the main suite does not have it), and create.test
will be executed with the specified additional options.
But adding an overlay never affects how the original suite is executed. That is, mtr always executes the original suite as if no overlay was present. And then, additionally, it executes a combined "union" of the overlay and the original suite. When doing that mtr takes care to avoid reexecuting tests that are not changed in the overlay. For example, creating t/create.opt
in the overlay of the main suite, will only cause create.test
to be executed in the overlay. But creating suite.opt
affects all tests - and it will cause all tests to be re-executed with the new options.
Combinations
Sample output
A typical mtr output looks like this
============================================================================== TEST WORKER RESULT TIME (ms) or COMMENT -------------------------------------------------------------------------- rpl.rpl_row_find_row_debug [ skipped ] Requires debug build main-pbxt.connect [ skipped ] No PBXT engine main-pbxt.mysqlbinlog_row [ disabled ] test expects a non-transactional engine rpl.rpl_savepoint 'mix,xtradb' w2 [ pass ] 238 rpl.rpl_stm_innodb 'innodb_plugin,row' w1 [ skipped ] Neither MIXED nor STATEMENT binlog format binlog.binlog_sf 'stmt' w2 [ pass ] 7 unit.dbug w2 [ pass ] 1 maria.small_blocksize w1 [ pass ] 23 sys_vars.autocommit_func3 'innodb_plugin' w1 [ pass ] 5 sys_vars.autocommit_func3 'xtradb' w1 [ pass ] 6 main.ipv6 w1 [ pass ] 131 ...
Every test is printed as "suitename.testname", and a suite name may include an overlay name (like in main-pbxt
). After the test name, mtr prints combinations that were applied to this test, if any.
A similar syntax can be used on the mtr command line to specify what tests to run. While it is currently not possible to specify what combinations to use, one can specify a test name and a suite name as follows:
$ ./mtr innodb
| search for innodb test in every suite from the default list, and run all that was found. |
$ ./mtr main.innodb
| run the innodb test from the main suite |
$ ./mtr main-pbxt.innodb
| run the innodb suite from the pbxt overlay of the main suite |
$ ./mtr main-.innodb
| run the innodb suite from the main suite and all its overlays. |
Plugin support
plugins (copy, --plugin-load, autodisable)
diffs - separate article
suite w/o tests - in aux files
important environment variables - in command-line option page