Selected Revision
Revision
9129
User
sergii
Date
2012-03-02 21:20
Compare To
Revision
9128
User
sergii
Date
2012-03-02 21:20
Additions Deletions

Excerpt Diff [195]

At the core, it is very simple. The client program mysqltest executes a test file and compares the produced output with the result file. If they match, the test is passed, otherwise it is failed.

Answer Diff [9]

<<toc>> ¶
=== The basics ¶

At the core, it is very simple. The client program ##[[mysqltest]]## executes a //test file// and compares the produced output with the //result file//. If they match, the test is passed, otherwise it is failed. Tthis approach can be used to test any SQL statements, as well as other executables (with the ##exec## command). ¶

The complete process of testing is governed and monitored by a //mysql-test-run.pl// driver script or //mtr// for short (for convenience, ##mtr## is created as a symbolic link to ##mysql-test-run.pl##). It is responsible for preparing the test environment, creating a list of all tests to run, running them, and producing the report at the end. It can run many tests in parallel, execute tests in the order that minimizes server restarts (as they are slow), run tests in a debugger or under valgrind or strace, and so on. ¶

Test files are located in //suites//. A suite is a directory that contains test files, result files, and optional configuration files. The mtr loos for suites in the ##mysql-test/suite## directory, and in the ##mysql-test## subdirectories of plugins and storage engine directories. For example, these are valid suite paths: ¶
* ##mysql-test/suite/rpl## ¶
* ##mysql-test/suite/handler## ¶
* ##storage/example/mysql-test/demo## ¶
* ##plugin/auth_pam/mysql-test/pam## ¶

In all cases, suite directory name is the suite name. A notable historical exception is the //main// suite, which is located directly in the ##mysql-test## ¶

Test files have ##.test## extension and can be placed directly in the suite directory (for example, ##mysql-test/suite/handler/interface.test##) or in the ##t## subdirectory (like ##mysql-test/suite/rpl/t/rpl_alter.test## or ##mysql-test/t/grant.test##). Similarly, result files have ##.result## extenstion and can be in the suite directory too, or in the ##r## subdirectory. ¶

A test file can include other files (with the ##source## command). These include files can have any name and be placed anywhere, but customarily they have ##.inc## extension and located in the suite directory or in the ##inc## or ##include## subdirectory (for example, ##mysql-test/suite/handler/init.inc## or ##mysql-test/include/start_slave.inc##). ¶

Other files that affect testing, while not being tests themselves, are: ¶
* ##disabled.def## ¶
* ##suite.opt## ¶
* other ##*.opt## files ¶
* ##my.cnf## ¶
* other ##*.cnf## files ¶
* ##combinations## ¶
* other ##*.combinations## files ¶
* ##suite.pm## ¶
* ##*.sh## files ¶
* ##*.require## files ¶
* ##*.rdiff## files ¶
* ##valgrind.supp## ¶

See [[mtr-auxiliary-files|Auxiliary files]] for details. ¶

=== Overlays ¶

In addition to regular suite directories, mtr supports //overlays//. An overlay is a directory with the same name as an existing suite, but located in a storage engine or plugin directory. For example, a ##storage/myisam/mysql-test/rpl## could be a //myisam// overlay of the //rpl// suite in ##mysql-test/suite/rpl##. And ##plugin/daemon_example/mysql-test/demo## could be a //daemon_example// overlay of the //demo// suite in ##storage/example/mysql-test/demo##. As a special exception, an overlay of the main suite, should be called ##main##, as in ##storage/pbxt/mysql-test/main##. ¶

An overlay is like a second transparent layer in a graphics editor. It can obscure, extend, or modify the background image. Also, one can notice that an overlay is very close to a //UnionFS//, but implemented in perl inside mtr. ¶

An overlay can replace almost any file in the overlayed suite, or add new files. For example, if some overlay of the main suite contains ##include/have_innodb.inc## file, than all tests that include it will see and use the overlayed version. Or, it can create ##t/create.opt## file (even though the main suite does not have it), and ##create.test## will be executed with the specified additional options. ¶

But adding an overlay never affects how the original suite is executed. That is, mtr always executes the original suite as if no overlay was present. And then, additionally, it executes a combined "union" of the overlay and the original suite. When doing that mtr takes care to avoid reexecuting tests that are not changed in the overlay. For example, creating ##t/create.opt## in the overlay of the main suite, will only cause ##create.test## to be executed in the overlay. But creating ##suite.opt## affects all tests - and it will cause all tests to be re-executed with the new options. ¶

=== Combinations ¶

In certain cases it make sense to run a specific test or a group of tests several times with different server settings. It can be done using so called //combinations//. Combinations are groups of settings that are used alternatively. A combinations file defines these alternatives using ##my.cnf## syntax, for example ¶
<<code>> ¶
[row] ¶
binlog-format=row ¶

[stmt] ¶
binlog-format=statement ¶

[mix] ¶
binlog-format=mixed ¶
<</code>> ¶
And all tests where this combinations file applies will be run three times: once for the combination called "row", and ##--binlog-format=row## on the server command line, once for the "stmt" combination, and once for the "mix" combination. ¶

More than one combinations file may be applicable to a given test file. In this case, mtr will run the test for all possible combinations of combinations. A test that uses replication (three combinations as above) and innodb (two combinations - innodb and xtradb), will be run six times. ¶

=== Sample output ¶

A typical mtr output looks like this ¶

<<code>> ¶
============================================================================== ¶
TEST WORKER RESULT TIME (ms) or COMMENT ¶
-------------------------------------------------------------------------- ¶
rpl.rpl_row_find_row_debug [ skipped ] Requires debug build ¶
main-pbxt.connect [ skipped ] No PBXT engine ¶
main-pbxt.mysqlbinlog_row [ disabled ] test expects a non-transactional engine ¶
rpl.rpl_savepoint 'mix,xtradb' w2 [ pass ] 238 ¶
rpl.rpl_stm_innodb 'innodb_plugin,row' w1 [ skipped ] Neither MIXED nor STATEMENT binlog format ¶
binlog.binlog_sf 'stmt' w2 [ pass ] 7 ¶
unit.dbug w2 [ pass ] 1 ¶
maria.small_blocksize w1 [ pass ] 23 ¶
sys_vars.autocommit_func3 'innodb_plugin' w1 [ pass ] 5 ¶
sys_vars.autocommit_func3 'xtradb' w1 [ pass ] 6 ¶
main.ipv6 w1 [ pass ] 131 ¶
... ¶
<</code>> ¶

Every test is printed as "suitename.testname", and a suite name may include an overlay name (like in ##main-pbxt##). After the test name, mtr prints combinations that were applied to this test, if any. ¶

A similar syntax can be used on the mtr command line to specify what tests to run. While it is currently not possible to specify what combinations to use, one can specify a test name and a suite name as follows: ¶

<<style class="borders">> ¶
| ##$ ./mtr innodb## | search for //innodb// test in every suite from the default list, and run all that was found. ¶
| ##$ ./mtr main.innodb## | run the //innodb// test from the //main// suite ¶
| ##$ ./mtr main-pbxt.innodb## | run the //innodb// suite from the //pbxt// overlay of the //main// suite ¶
| ##$ ./mtr main-.innodb## | run the //innodb// suite from the //main// suite and all its overlays. ¶
<</style>> ¶

=== Plugin support ¶

The mtr driver has special support for MariaDB plugins. ¶

First, on startup it copies or symlinks all dynamically build plugins into ##var/plugins##. This allows to have many plugins loaded at the same time. For example, one can load Federated and InnoDB engines together. Also, mtr creates environment variables for every plugin with the corresponding plugin name. For example, if InnoDB engine was built, ##$HA_INNODB_SO## will be set to ##ha_innodb.so## or ##ha_innodb.dll## on Windows. And the test can safely use the corresponding environment variable on all platforms to refer to a plugin file, it will always have the correct platform dependent extension. ¶

Second, when combining server command line options (that may come from many different sources) into one long list before starting ##mysqld##, mtr treats ##--plugin-load## specially. Normal server semantics is to use the latest value of any particular option on the command line. If one starts the server with, for example, ##--port=2000 --port=3000##, the server will use the last value for the port, that is 3000. To allow different ##.opt## files to require different plugins, mtr goes through the assembled server command line, and joins all ##--plugin-load## options into one. Additionally it removes all emtpy ##--plugin-load## options. For example, if some test is affected by three ##.opt## files, that contain, respectively ¶
<<code>> ¶
--plugin-load=$HA_INNODB_SO ¶
<</code>> ¶
<<code>> ¶
--plugin-load=$AUTH_PAM_SO ¶
<</code>> ¶
<<code>> ¶
--plugin-load=$HA_EXAMPLE_SO ¶
<</code>> ¶
and, let's assume the Example engine was not built (##$HA_EXAMPLE_SO## is ¶
empty). Then the server will get ¶
<<code>> ¶
--plugin-load=ha_innodb.so:auth_pam.so ¶
<</code>> ¶
instead of ¶
<<code>> ¶
--plugin-load=ha_innodb.so --plugin-load=auth_pam.so --plugin-load= ¶
<</code>> ¶

Third, to allow plugin sources to be simply copied into the ##plugin/## or ##storage/## directories, and still not affect existing tests (even if new plugins are statically linked into the server), mtr automatically disables all optional plugins on server startup. A plugin is optional, if it can be disabled with the corresponding ##--skip-XXX## server command line option. Mandatory plugins, like MyISAM or MEMORY, do not have ##--skip-XXX## options (like, there is no ##--skip-myisam## option). This mtr behavior means that no plugin, statically or dynamically built, has any effect on the server, unless it was explicitly enabled. A convenient way to enable a plugin XXX for specific tests is to create a ##have_XXX.opt## file that contains necessary command-line options, and ##have_XXX.inc## file that checks whether a plugin was loaded. Then any test that needs this plugin can source the ##have_XXX.inc## file and have the plugin loaded automatically. ¶