Selected Revision
Revision
9137
User
sergii
Date
2012-03-03 20:02
Compare To
Revision
9134
User
sergii
Date
2012-03-03 19:52
Additions Deletions

Answer Diff [476]

<<toc>> ¶

In mysql-test framework besides test anf result files, there are many other files that affect the testing process. ¶

=== ##disabled.def## file ¶

This file can be used to disable certain tests temporarily. For example, if one test fails and you are working on that, you may want to push the changeset that disables it in the test suite so that others won't be disturbed by this failure. ¶

File contains test names and the comment (that should explain why the test was disabled) separated by a colon. Lines that start from a hash sign (##{{#}}##) are ignored. A typical ##disabled.def## may look like this (note that a hash sign in the middle of a line does not start a comment): ¶
<<code>> ¶
# List of disabled tests ¶
# test name : comment ¶
rpl_redirect : Fails due to bug#49978 ¶
events_time_zone : need to fix the timing ¶
<</code>> ¶

During testing, mtr will print disabled tests like this: ¶
<<code>> ¶
... ¶
rpl.rpl_redirect [ disabled ] Fails due to bug#49978 ¶
rpl.events_time_zone [ disabled ] need to fix the timing ¶
... ¶
<</code>> ¶

This file should be located in the suite directory. ¶

---- ¶
=== ##suite.opt## file ¶

This file lists server options that will be added to the ##mysqld## command line for every test of this suite. It can refer to environment variables with the ##$NAME## syntax. Shell meta-characters should be quoted. For example ¶

<<code>> ¶
--plugin-load=$AUTH_PAM_SO ¶
--max-connections=40 --net_read_timeout=5 ¶
"--replicate-rewrite-db=test->rewrite" ¶
<</code>> ¶

Note that options may be put on one line or or different lines. It is a good idea to start an option name with ##--loose-## prefix if the server may or may not recognize it depending on the configuration. An unknown option in the ##.opt## file will not let server to start and the test will be aborted. ¶

This file should be located in the suite directory. ¶

---- ¶
=== other ##*.opt## files ¶

For every test or include file ##somefile.test## or ##somefile.inc## mtr will look for ##somefile.opt##, ##somefile-master.opt## and ##somefile-slave.opt##. These files have exactly the same syntax as the ##suite.opt## above. Options from these files will also be added to the server command line (all servers started for this test, only master, or only slave respectively) for all affected tests, for example, for all tests that include ##somefile.inc## directly or indirectly. ¶

A typical usage example is, for example, ##include/have_blackhole.inc## and ##include/have_blackhole.opt##. The latter contains the necessary command line options to load the Blackhole storage engine, while the former verifies that it was really loaded. And any test that needs Blackhole engine, only needs to start from ##source include/have_blackhole.inc;## and the engine will be automatically loaded for it. ¶

---- ¶
=== ##my.cnf## file ¶

This is not the ##my.cnf## file that tests from this suite will use, but rather a //template// of it. It will be converted later to an actual ##my.cnf##. If a suite contains no ##my.cnf## template, a default one <<entity>>mdash<</entity>> ##include/default_my.cnf## <<entity>>mdash<</entity>> will be used. Or ##suite/rpl/my.cnf## if the test includes ##master-slave.inc## (it's one of the few bits of the old MySQL mysql-test-run magic that we did not remove yet). Typically a suite template will not contain a complete server configuration, but rather start from ¶
<<code>> ¶
!include include/default_my.cnf ¶
<</code>> ¶
and then add the necessary modifications. ¶

The syntax of ##my.cnf## template is the same of a normal ##my.cnf## file, with a few extensions and assumptions. They are: ¶

* For any group with the name ##[mysqld.**N**]##, where **N** is a number, mtr will start one ##mysqld## process. Usually one needs to have only ##[mysqld.1]## group, and ##[mysqld.2]## group for replication tests. ¶

* There can be groups with non-standard names (##[foo]##, ##[bar]##, whatever), not used by ##mysqld##. The ##suite.pm## files (see below) may use them somehow. ¶

* Values can refer to each other using the syntax ##@groupname.optionname## <<entity>>mdash<</entity>> these references be expanded as needed. For example ¶
<<code>> ¶
[mysqld.2] ¶
master-port= @mysqld.1.port ¶
<</code>> ¶
*none it sets the value of the ##master-port## in the ##[mysqld.2]## group to the value of ##port## in the ##[mysqld.1]## group. ¶

* An option name may start from a hash sign ##{{#}}##. In the resulting ##my.cnf## it will look like a comment, but it still can be referred to. For example: ¶
<<code>> ¶
[example] ¶
#location = localhost:@mysqld.1.port ¶
bar = server:@example.#location/data ¶
<</code>> ¶

* There is the ##[ENV]## group. It sets values for the environment variables. For example ¶
<<code>> ¶
[ENV] ¶
MASTER_MYPORT = @mysqld.1.port ¶
<</code>> ¶
*none Also, one can refer to values of environment variables via this group: ¶
<<code>> ¶
[mysqld.1] ¶
user = @ENV.LOGNAME ¶
<</code>> ¶

* There is the ##[OPT]## group. It is used to allow invoke functions and generate values. Currently it contains only one option <<entity>>mdash<</entity>> ##@OPT.port##. Every time this option is referred to in some other group in the ##my.cnf## template, a new unique port number is generated. It will not match any other port number used by this test run. For example ¶
<<code>> ¶
[ENV] ¶
SPHINXSEARCH_PORT = @OPT.port ¶
<</code>> ¶

This file should be located in the suite directory. ¶

---- ¶
=== other ##*.cnf## files ¶

For every test file ##somefile.test## (but for not included files) mtr will look for ##somefile.cnf## file. If such a file exists, it will be used as a template instead of suite ##my.cnf## or a default ##include/default_my.cnf## templates. ¶

---- ¶
=== ##combinations## file ¶

The ##combinations## file defines few sets of alternative configurations, and every test in this suite will be run many times - once for every configuration. This can be used, for example, to run all replication tests in the //rpl// suite for all three binlog format modes (row, statement, and mixed). A corresponding ##combinations## file would look as following ¶
<<code>> ¶
[row] ¶
binlog-format=row ¶

[stmt] ¶
binlog-format=statement ¶

[mix] ¶
binlog-format=mixed ¶
<</code>> ¶

It uses ##my.cnf## file syntax, with groups (where group names define combination names) and options. But, despite the similarity, it is not a ##my.cnf## template, and it cannot use the templating extentions. Instead, options from the ##combinations## file are added to the server command line. In this regard, combination file is closer to ##suite.opt## file. And just like it, combination file can use environment variables using the ##$NAME## syntax. ¶

Not all tests will necessarily run for all combinations. A particular test may require to be run only in one specific combination. For example, in replication, if a test can only be run with the row binlog format, it will have ##--binlog-format=row## in one of the ##.opt## files. In this case, mtr will notice that server command line already has an option that matches one of the combinations, and will skip all other combinations for this particular test. ¶

The ##combinations## file should be located in the suite directory. ¶

---- ¶
=== other ##*.combinations## files ¶

Just like with the ##*.opt## files, mtr will use ##somefile.combinations## file for any ##somefile.test## and ##somefile.inc## that is used in testing. These files have exactly the same format as a suite ##combinations## file. ¶

This can cause many combination files affecting one test file (if a test includes two ##.inc## files, and both of them have corresponding ##.combinations## files). In this case, mtr will run the test for all combinations of combinations from both files. In MariaDB 5.5, for example, ##rpl_init.inc## adds combinations for row/statement/mixed, and ##have_innodb.inc## adds combinations for innodb/xtradb. Thus any replication test that uses innodb will be run six times. ¶

---- ¶
=== ##suite.pm## file ¶

This (optional) file is a perl module. It must declare a ¶
package that inherits from ##My::Suite##. ¶

This file must normally end with ##bless {}## <<entity>>mdash<</entity>> that is it must return an object of that class. It can also return a string <<entity>>mdash<</entity>> in this case all tests in the suite ¶
will be skipped, with this string being printed as a reason (for example "PBXT engine was not compiled"). ¶

A suite class can define the following methods: ¶
* ##config_files()## ¶
* ##servers()## ¶
* ##list_cases()## ¶
* ##start_test()## ¶
* ##skip_combinations()## ¶

A ##config_files()## method returns a list of additional config files (besides ##my.cnf##), that this suite needs to be created. For every file it specifies a function that will create it, when given a ##My::Config## object. For example: ¶
<<code lang=perl>> ¶
sub config_files {( ¶
'config.ini' => \&write_ini, ¶
'new.conf' => \&do_new ¶
)} ¶
<</code>> ¶

A ##servers()## method returns a list of processes that needs to be started for ¶
this suite. A process is specified as a [regex, hash] pair. The regular expression must match a section in the ##my.cnf## template (for example, ##qr/mysqld\./## corresponds to all ##mysqld## processes), the hash contains these options: ¶

<<style class="borders">> ¶
| ##SORT## |a number. Processes are started in the order of increasing ##SORT## values (and stopped in the reverse order). ##mysqld## has number 300. ¶
| ##START## |a function to start a process. It takes two arguments, ##My::Config::Group## and ##My::Test##. If ##START## is undefined a process will not be started. ¶
| ##WAIT## |a function to wait for the process to be started. It takes ##My::Config::Group## as an argument. Internally mtr first invokes ##START## for all processes, then ##WAIT## for all started processes. ¶
<</style>> ¶
<<code lang=perl>> ¶
sub servers {( ¶
qr/^foo$/ => { SORT => 200, # start foo before mysqld ¶
START => \&start_foo, ¶
WAIT => \&wait_foo } ¶
)} ¶
<</code>> ¶

See the //sphinx// suite for a working example. ¶

A ##list_cases()## method returns a complete list of tests for this suite. By default it will be the list of files that have ##.test## extension, but without the extension. This list will be filtered by mtr, subject to different mtr options (##--big-test##, ##--start-from##, etc), the suite object does not have to do it. ¶

A ##start_test()## method starts one test process, by default it will be ##mysqltest##. ¶
See the //unit// suite for a working example of ##list_cases()## and ##start_test()## methods. ¶

A ##skip_combinations()## method returns a hash that maps file names (where combinations are defined) to a list of combinations that should be skipped. As a special case, it can disable a complete file by using a string instead of a hash. For example ¶

<<code lang=perl>> ¶
sub skip_combinations {( ¶
'combinations' => [ 'mix', 'rpl' ], ¶
'inc/many.combinations' => [ 'a', 'bb', 'c' ], ¶
'windows.inc' => "Not on windows", ¶
)} ¶
<</code>> ¶

The last line will cause all tests of this suite that include ##windows.inc## to be skipped with the reason being "Not on windows". ¶

---- ¶
=== ##*.sh## files ¶

For every test file ##sometest.test## mtr looks for ##sometest-master.sh## and ##sometest-slave.sh##. If either of these files is found, it will be run before the test itself. ¶

---- ¶
=== ##*.require## files ¶

These files are obsolete. Do not use them anymore. If you need to skip a test use the ##skip## command instead. ¶

---- ¶
=== ##*.rdiff## files ¶

These files also define what the test result should be. But unlike ##*.result## files, they contain a patch that should be applied to one result file to create a new result file. This is very useful when a result of some test in one combination differs slightly from the result of the same test, but in another combination. Or when a result of a test in an overlay differs from the test result in the overlayed suite. ¶

Because a combination can be part of the ##.result## or ##.rdiff## file name, mtr has to look in many different placIt is quite difficult to edit ##.rdiff## files to update them after the test file has changed. But luckily, it is never needed. When a test for a test result. For example, conseails, mtr creates a ##.reject## file. Having it, one can create ##.rdiff## file as easy as (for example) ¶
<<code>> ¶
diff -u r/foo.result r/foo.reject > r/foo,comb.rdiff ¶
<</code>> ¶

Because a combination can be part of the ##.result## or ##.rdiff## file name, mtr has to look in many different places for a test result. For example, consi
der a test ##foobar.test## in the combination pair ##aa,bb##, that is run in the overlay //rty// of the suite //qwe//, in other words, for the test that mtr prints as ¶
<<code>> ¶
qwe-rty.foobar 'aa,bb' [ pass ] ¶
<</code>> ¶
For this test a result can be in ¶
* either ##.rdiff## or ##.result## file ¶
* either in the overlay "##rty/##" or in the overlayed suite "##qwe/##" ¶
* with or without combinations in the file name ("##,a##", "##,b##", "##,a,b##", or nothing) ¶

which means any of the following 15 file names can be used: ¶

# ##rty/r/foo,aa,bb.result## ¶
# ##rty/r/foo,aa,bb.rdiff## ¶
# ##qwe/r/foo,aa,bb.result## ¶
# ##qwe/r/foo,aa,bb.rdiff## ¶
# ##rty/r/foo,aa.result## ¶
# ##rty/r/foo,aa.rdiff## ¶
# ##qwe/r/foo,aa.result## ¶
# ##qwe/r/foo,aa.rdiff## ¶
# ##rty/r/foo,bb.result## ¶
# ##rty/r/foo,bb.rdiff## ¶
# ##qwe/r/foo,bb.result## ¶
# ##qwe/r/foo,bb.rdiff## ¶
# ##rty/r/foo.result## ¶
# ##rty/r/foo.rdiff## ¶
# ##qwe/r/foo.result## ¶

They are listed, precisely, in the order of preference, and mtr will walk that list from top to bottom and the first file that is found will be used. ¶

If this found file is a ##.rdiff##, mtr continues walking down the list until the first ##.result## file is found. A ##.rdiff## is applied to that ##.result##. ¶

---- ¶
=== ##valgrind.supp## file ¶

This file defines valgrind suppressions, and it is used when mtr is started with a ##--valgrind## option. ¶