Debugging MariaDB with mysql-test-run

Checking that MariaDB is compiled for debugging

Excute:

mysqld --debug --help

If you get an error 'unknown option '--debug', then MariaDB is not compiled for debugging and tracing.

If it's not compiled for debugging, see Compiling MariaDB for debugging

Using the mysql-test framework for debugging

Creating a mysql-test test

The mysql-test test framework is in the mysql-test directory. In this directory you can find a README that describes how to add new tests.

The main tests can be found in the 't' directory and the corresponding results file can be found in the 'r' directory.

Here follows a short description of how to add a new test and debug MariaDB with the test suite. We are using the t/alias.test suite as base for the new test. Beware that you need to compile MariaDB first, before you can run the tests.

cp t/alias.test t/new.test
xemacs t/new.test           # Edit the test. Using 'vim' is also ok :)
./mysql-test-run --record t/new.test

The last row runs your test and records the result in 'r/new.result'. For new test runs you should not use the --record option.

In the case you have done a change in the MariaDB server and the test result changes, you should examine the output from the test run to see if the new result makes sence. You can also diff the original and new result file with:

diff r/new.result r/new.reject

When you are sure the new code is correct you can replace the result with:

mv r/new.result r/new.reject
or
mv r/new.re*

For a full set of options to mysql-test-run, use:

./mysql-test-run --help

Debugging a failing or crashing test

The directory var/log contains logs from mysqltest, mysqld and other parts of the test suite that should help you track down the problem.

If this doesn't help, one of the following methods may provide more information of what's going wrong:

Using debug tracing

Running the test suite with the --debug option will generate a trace file var/log/mysqld.1.trace or var/log/mysqld.2.trace. (The second file is for the slave if you run a test that uses a master/slave setup).

mysql-test-run --debug t/new.test

A trace is very useful in finding where mysqld died and what it did just before it died.

It's also very useful for discovering what changed between two different MariaDB instances (for example if you added some new functionality to MariaDB that now changes how MariaDB works).

In this case, generate a --debug file with your old server and with your new server and compare these near where the error append to find out what has changed!

As the log file contains memory references it's sometimes hard to compare the files. Here follows a short perl script that changes the memory references to '#' which makes the comparison easier:

#!/usr/bin/perl -i

while (<>)
{
  s/^T@[0-9]+://g;
  s/0x[0-9a-f]+(\s|\n|\))/#$1/g;
#  s/[0-9a-f]{5,}/#/g;
  s/size: [0-9]+/size: #/g;
  print $_;
}

Save this as 'convert-debug-for-diff' and run it on your trace files before comparing them. The above test script can be found in newer releases of MariaDB in scripts/convert-debug-for-diff.

To make the trace files smaller you can try running the test suite with:

mysql-test-run --mysqld=--debug=d,info,error,query,general,where:O,/tmp/mysqld.trace new.test

The above doesn't' contain function tracing and only prints selected DBUG_PRINT tags. The trace file in this case will be stored in /tmp/mysqld.trace.

Another common debug string with slightly more information is:

-mysqld=--debug=d:t:i:-f,mysql_select/,open_tables/,close_thread_tables/,\
log_general/,find_block/,make_lock_and_pin/,reg_requests/,unreg_request/,\
fix_paths/,translog_write_variable_record/:O,/tmp/mysqld.trace

The -f option disables all logging for the given functions. Function names ending with / stands for all functions called by the given one.

See the file dbug/dbug.t for a full documentation of the debug options. You can find the source in dbug/dbug.c

Running a test case against a running server

If you want to run the test case against an existing server, for example mysqld running in a debugger, you can use the --extern option:

./mysql-test-run --extern socket=/tmp/mysql.sock t/new.test
or
mysql-test-run --extern "socket=/tmp/mysql.sock" --extern user=anna --extern password=guesswhat t/new.test

Running test with gdb / ddd

./mysql-test-run --gdb t/new.test
./mysql-test-run --ddd t/new.test