How to Produce a Full Stack Trace for mysqld

When mysqld fails hard (core dump) it will, by default, write a stack trace in the 'hostname'.err file in the database directory. However, in some cases this is not enough to find out exactly where the problem is.

The following instructions describe how to produce a full stack trace and a core dump on Unix.

To be able to pinpoint exactly where the problem is, a MariaDB developer would need one or more of the following:

• A mysqld compiled for debugging or at least a binary that is not stripped.
• A full stack trace. This can in many case be found in the mysqld.err file. If not you can get it from a core file.
• A core file (an image of the crashed program)

The steps to create the above are:

1. Download a source tar.gz file (like mariadb-5.2.9.tar.gz). For some systems there are already binaries compiled for debugging, in which you can skip the next step.
2. Compile it for debugging
3. Update your my.cnf file to ensure you get a core and a stack trace (see below).
4. Install the debug mysqld instead of your normal one (see below).
5. Run with the debug version until you get a new crash.
6. Restore your old mysqld version.

Checking that your binary is not stripped

<<toc>> When things goes wrong, It's always better to have a version of the mysqld daemon that is not stripped.

shell> file /usr/sbin/mysqld

If this doesn't say 'stripped' then you are fine. If not, you should either download a binary with debugging information or compile it, without stripping the binary.

Updating your my.cnf

Add the following to the end of your ~/.my.cnf file:

[mysqld]
stack-trace
core-file
disable-gdb
[mysqld-safe]
--core-file-size=unlimited

These are safe to leave there also in the future.

You can check your current values with:

my_print_defaults --defaults-file=my.cnf mysqld
my_print_defaults --defaults-file=my.cnf mysqld-safe

If you are not using mysqld-safe, then you should ensure that you don't have a system limit for the size of your core file. You can ensure this by doing the following in the shell script that start mysqld.

ulimit -c unlimited

Temporarily replacing your standard mysqld with the debug version

This is how to do it on Open SuSE. Other Linux distributions may put mysqld in a different location.

mysqladmin shutdown
cd /usr/local/mysql/libexec
mv mysqld mysqld-orig         # Do not use cp, because cp loses the file timestamp.
cp ~/mariadb/current/sql/mysqld mysqld-debug
ln -s mysqld-debug mysqld

The above commands replace the current mysqld with the debug version you compiled, but do so in a way which makes it trivial to revert back to the original version.

Once the debug version is in place, start mysqld again (with something like):

/etc/rc.d/mysql start

Restoring your original mysqld version

If you want to restore your original mysqld binary, you can do it with:

cd /usr/local/mysql/libexec
rm mysqld
mv mysqld-orig mysqld

Notice that the debug binary was not deleted, all that was removed was the sym-link (and it was immediately replaced by the original binary). This way, the debug binary is still there if you need it in the future.

Using the core file

If the debug version of mysqld crashes, you should now have:

• A more precise stack trace in the 'hostname'.err file in the data directory.
• A core or core.## file in your data directory.

To examine the stack trace and other information in the gdb debugger you can do:

cd ~/mariadb/current
cp mariadb-database-directory/core* core
gdb ~/mariadb/current/sql/mysqld core
backtrace full

If you want to have a back trace for all threads, you can do the following in gdb:

thread apply all bt

What to do if you don't get a core file when mysqld crashes

On some systems there is a limit of the size of the core file. You can check and increase the limit with:

ulimit -c
ulimit -c unlimited

The above are shell limits and you should do this before you start mysqld again.

On some Linux systems the core files are disabled in the kernel. You can check this with:

sysctl -a | grep k.*core
sysctl -a | grep dumpable

Ensure that fs.suid_dumpable=2. You can set it with:

/sbin/sysctl -w fs.suid_dumpable=2

Another solution is to start the server while logged in as the mysql user. This is because on some systems a process that changes uid is not allowed to dump core.

See also http://www.cyberciti.biz/tips/linux-core-dumps.html#comments

Forcing a core file

To force a core file for mysqld you can send the process the sigabrt (6) signal. This is very useful to get the state of the unresponsive mysqld process. Note that this will crash mysqld (like a kill -9) and crash recovery will be run on restart.

kill -6 `pidof mysqld`

As an alternative to `pidof mysqld`, you can find the process number either with ps or in the *.pid file that is in the datadir directory.

Getting a stack trace for a running mysqld server

To do a per threads bacEven if you don't have a server that is compiled for debugging, there are still ways to get more information out from it if things go wrong. ktrace of a running mysqld process:

gdb --batch --eval-command="thread apply all bt" program $(pgrep -xn mysqld)

Running a copy of the database directory

If you are concerned with running the debug binary on your production database you can also copy the database to another location and use the new debug-enabled version with this. In this case you can skip the above instructions of installing MariaDB and instead run mysqld directly from the source directory.

This is useful when you know which statement crashed the server.

Just start mysqld with the option --datadir=/copy-of-original-data-directory

Reporting the problem

Report the problem, including the stack information in the MariaDB bugs database.

For very difficult or critical errors, you should consider sending the debug version of mysqld and the core file and some information of how to contact you in a .tar or .zip file to the MariaDB developers at the MariaDB Corporation so they can analyze it and try to create a fix.