How to Produce a Full Stack Trace for mysqld

When mysqld fails hard (core dump) it will, by default, write a stack trace in the 'hostname'.err file in the database directory. However, in some cases this is not enough to find out exactly where the problem is.

The following instructions describe how to produce a full stack trace and a core dump on Unix.

To be able to pinpoint exactly where the problem is, a MariaDB developer would need one or more of the following:

• A mysqld compiled for debugging or at least a binary that is not stripped.
• A full stack trace. This can in many case be found in the mysqld.err file. If not you can get it from a core file.
• A core file (an image of the crashed program)

The steps to create the above are:

1. Download a source tar.gz file (like mariadb-5.2.9.tar.gz). For some systems there are already binaries compiled for debugging, in which case you can skip the next step.
2. Compile it for debugging
3. Update your my.cnf file to ensure you get a core and a stack trace (see below).
4. Install the debug mysqld instead of your normal one (see below).
5. Run with the debug version until you get a new crash.
6. Restore your old mysqld version.

Checking That Your Binary is Not Stripped

<<toc>> When things goes wrong, It's always better to have a version of the mysqld daemon that is not stripped.

shell> file /usr/sbin/mysqld

If this doesn't say 'stripped' then you are fine. If not, you should either download a binary with debugging information or compile it, without stripping the binary.

Updating Your my.cnf

Add the following to the end of your ~/.my.cnf file:

[mysqld]
stack-trace
core-file
disable-gdb
[mysqld-safe]
--core-file-size=unlimited

These are safe to leave there.

You can check your current values with:

my_print_defaults --defaults-file=my.cnf mysqld
my_print_defaults --defaults-file=my.cnf mysqld-safe

If you are not using mysqld-safe, then you should ensure that you don't have a system limit for the size of your core file. You can ensure this by doing the following in the shell script that start mysqld.

ulimit -c unlimited

Temporarily Replacing Your Standard mysqld With the Debug Version

This is how to do it on Open SuSE. Other Linux distributions may put mysqld in a different location.

mysqladmin shutdown
cd /usr/local/mysql/libexec
mv mysqld mysqld-orig         # Do not use cp, because cp loses the file timestamp.
cp ~/mariadb/current/sql/mysqld mysqld-debug
ln -s mysqld-debug mysqld

The above commands replace the current mysqld with the debug version you compiled, but do so in a way which makes it trivial to revert back to the original version.

Once the debug version is in place, start mysqld again (with something like):

/etc/rc.d/mysql start

Restoring Your Original mysqld Version

If you want to restore your original mysqld binary, you can do it with:

cd /usr/local/mysql/libexec
rm mysqld
mv mysqld-orig mysqld

Notice that the debug binary was not deleted, all that was removed was the sym-link (and it was immediately replaced by the original binary). This way, the debug binary is still there if you need it in the future.

Using the Core File

If the debug version of mysqld crashes, you should now have:

• A more precise stack trace in the 'hostname'.err file in the data directory.
• A core or core.## file in your data directory.

To examine the stack trace and other information in the gdb debugger you can do:

Replace "/usr/sbin" and "mariadb-database-directory" with the paths matching your setup. If you have more than one core file, replace core* with the name of the newest core file.

cd  /tmp
cp mariadb-database-directory/core* core
gdb /usr/sbin/mysqld  mariadb-database-directory/core*
set logging on
backtrace full

If you want to have a back trace for all threads, you can do the following while you are still in gdb:

thread apply all bt

You can get out from gdb with the quit command.

The set logging on command creates a gdb.txt in your current directory of the output from gdb that you can later examine or send to a MariaDB developer.

What To Do If You Don't Get a Core File When mysqld Crashes

On some systems there is a limit of the size of the core file. You can check and increase the limit with:

ulimit -c
ulimit -c unlimited

The above are shell limits and you should do this before you start mysqld again.

If the mysqld server is built with ASAN (Address Sanitizer) then it will not be able to generate a core file.

On some Linux systems the core files are disabled in the kernel. You can check this with:

sysctl -a | grep k.*core
sysctl -a | grep dumpable

Ensure that fs.suid_dumpable=2. You can set it with:

/sbin/sysctl -w fs.suid_dumpable=2

Another solution is to start the server while logged in as the mysql user. This is because on some systems a process that changes uid is not allowed to dump core.

See also http://www.cyberciti.biz/tips/linux-core-dumps.html#comments

Getting a core file under systemd

Systems using systemd writes by default core files to the journal that you can retrieve with coredumpctl list,if you are lucky. This is the case with openSUSE 42.3.

To get the core files in the current directory of the application, which is what mysql-test-run expects, you can execute as root:

echo 'core.%p' > /proc/sys/kernel/core_pattern

To make this setting permanent, you have to also edit the file /usr/lib/sysctl.d/50-coredump.conf and change this to say:

kernel.core_pattern=core.%p

Forcing a Core File

To force a core file for mysqld you can send the process the sigabrt (6) signal. This is very useful to get the state of the unresponsive mysqld process. Note that this will crash mysqld (like a kill -9) and crash recovery will be run on restart.

kill -6 `pidof mysqld`

As an alternative to `pidof mysqld`, you can find the process number either with ps or in the *.pid file that is in the datadir directory.

Getting a Stack Trace For a Running mysqld Server

Even if you don't have a server that is compiled for debugging, there are still ways to get more information out from it if things go wrong.

To do a per threads backtrace of a running mysqld process:

gdb --batch --eval-command="thread apply all bt" program $(pgrep -xn mysqld)

Running a Copy of the Database Directory

If you are concerned with running the debug binary on your production database you can also copy the database to another location and use the new debug-enabled version with this. In this case you can skip the above instructions of installing MariaDB and instead run mysqld directly from the source directory.

This is useful when you know which statement crashed the server.

Just start mysqld with the option --datadir=/copy-of-original-data-directory --core-file --stack-trace

Reporting the Problem

Report the problem, including the stack information in the MariaDB bugs database.

For very difficult or critical errors, you should consider sending the debug version of mysqld and the core file and some information of how to contact you in a .tar or .zip file to the MariaDB developers at the MariaDB Corporation so they can analyze it and try to create a fix.