How to produce a full stack trace for mysqld (The MariaDB server)
Contents
When mysqld
fails hard (core dump) it will by
default write a stack trace in the 'hostname'.err
file in the database directory. In some cases this is however not
enough to find out exactly where the problem.
The following instructions is how to produce a full stack trace on Unix.
To be able to exactly pinpoint where the problem is a MariaDB developer would need one or more of the following:
- A full stack trace (that one can only get from a binary compiled with debugging)
- A core file (an image of the crashed program)
- A mysqld compiled for debugging.
The steps to create the above are:
- Download a source tar.gz file (like
mariadb-5.2.9.tar.gz
). - Compile it for debugging
- Update your .my.cnf file to ensure you get a
core
and stack trace. - Install the debug version instead of your normal one.
- Run with the debug version until you get a new crash.
- Restore your old
mysqld
version.
Compiling MariaDB for debugging
Here is an example of how to compile MariaDB for debugging in our home directory with MariaDB 5.2.9 as an example:
cd ~ mkdir mariadb cd mariadb tar xvf mariadb-5.2.9.tar.gz ln -s mariadb-5.2.9 current cd current ./BUILD/compile-pentium64-debug
The last command will produce a debug version of
sql/mysqld
. If you have another system than 64 bit
Intel/Amd on Linux you can use another BUILD/...-debug
file. If this fails, you can try with:
./BUILD/autorun.sh ./configure --with-debug=full -with-extra-charsets=complex \ --with-plugin-aria --with-aria-tmp-tables --without-plugin-innodb_plugin \ --with-mysqld-ldflags=-all-static --with-client-ldflags=-all-static make
Updating your my.cnf
Add the following last to your ~/.my.cnf
file:
[[mysqld]] --stack-trace --core-file [[mysqld-safe]] --core-file-size=unlimited
These are safe to leave there also in the future.
Note that if you are using safe_mysqld
and running
mysqld
as root
, then no
core
file is created on some systems. The solution
is to run mysqld
as another user.
Temporary replacing your standard mysqld with the debug version
This is how you do it on Open SuSE. Other version may have a different
place for where mysqld
is stored.
mysqladmin shutdown cd /usr/local/mysql/libexec mv mysqld mysqld-org cp ~/mariadb/current/sql/mysqld mysqld-debug ln -s mysqld-debug mysqld
The above replaced your mysqld version with the debug version, but so that it's trivial for your to revert back to your original version.
And then start mysqld again (normally with something like
/etc/rc.d/mysql start
Restoring your original mysqld version
If you want to restore your old version that you replaced above, you do it with:
cd /usr/local/mysql/libexec rm mysqld ln -s mysqld-org mysqld
Using the core file
If the debug version of mysqld crashes, you should now have:
- A more precise stack trace in the
'hostname'.err
in the data directory. - A
core
,core.
file in your data directory.##
To examine the stack trace and other information in the
gdb
debugger you can do:
cd ~/mariadb/current cp mariadb-database-directory/core* core gdb ~/mariadb/current/sql/mysqld core backtrace full
Running a copy of the database directory
If you are concerned with running the debug binary on your production database you can also copy the database to another location and use the new debug version with this. In this case you can skip the above instructions of installing MariaDB and instead run mysqld directly from the source directory.
This is useful when you know which statement crashed the server.
Just start mysqld with the option
--datadir=/copy-of-original-data-directory
Reporting the problem
Report the problem, including the stack information in the MariaDB bugs database.
For very difficult or critical errors, you should consider
sending the debug version of mysqld
and the
core
file and some information of how to contact you
in a .tar
or .zip
file to the
MariaDB developers at Monty Program Ab so that we can analyze it and
try to produce a fix.