How to Produce a Full Stack Trace for mysqld
mysqld fails hard (core dump) it will, by default,
write a stack trace in the
'hostname'.err file in the
database directory. However, in some cases this is not enough to find out
exactly where the problem is.
The following instructions describe how to produce a full stack trace and a core dump on Unix.
To be able to pinpoint exactly where the problem is, a MariaDB developer would need one or more of the following:
- A full stack trace (which one can only get from a binary compiled with debugging)
- A core file (an image of the crashed program)
- A mysqld compiled for debugging.
The steps to create the above are:
- Download a source tar.gz file
- Compile it for debugging
- Update your my.cnf file to ensure you get a
coreand a stack trace (see below).
- Install the debug mysqld instead of your normal one (see below).
- Run with the debug version until you get a new crash.
- Restore your old
Updating your my.cnf
Add the following to the end of your
[mysqld] stack-trace core-file disable-gdb [mysqld-safe] --core-file-size=unlimited
These are safe to leave there also in the future.
If you are not using mysqld-safe, then you should ensure that you don't have a system limit for the size of your core file. You can ensure this by doing the following in the shell script that start
ulimit -c unlimited
Note: If you are using
mysqld_safe and running
mysqld as the
root user, then no
core file is created on some systems. The solution
is to run
mysqld as another user.
Temporarily replacing your standard mysqld with the debug version
This is how to do it on Open SuSE. Other Linux distributions may put
mysqld in a different location.
mysqladmin shutdown cd /usr/local/mysql/libexec mv mysqld mysqld-orig # Do not use cp, because cp loses the file timestamp. cp ~/mariadb/current/sql/mysqld mysqld-debug ln -s mysqld-debug mysqld
The above commands replace the current
mysqld with the debug version you
compiled, but do so in a way which makes it trivial to revert back to the
Once the debug version is in place, start mysqld again (with something like):
Restoring your original mysqld version
If you want to restore your original mysqld binary, you can do it with:
cd /usr/local/mysql/libexec rm mysqld mv mysqld-orig mysqld
Notice that the debug binary was not deleted, all that was removed was the sym-link (and it was immediately replaced by the original binary). This way, the debug binary is still there if you need it in the future.
Using the core file
If the debug version of mysqld crashes, you should now have:
- A more precise stack trace in the
'hostname'.errfile in the data directory.
core.file in your data directory.
To examine the stack trace and other information in the
gdb debugger you can do:
cd ~/mariadb/current cp mariadb-database-directory/core* core gdb ~/mariadb/current/sql/mysqld core backtrace full
What to do if you don't get a core file when mysqld crashes
On some systems there is a limit of the size of the core file. You can check and increase the limit with:
ulimit -c ulimit -c unlimited
The above are shell limits and you should do this before you start mysqld again.
On some Linux systems the core files are disabled in the kernel. You can check this with:
sysctl -a | grep k.*core sysctl -a | grep dumpable
fs.suid_dumpable=2. You can set it with:
/sbin/sysctl -w fs.suid_dumpable=2
Another solution is to start the server while logged in as the
mysql user. This is because on some systems a process that changes uid is not allowed to dump core.
Forcing a core file
To force a core file for
mysqld you can send the process the
sigabrt (6) signal. This is very useful to get the state of the unresponsive
mysqld process. Note that this will crash
mysqld (like a
kill -9) and crash recovery will be run on restart.
kill -6 `pidof mysqld`
As an alternative to
`pidof mysqld`, you can find the process number either with
ps or in the
*.pid file that is in the
Running a copy of the database directory
If you are concerned with running the debug binary on your production database you can also copy the database to another location and use the new debug-enabled version with this. In this case you can skip the above instructions of installing MariaDB and instead run mysqld directly from the source directory.
This is useful when you know which statement crashed the server.
Just start mysqld with the option
Reporting the problem
For very difficult or critical errors, you should consider
sending the debug version of
mysqld and the
core file and some information of how to contact you
.zip file to the
MariaDB developers at the MariaDB Corporation so they can analyze it and try to
create a fix.