How to Produce a Full Stack Trace for mysqld
mysqld fails hard (core dump) it will, by default,
write a stack trace in the
'hostname'.err file in the
database directory. However, in some cases this is not enough to find out
exactly where the problem is.
The following instructions describe how to produce a full stack trace on Unix.
To be able to pinpoint exactly where the problem is, a MariaDB developer would need one or more of the following:
- A full stack trace (which one can only get from a binary compiled with debugging)
- A core file (an image of the crashed program)
- A mysqld compiled for debugging.
The steps to create the above are:
- Download a source tar.gz file
- Compile it for debugging
- Update your my.cnf file to ensure you get a
coreand a stack trace.
- Install the debug mysqld instead of your normal one.
- Run with the debug version until you get a new crash.
- Restore your old
Updating your my.cnf
Add the following to the end of your
[mysqld] --stack-trace --core-file [mysqld-safe] --core-file-size=unlimited
These are safe to leave there also in the future.
Note: If you are using
safe_mysqld and running
root, then no
core file is created on some systems. The solution
is to run
mysqld as another user.
Temporarily replacing your standard mysqld with the debug version
This is how to do it on Open SuSE. Other Linux distributions may put
mysqld in a different location.
mysqladmin shutdown cd /usr/local/mysql/libexec mv mysqld mysqld-orig cp ~/mariadb/current/sql/mysqld mysqld-debug ln -s mysqld-debug mysqld
The above commands replace the current
mysqld with the debug version you
compiled, but do so in a way which makes it trivial to revert back to the
Once the debug version is in place, start mysqld again (with something like):
Restoring your original mysqld version
If you want to restore your original mysqld binary, you can do it with:
cd /usr/local/mysql/libexec rm mysqld ln -s mysqld-orig mysqld
Notice that the debug binary was not deleted, all that was removed was the sym-link (and it was immediately recreated and pointed at the original binary). This way, the debug binary is still there if you need it in the future.
Using the core file
If the debug version of mysqld crashes, you should now have:
- A more precise stack trace in the
'hostname'.errfile in the data directory.
core.file in your data directory.
To examine the stack trace and other information in the
gdb debugger you can do:
cd ~/mariadb/current cp mariadb-database-directory/core* core gdb ~/mariadb/current/sql/mysqld core backtrace full
What to do if you don't get a core file when mysqld crashes
On some systems there is a limit of the size of the core file. You can check and increase the limit with:
ulimit -c ulimit -c unlimited
The above are shell limits and you should do this before you start mysqld again.
On some Linux systems the core files are disabled in the kernel. You can check this with:
sysctl -a | grep k.*core sysctl -a | grep dumpable
fs.suid_dumpable=2. You can set it with:
/sbin/sysctl -w fs.suid_dumpable=2
Another solution is to start the server while logged in as the
mysql user. This is because on some systems a process that changes uid is not allowed to dump core.
Running a copy of the database directory
If you are concerned with running the debug binary on your production database you can also copy the database to another location and use the new debug-enabled version with this. In this case you can skip the above instructions of installing MariaDB and instead run mysqld directly from the source directory.
This is useful when you know which statement crashed the server.
Just start mysqld with the option
Reporting the problem
For very difficult or critical errors, you should consider
sending the debug version of
mysqld and the
core file and some information of how to contact you
.zip file to the
MariaDB developers at the MariaDB Corporation so they can analyze it and try to
create a fix.