How to produce a full stack trace for mysqld (The MariaDB server)
Contents
- Compiling MariaDB for debugging
- Updating your my.cnf
- Temporarily replacing your standard mysqld with the debug version
- Restoring your original mysqld version
- Using the core file
- What to do if you don't get a core file when mysqld crashes
- Running a copy of the database directory
- Reporting the problem
When mysqld
fails hard (core dump) it will, by default,
write a stack trace in the 'hostname'.err
file in the
database directory. However, in some cases this is not enough to find out
exactly where the problem is.
The following instructions describe how to produce a full stack trace on Unix.
To be able to pinpoint exactly where the problem is, a MariaDB developer would need one or more of the following:
- A full stack trace (which one can only get from a binary compiled with debugging)
- A core file (an image of the crashed program)
- A mysqld compiled for debugging.
The steps to create the above are:
- Download a source tar.gz file
(like
mariadb-5.2.9.tar.gz
). - Compile it for debugging
- Update your my.cnf file to ensure you get a
core
and a stack trace. - Install the debug mysqld instead of your normal one.
- Run with the debug version until you get a new crash.
- Restore your old
mysqld
version.
Compiling MariaDB for debugging
Here is an example of how to compile MariaDB for debugging in our home directory with MariaDB 5.2.9 as an example:
cd ~ mkdir mariadb cd mariadb tar xvf mariadb-5.2.9.tar.gz ln -s mariadb-5.2.9 current cd current ./BUILD/compile-pentium64-debug-max
The last command will produce a debug version of sql/mysqld
.
If you have a system other than 64 bit Intel/AMD on Linux you can use a
different BUILD/...-debug-max
file. If this fails, you can
try with:
./BUILD/autorun.sh ./configure --with-debug=full -with-extra-charsets=complex \ --with-plugin-aria --with-aria-tmp-tables --without-plugin-innodb_plugin \ --with-plugins=max \ --with-mysqld-ldflags=-all-static --with-client-ldflags=-all-static make
Updating your my.cnf
Add the following to the end of your ~/.my.cnf
file:
[[mysqld]] --stack-trace --core-file [[mysqld-safe]] --core-file-size=unlimited
These are safe to leave there also in the future.
Note: If you are using safe_mysqld
and running
mysqld
as root
, then no
core
file is created on some systems. The solution
is to run mysqld
as another user.
Temporarily replacing your standard mysqld with the debug version
This is how to do it on Open SuSE. Other Linux distributions may put
mysqld
in a different location.
mysqladmin shutdown cd /usr/local/mysql/libexec mv mysqld mysqld-orig cp ~/mariadb/current/sql/mysqld mysqld-debug ln -s mysqld-debug mysqld
The above commands replace the current mysqld
with the debug version you
compiled, but do so in a way which makes it trivial to revert back to the
original version.
Once the debug version is in place, start mysqld again (with something like):
/etc/rc.d/mysql start
Restoring your original mysqld version
If you want to restore your original mysqld binary, you can do it with:
cd /usr/local/mysql/libexec rm mysqld ln -s mysqld-orig mysqld
Notice that the debug binary was not deleted, all that was removed was the sym-link (and it was immediately recreated and pointed at the original binary). This way, the debug binary is still there if you need it in the future.
Using the core file
If the debug version of mysqld crashes, you should now have:
- A more precise stack trace in the
'hostname'.err
file in the data directory. - A
core
orcore.
file in your data directory.##
To examine the stack trace and other information in the
gdb
debugger you can do:
cd ~/mariadb/current cp mariadb-database-directory/core* core gdb ~/mariadb/current/sql/mysqld core backtrace full
What to do if you don't get a core file when mysqld crashes
On some systems there is a limit of the size of the core file. You can check and increase the limit with:
ulimit -c ulimit -c unlimited
The above are shell limits and you should do this before you start mysqld again.
On some Linux systems the core files are disabled in the kernel. You can check this with:
sysctl -a | grep k.*core sysctl -a | grep dumpable
Ensure that fs.suid_dumpable=2
. You can set it with:
/sbin/sysctl -w fs.suid_dumpable=2
Another solution is to start the server while logged in as the mysql
user. This is because on some systems a process that changes uid is not allowed to dump core.
See also http://www.cyberciti.biz/tips/linux-core-dumps.html#comments
Running a copy of the database directory
If you are concerned with running the debug binary on your production database you can also copy the database to another location and use the new debug-enabled version with this. In this case you can skip the above instructions of installing MariaDB and instead run mysqld directly from the source directory.
This is useful when you know which statement crashed the server.
Just start mysqld with the option
--datadir=/copy-of-original-data-directory
Reporting the problem
Report the problem, including the stack information in the MariaDB bugs database.
For very difficult or critical errors, you should consider
sending the debug version of mysqld
and the
core
file and some information of how to contact you
in a .tar
or .zip
file to the
MariaDB developers at Monty Program Ab so they can analyze it and try to
create a fix.