When mysqld fails hard (core dump) it will, by default, write a stack trace in the 'hostname'.err file in the database directory. However, in some cases this is not enough to find out exactly where the problem is.

The following instructions describe how to produce a full stack trace on Unix.

To be able to pinpoint exactly where the problem is, a MariaDB developer would need one or more of the following:

  • A full stack trace (which one can only get from a binary compiled with debugging)
  • A core file (an image of the crashed program)
  • A mysqld compiled for debugging.

The steps to create the above are:

  1. Download a source tar.gz file (like mariadb-5.2.9.tar.gz).
  2. Compile it for debugging
  3. Update your my.cnf file to ensure you get a core and a stack trace.
  4. Install the debug mysqld instead of your normal one.
  5. Run with the debug version until you get a new crash.
  6. Restore your old mysqld version.

Updating your my.cnf

Add the following to the end of your ~/.my.cnf file:

[mysqld]
--stack-trace
--core-file
[mysqld-safe]
--core-file-size=unlimited

These are safe to leave there also in the future.

Note: If you are using safe_mysqld and running mysqld as root, then no core file is created on some systems. The solution is to run mysqld as another user.

Temporarily replacing your standard mysqld with the debug version

This is how to do it on Open SuSE. Other Linux distributions may put mysqld in a different location.

mysqladmin shutdown
cd /usr/local/mysql/libexec
mv mysqld mysqld-orig
cp ~/mariadb/current/sql/mysqld mysqld-debug
ln -s mysqld-debug mysqld

The above commands replace the current mysqld with the debug version you compiled, but do so in a way which makes it trivial to revert back to the original version.

Once the debug version is in place, start mysqld again (with something like):

/etc/rc.d/mysql start

Restoring your original mysqld version

If you want to restore your original mysqld binary, you can do it with:

cd /usr/local/mysql/libexec
rm mysqld
ln -s mysqld-orig mysqld

Notice that the debug binary was not deleted, all that was removed was the sym-link (and it was immediately recreated and pointed at the original binary). This way, the debug binary is still there if you need it in the future.

Using the core file

If the debug version of mysqld crashes, you should now have:

  • A more precise stack trace in the 'hostname'.err file in the data directory.
  • A core or core.## file in your data directory.

To examine the stack trace and other information in the gdb debugger you can do:

cd ~/mariadb/current
cp mariadb-database-directory/core* core
gdb ~/mariadb/current/sql/mysqld core
backtrace full

What to do if you don't get a core file when mysqld crashes

On some systems there is a limit of the size of the core file. You can check and increase the limit with:

ulimit -c
ulimit -c unlimited

The above are shell limits and you should do this before you start mysqld again.

On some Linux systems the core files are disabled in the kernel. You can check this with:

sysctl -a | grep k.*core
sysctl -a | grep dumpable

Ensure that fs.suid_dumpable=2. You can set it with:

/sbin/sysctl -w fs.suid_dumpable=2

Another solution is to start the server while logged in as the mysql user. This is because on some systems a process that changes uid is not allowed to dump core.

See also http://www.cyberciti.biz/tips/linux-core-dumps.html#comments

Running a copy of the database directory

If you are concerned with running the debug binary on your production database you can also copy the database to another location and use the new debug-enabled version with this. In this case you can skip the above instructions of installing MariaDB and instead run mysqld directly from the source directory.

This is useful when you know which statement crashed the server.

Just start mysqld with the option --datadir=/copy-of-original-data-directory

Reporting the problem

Report the problem, including the stack information in the MariaDB bugs database.

For very difficult or critical errors, you should consider sending the debug version of mysqld and the core file and some information of how to contact you in a .tar or .zip file to the MariaDB developers at Monty Program Ab so they can analyze it and try to create a fix.

Comments

Comments loading...