How to produce a full stack trace for mysqld (The MariaDB server)

You are viewing an old version of this article. View the current version here.

When mysqld fails hard (core dump) it will by default write a stack trace in the 'hostname'.err file in the database directory. In some cases this is however not enough to find out exactly where the problem.

The following instructions is how to produce a full stack trace on Unix.

To be able to exactly pinpoint where the problem is a MariaDB developer would need one or more of the following:

  • A full stack trace (that one can only get from a binary compiled with debugging)
  • A core file (an image of the crashed program)
  • A mysqld compiled for debugging.

The steps to create the above are:

  • Download a source tar.gz file (like mariadb-5.2.9.tar.gz).
  • Compile it for debugging
  • Update your .my.cnf file to ensure you get a core and stack trace.
  • Install the debug version instead of your normal one.
  • Run with the debug version until you get a new crash.
  • Restore your old mysqld version.

Compiling MariaDB for debugging

Here is an example of how to compile MariaDB for debugging in our home directory with MariaDB 5.2.9 as an example:

cd ~
mkdir mariadb
cd mariadb
tar xvf mariadb-5.2.9.tar.gz
ln -s mariadb-5.2.9 current
cd current
./BUILD/compile-pentium64-debug

The last command will produce a debug version of sql/mysqld. If you have another system than 64 bit Intel/Amd on Linux you can use another BUILD/...-debug file. If this fails, you can try with:

./BUILD/autorun.sh
./configure --with-debug=full -with-extra-charsets=complex \
--with-plugin-aria --with-aria-tmp-tables --without-plugin-innodb_plugin \
--with-mysqld-ldflags=-all-static  --with-client-ldflags=-all-static 
make

Updating your my.cnf

Add the following last to your ~/.my.cnf file:

[[mysqld]]
--stack-trace
--core-file
[[mysqld-safe]]
--core-file-size=unlimited

These are safe to leave there also in the future.

Note that if you are using safe_mysqld and running mysqld as root, then no core file is created on some systems. The solution is to run mysqld as another user.

Temporary replacing your standard mysqld with the debug version

This is how you do it on Open SuSE. Other version may have a different place for where mysqld is stored.

mysqladmin shutdown
cd /usr/local/mysql/libexec
mv mysqld mysqld-org
cp ~/mariadb/current/sql/mysqld mysqld-debug
ln -s mysqld-debug mysqld

The above replaced your mysqld version with the debug version, but so that it's trivial for your to revert back to your original version.

And then start mysqld again (normally with something like

/etc/rc.d/mysql start

Restoring your original mysqld version

If you want to restore your old version that you replaced above, you do it with:

cd /usr/local/mysql/libexec
rm mysqld
ln -s mysqld-org mysqld

Using the core file

If the debug version of mysqld crashes, you should now have:

  • A more precise stack trace in the 'hostname'.err in the data directory.
  • A core, core.## file in your data directory.

To examine the stack trace and other information in the gdb debugger you can do:

cd ~/mariadb/current
cp mariadb-database-directory/core* core
gdb ~/mariadb/current/sql/mysqld core
backtrace full

Running a copy of the database directory

If you are concerned with running the debug binary on your production database you can also copy the database to another location and use the new debug version with this. In this case you can skip the above instructions of installing MariaDB and instead run mysqld directly from the source directory.

This is useful when you know which statement crashed the server.

Just start mysqld with the option --datadir=/copy-of-original-data-directory

Reporting the problem

Report the problem, including the stack information in the MariaDB bugs database.

For very difficult or critical errors, you should consider sending the debug version of mysqld and the core file and some information of how to contact you in a .tar or .zip file to the MariaDB developers at Monty Program Ab so that we can analyze it and try to produce a fix.

Comments

Comments loading...
Content reproduced on this site is the property of its respective owners, and this content is not reviewed in advance by MariaDB. The views, information and opinions expressed by this content do not necessarily represent those of MariaDB or any other party.