The Ultimate Guide to High Availability with MariaDB

How to Produce a Full Stack Trace for mysqld

You are viewing an old version of this article. View the current version here.

Contents

  1. Updating your my.cnf
  2. Temporarily replacing your standard mysqld with the debug version
  3. Restoring your original mysqld version
  4. Using the core file
  5. What to do if you don't get a core file when mysqld crashes
  6. Forcing a core file
  7. Running a copy of the database directory
  8. Reporting the problem

When mysqld fails hard (core dump) it will, by default, write a stack trace in the 'hostname'.err file in the database directory. However, in some cases this is not enough to find out exactly where the problem is.

The following instructions describe how to produce a full stack trace and a core dump on Unix.

To be able to pinpoint exactly where the problem is, a MariaDB developer would need one or more of the following:

  • A full stack trace (which one can only get from a binary compiled with debugging)
  • A core file (an image of the crashed program)
  • A mysqld compiled for debugging.

The steps to create the above are:

  1. Download a source tar.gz file (like mariadb-5.2.9.tar.gz).
  2. Compile it for debugging
  3. Update your my.cnf file to ensure you get a core and a stack trace (see below).
  4. Install the debug mysqld instead of your normal one (see below).
  5. Run with the debug version until you get a new crash.
  6. Restore your old mysqld version.

Updating your my.cnf

Add the following to the end of your ~/.my.cnf file:

[mysqld]
stack-trace
core-file
disable-gdb
[mysqld-safe]
--core-file-size=unlimited

These are safe to leave there also in the future.

If you are not using mysqld-safe, then you should ensure that you don't have a system limit for the size of your core file. You can ensure this by doing the following in the shell script that start mysqld.

ulimit -c unlimited

Note: If you are using mysqld_safe and running mysqld as the root user, then no core file is created on some systems. The solution is to run mysqld as another user.

Temporarily replacing your standard mysqld with the debug version

This is how to do it on Open SuSE. Other Linux distributions may put mysqld in a different location.

mysqladmin shutdown
cd /usr/local/mysql/libexec
mv mysqld mysqld-orig         # Do not use cp, because cp looses the file timestamp.
cp ~/mariadb/current/sql/mysqld mysqld-debug
ln -s mysqld-debug mysqld

The above commands replace the current mysqld with the debug version you compiled, but do so in a way which makes it trivial to revert back to the original version.

Once the debug version is in place, start mysqld again (with something like):

/etc/rc.d/mysql start

Restoring your original mysqld version

If you want to restore your original mysqld binary, you can do it with:

cd /usr/local/mysql/libexec
rm mysqld
mv mysqld-orig mysqld

Notice that the debug binary was not deleted, all that was removed was the sym-link (and it was immediately replaced by the original binary). This way, the debug binary is still there if you need it in the future.

Using the core file

If the debug version of mysqld crashes, you should now have:

  • A more precise stack trace in the 'hostname'.err file in the data directory.
  • A core or core.## file in your data directory.

To examine the stack trace and other information in the gdb debugger you can do:

cd ~/mariadb/current
cp mariadb-database-directory/core* core
gdb ~/mariadb/current/sql/mysqld core
backtrace full

What to do if you don't get a core file when mysqld crashes

On some systems there is a limit of the size of the core file. You can check and increase the limit with:

ulimit -c
ulimit -c unlimited

The above are shell limits and you should do this before you start mysqld again.

On some Linux systems the core files are disabled in the kernel. You can check this with:

sysctl -a | grep k.*core
sysctl -a | grep dumpable

Ensure that fs.suid_dumpable=2. You can set it with:

/sbin/sysctl -w fs.suid_dumpable=2

Another solution is to start the server while logged in as the mysql user. This is because on some systems a process that changes uid is not allowed to dump core.

See also http://www.cyberciti.biz/tips/linux-core-dumps.html#comments

Forcing a core file

To force a core file for mysqld you can send the process the sigabrt (6) signal. This is very useful to get the state of the unresponsive mysqld process. Note that this will crash mysqld (like a kill -9) and crash recovery will be run on restart.

kill -6 `pidof mysqld`

Alternatively of `pidof mysqld`, you can find the process number either with ps or in the *.pid file that is in the datadir directory.

Running a copy of the database directory

If you are concerned with running the debug binary on your production database you can also copy the database to another location and use the new debug-enabled version with this. In this case you can skip the above instructions of installing MariaDB and instead run mysqld directly from the source directory.

This is useful when you know which statement crashed the server.

Just start mysqld with the option --datadir=/copy-of-original-data-directory

Reporting the problem

Report the problem, including the stack information in the MariaDB bugs database.

For very difficult or critical errors, you should consider sending the debug version of mysqld and the core file and some information of how to contact you in a .tar or .zip file to the MariaDB developers at the MariaDB Corporation so they can analyze it and try to create a fix.

Comments

Comments loading...
Content reproduced on this site is the property of its respective owners, and this content is not reviewed in advance by MariaDB. The views, information and opinions expressed by this content do not necessarily represent those of MariaDB or any other party.