How to Produce a Full Stack Trace for mysqld

Partial Stack Traces in the Error Log

When mysqld crashes, it will write a stack trace in the error log by default. This is because the stack_trace option defaults to ON. With a normal release build, this stack trace in the error log may look something like this:

2019-03-28 23:31:08 0x7ff4dc62d700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.2.23/storage/innobase/rem/rem0rec.cc line 574
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
190328 23:31:08 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.2.23-MariaDB-10.2.23+maria~stretch
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=234
max_threads=752
thread_count=273
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1783435 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7ff4d8001f28
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7ff4dc62ccc8 thread_stack 0x49000
*** buffer overflow detected ***: /usr/sbin/mysqld terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x70bfb)[0x7ffa09af5bfb]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7ffa09b7e437]
/lib/x86_64-linux-gnu/libc.so.6(+0xf7570)[0x7ffa09b7c570]
/lib/x86_64-linux-gnu/libc.so.6(+0xf93aa)[0x7ffa09b7e3aa]
/usr/sbin/mysqld(my_addr_resolve+0xe2)[0x55ca42284922]
/usr/sbin/mysqld(my_print_stacktrace+0x1bb)[0x55ca4226b1eb]
/usr/sbin/mysqld(handle_fatal_signal+0x41d)[0x55ca41d0a01d]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110e0)[0x7ffa0b4180e0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf)[0x7ffa09ab7fff]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7ffa09ab942a]
/usr/sbin/mysqld(+0x40f971)[0x55ca41ab8971]
/usr/sbin/mysqld(+0x887df6)[0x55ca41f30df6]
/usr/sbin/mysqld(+0x863673)[0x55ca41f0c673]
/usr/sbin/mysqld(+0x96648e)[0x55ca4200f48e]
/usr/sbin/mysqld(+0x89b559)[0x55ca41f44559]
/usr/sbin/mysqld(+0x8a15e4)[0x55ca41f4a5e4]
/usr/sbin/mysqld(+0x8a2187)[0x55ca41f4b187]
/usr/sbin/mysqld(+0x8b1a20)[0x55ca41f5aa20]
/usr/sbin/mysqld(+0x7f5c04)[0x55ca41e9ec04]
/usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x107)[0x55ca41d140d7]
/usr/sbin/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_info+0x72)[0x55ca41b4b992]
/usr/sbin/mysqld(_Z12mysql_insertP3THDP10TABLE_LISTR4ListI4ItemERS3_IS5_ES6_S6_15enum_duplicatesb+0x1206)[0x55ca41b560f6]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x3f68)[0x55ca41b6bee8]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x28a)[0x55ca41b70e4a]
/usr/sbin/mysqld(+0x4c864f)[0x55ca41b7164f]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1a7c)[0x55ca41b737fc]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x176)[0x55ca41b748a6]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x25a)[0x55ca41c3ec0a]
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x55ca41c3ed7d]
/usr/sbin/mysqld(+0xb75791)[0x55ca4221e791]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4)[0x7ffa0b40e4a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7ffa09b6dd0f]

If you plan to report a bug about the problem, then this information can be very useful for MariaDB's developers to track down the root cause. However, notice that some of the function names in the call stack are missing. In some cases, this partial stack trace may not be enough to find out exactly where the problem is.

A full stack trace can only be produced if you have debugging symbols for your mysqld binary.

Obtaining Debugging Symbols for Your mysqld Binary

If you want to get full stack traces, then your mysqld binary must have debugging symbols. These debugging symbols can be obtained in one of the ways listed below.

If your server is running on Linux, then you can do the following:

• If your mysqld binary is not stripped, then it will have debugging symbols. See Checking Whether Your mysqld Binary is Stripped on Linux for more information.
• If your mysqld binary is a release build instead of a debug build, then debuginfo packages that contain debugging symbols may be available on some Linux distributions. See Installing Debug Info Packages on Linux for more information.

If your server is running on Windows, then you can do the following:

• If your mysqld binary is a release build instead of a debug build, then debug symbols may be available to install. See Installing Debugging Symbols on Windows for more information.

If none of the above fit your situation, then you can do the following, regardless of the platform:

• You could compile a debug build of mysqld. See Installing a Debug Build for more information.

Once you have a mysqld binary with debugging symbols, you can get full stack traces from one of the following sources:

• If the server crashes, then a full stack trace can be found in the error log.
• If the server crashes, then the server can also create a core dump, which is essentially an image of the crashed program. The full stack trace can be obtained from the resulting core file. This will only occur if core dumps are enabled.
• If the server hangs or stalls, then a full stack trace can be obtained from the running mysqld process by using a debugger like gdb.

Checking Whether Your mysqld Binary is Stripped on Linux

On Linux, you can find out if your mysqld binary is stripped by executing the following command:

file /usr/sbin/mysqld

If this doesn't say 'stripped', then you are fine. If not, then you need to use one of the other options to obtain a mysqld binary with debugging symbols.

Installing Debug Info Packages on Linux

On some Linux distributions, you may be able to install debuginfo packages that contain debugging symbols.

Currently, debuginfo packages may not allow the server to print a nice stack trace in the error log. They also allow users to extract full stack traces from core dumps. See MDEV-20738 for more information.

Installing Debug Info Packages with yum/dnf

On RHEL, CentOS, Fedora, and other similar Linux distributions, it is highly recommended to install the relevant RPM package from MariaDB's repository using yum or dnf. Starting with RHEL 8 and Fedora 22, yum has been replaced by dnf, which is the next major version of yum. However, yum commands still work on many systems that use dnf. For example:

sudo yum install MariaDB-server-debuginfo

See Installing MariaDB with yum/dnf: Installing Debug Info Packages with YUM for more information.

Installing Debug Info Packages with zypper

On SLES, OpenSUSE, and other similar Linux distributions, it is highly recommended to install the relevant RPM package from MariaDB's repository using zypper. For example:

sudo zypper install MariaDB-server-debuginfo

See Installing MariaDB with zypper: Installing Debug Info Packages with ZYpp for more information.

Installing Debug Info Packages from MariaDB Debian or Ubuntu

These are for when you already installed MariaDB from a MariaDB mirror.

For Ubuntu and additional repository step is needed:

sudo add-apt-repository 'deb [arch=amd64,arm64,ppc64el]  https://ftp.osuosl.org/pub/mariadb/repo/10.5/ubuntu focal main/debug'

Adjust 10.5 to the major version you are debugging and focal to the required distribution.

apt-get update && apt-get install -y mariadb-server-core-10.5-dbgsym 

Installing Debug Info Packages from Ubuntu or Debian

If you used the MariaDB versions provided by Debian or Ubuntu see the following links.

For Debian see https://wiki.debian.org/AutomaticDebugPackages

For Ubuntu see https://wiki.ubuntu.com/Debug%20Symbol%20Packages

Installing Debugging Symbols on Windows

Debugging symbols are available to install on Windows.

Installing Debugging Symbols with the MSI Installer on Windows

Debugging symbols can be installed with the MSI installer. Debugging symbols are not installed by default. You must perform a custom installation and explicitly choose to install debugging symbols.

The MSI installer can be downloaded from the MariaDB downloads page.

Installing Debugging Symbols with the ZIP Package on Windows

MariaDB also provides a ZIP package that contains debugging symbols on Windows.

The ZIP package that contains debugging symbol can be downloaded from the MariaDB downloads page.

Installing a Compiled Debug Build

When trying to find the root cause of complicated problems, it can be helpful to use a debug build of mysqld.

If a debug build of mysqld crashes, then it will provide the following information:

• If the binaries are not stripped, then the error log will contain a more precise stack trace.
• If core dumps are enabled, then a core file will be created.
  • On Linux, the name of the core file is usually something like core or core.${PID}, and the core file is usually located in the datadir by default. However, the file name and location of the core file are configurable. See Enabling Core Dumps: Setting the Path on Linux for more information.
• Debug builds contain more runtime checks than release builds, so the error log may contain a more-detailed assertion failure.

The suggested steps are:

1. Compile the debug build.
2. Temporarily install the debug mysqld binary instead of your release mysqld binary.
3. If your issue involves a crash, then enable core dumps.
4. Reproduce your issue with the debug mysqld binary.
5. When you've collected enough information, reinstall your release mysqld binary.

The process is described in more detail below. The process assumes that you are using Linux, so the process would be different on other platforms, such as Windows.

Enabling Core Dumps

To enable core dumps, see Enabling Core Dumps for details.

Analyzing a Core File with gdb on Linux

To analyze the core file on Linux, you can use gdb.

For example, to open a core file with gdb, you could execute the following:

sudo gdb /usr/sbin/mysqld  /var/lib/mysql/core.932

Be sure to replace /usr/sbin/mysqld with the path to your mysqld binary and to also replace /var/lib/mysql/core.932 with the path to your core file.

Once gdb has opened the core file, if you want to log all output to a file, then you could execute the following commands:

set logging file /tmp/gdb_output.log
set logging on

If you do not execute set logging file, then the set logging on command creates a gdb.txt in your current working directory. Redirecting the output to a file is useful, because it can make it easier to analyze. It also makes it easier to send the information to a MariaDB developer, if that becomes necessary.

Do any commands that you would like to do. For example, you could get the backtraces.

Once you are done, you can exit gdb by executing the quit command.

Getting Backtraces with gdb on Linux

On Linux, once you have debugging symbols for your mysqld binary, you can use the gdb utility to get backtraces, which are what gdb calls stack traces. Backtraces can be obtained from a core file or from a running mysqld process.

If you want to get a backtrace of the main thread, then you could execute the following:

bt

If you want to get a backtrace of all threads, then you could execute the following:

thread apply all bt

Sometimes it can be helpful to get full backtraces. The full backtraces will contain function arguments, which can contain useful information such as query strings, so it can make the information easier to analyze.

If you want to get a full backtrace of the main thread, then you could execute the following:

bt full

If you want to get a full backtrace of all threads, then you could execute the following:

thread apply all bt full

If you want to get a full backtrace to report a bug, the recommended way is to use gdb:

set logging on
set pagination off
set print frame-arguments all
thread apply all bt full
set logging off

This will write the full backtrace into the file gdb.txt.

Getting Backtraces From a Core File with gdb on Linux

It is possible to get backtraces for all threads from a core file using gdb.

Getting Backtraces For All Threads From a Core File

To get backtraces for all threads from a running mysqld process, execute a command like the following:

sudo gdb --batch --eval-command="thread apply all bt" /usr/sbin/mysqld /var/lib/mysql/core.932  > mysqld_bt_all_threads.txt

Be sure to replace /usr/sbin/mysqld with the path to your mysqld binary and to also replace /var/lib/mysql/core.932 with the path to your core dump.

The backtraces will be output to the file mysqld_bt_all_threads.txt.

Getting Full Backtraces For All Threads From a Core File

Sometimes it can be helpful to get full backtraces for all threads. The full backtraces will contain function arguments, which can contain useful information such as query strings, so it can make the information easier to analyze.

To get full backtraces for all threads from a running mysqld process, execute a command like the following:

sudo gdb --batch --eval-command="thread apply all bt full" /usr/sbin/mysqld /var/lib/mysql/core.932  > mysqld_full_bt_all_threads.txt

Be sure to replace /usr/sbin/mysqld with the path to your mysqld binary and to also replace /var/lib/mysql/core.932 with the path to your core dump.

The backtraces will be output to the file mysqld_full_bt_all_threads.txt.

Getting Backtraces From a Running mysqld Process with gdb on Linux

It is possible to get backtraces for all threads from a running mysqld process using gdb.

Getting Backtraces For All Threads From a Running mysqld Process

To get backtraces for all threads from a running mysqld process, execute a command like the following:

sudo gdb --batch --eval-command="thread apply all bt" /usr/sbin/mysqld $(pgrep -xn mysqld)  > mysqld_bt_all_threads.txt

Be sure to replace /usr/sbin/mysqld with the path to your mysqld binary.

The backtraces will be output to the file mysqld_bt_all_threads.txt.

Getting Full Backtraces For All Threads From a Running mysqld Process

Sometimes it can be helpful to get full backtraces for all threads. The full backtraces will contain function arguments, which can contain useful information such as query strings, so it can make the information easier to analyze.

To get full backtraces for all threads from a running mysqld process, execute a command like the following:

sudo gdb --batch --eval-command="thread apply all bt full" /usr/sbin/mysqld $(pgrep -xn mysqld)  > mysqld_full_bt_all_threads.txt

Be sure to replace /usr/sbin/mysqld with the path to your mysqld binary.

The backtraces will be output to the file mysqld_full_bt_all_threads.txt.

Running a Copy of the Database Directory

If you are concerned with running the debug binary on your production database you can also copy the database to another location and use the new debug-enabled version with this. In this case you can skip the above instructions of installing MariaDB and instead run mysqld directly from the source directory.

This is useful when you know which statement crashed the server.

Just start mysqld with the option --datadir=/copy-of-original-data-directory --core-file --stack-trace

Disabling Stack Traces in the Error Log

In order to disable stack traces in the error log, you can configure the skip_stack_trace option either on the command-line or in a relevant server option group in an option file. For example:

[mariadb]
...
skip_stack_trace

Reporting the Problem

If you encounter some problem in MariaDB, then MariaDB's developers would appreciate if you would report a bug at the MariaDB JIRA bug tracker. Please include the following information:

• Your full stack trace.
• Your error log.
• Your option files.
• How to reproduce the problem.

For very difficult or critical errors, you should consider uploading the following information to the MariaDB FTP server in a .tar.gz or .zip archive:

• Your debug build of mysqld.
• Your core file.
• Your contact information.
• The associated JIRA issue identifier for the bug, if you reported a bug.

This information will allow the MariaDB developers at the MariaDB Corporation to analyze it and try to create a fix.