How to Produce a Full Stack Trace for mariadbd

Partial Stack Traces in the Error Log

When mariadbd crashes, it will write a stack trace in the by default. This is because the option defaults to ON. With a normal release build, this stack trace in the may look something like this:

If you plan to report a bug about the problem, then this information can be very useful for MariaDB's developers to track down the root cause. However, notice that some of the function names in the call stack are missing. In some cases, this partial stack trace may not be enough to find out exactly where the problem is.

A full stack trace can only be produced if you have debugging symbols for your mariadbd binary.

Obtaining Debugging Symbols for Your mariadbd executable

Debug information is used by debugging tools to produce a meaningful stack trace. Importantly these packages do not replace any executables or any existing production executables or in any way interfere with the way the production server ran before these packages where installed.

If you are obtaining a backtrace for a coredump, you can move the core dump to a difference server that has the identical mariadb-server and debug info packages, and perform the backtrace there with no loss of information.

Installing Debug Info Packages on Linux

On some Linux distributions, you may be able to install debuginfo packages that contain debugging symbols.

Currently, debuginfo packages may not allow the server to print a nice stack trace in the error log. They also allow users to extract full stack traces from core dumps. See for more information.

Installing Debug Info Packages with yum/dnf

The MariaDB yum repository contains packages.

On RHEL, CentOS, Fedora, and other similar Linux distributions, it is highly recommended to install the relevant from MariaDB's repository using or . Starting with RHEL 8 and Fedora 22, yum has been replaced by dnf, which is the next major version of yum. However, yum commands still work on many systems that use dnf. For example:

See for more information.

Installing Debug Info Packages with zypper

The MariaDB zypper repository contains packages.

On SLES, OpenSUSE, and other similar Linux distributions, it is highly recommended to install the relevant from MariaDB's repository using . For example:

See for more information.

Installing Debug Info Packages from MariaDB's Debian or Ubuntu repository

These are for when you already installed MariaDB from a MariaDB mirror.

For Ubuntu an additional repository step is needed:

Adjust 10.5 to the major version you are debugging and focal to the required distribution.

From the version isn't included in the package name and mariadb-server-core-dbgsym can be used as the package name.

Installing Debug Info Packages packaged by Ubuntu or Debian

If you used the MariaDB versions provided by Debian or Ubuntu see the following links.

For Debian see

For Ubuntu see

Installing Debugging Symbols on Windows

Debugging symbols are available to install on Windows.

Installing Debugging Symbols with the MSI Installer on Windows

Debugging symbols can be installed with the installer. Debugging symbols are not installed by default. You must perform a custom installation and explicitly choose to install debugging symbols.

The installer can be downloaded from the .

Installing Debugging Symbols with the ZIP Package on Windows

MariaDB also provides a package that contains debugging symbols on Windows.

The package that contains debugging symbol can be downloaded from the .

Containers with Debug Symbols

Prebuilt Debug Containers

These are currently only per major release version and are generated out of CI. They are always the latest version in the main branch on .

There are available on at .

Use the container name quay.io/mariadb-foundation/mariadb-debug:VERSION.

Where VERSION corresponds to the major version you wish to test with.

Creating a Container with Debugging Symbols

Build using a Containerfile/Dockerfile:

Build with:

Then you will have a mariadb_debug container.

Example use:

Enabling Core Dumps

To enable core dumps, see for details.

Where is the Core File on Linux?

At the bottom of the error log there will be some text about the core location including:

If the was a core limit in the resource limits there may be limited or no core file information.

If the core pattern begins with a |, then the following is the executable that handled the core file during the crash. The following show a few techniques to access the core depending on the pattern. If another program is used, look at its manual page to see how to get access to the core file.

If a plain core filename is in the "Core pattern" there's a good chance it will be in the Working directory location. It might have a .{process number} suffix on the filename.

Extracting a core file from a container

If you are running MariaDB in a container, the locations where the core dump can be generated are limited. Looking at the container log, this will likely be where the error log information is. The "Core pattern" of a Linux system is currently a global fixed value. The consequence is if this core pattern refers to a program, that program isn't likely to be in the container and won't be executed on the crash.

The system wide crash handler can be changed with sysctl kernel.core_pattern=core to set this back to a file based crash. With this, the crash should occur in the working directory, normally the /var/lib/mysql data directory of the container volume.

Extracting a core file from systemd-coredump

For systemd-coredump, there is a program coredumpctl to manage access.

To access the program using gdb, coredumpctl debug (defaults to last crash), will load the core dump in gdb. The instructions in the for extracting information.

See also: .

Extract a core file from abrt

A core pattern of |/usr/libexec/abrt-hook-ccpp indicates abrt system is used.

is a command line user interface for access the core file.

Extract a core file from apport

A core pattern of [|/usr/share/apport/apport indicates apport.

For more information see .

allows you to "Examine Locally" and run a gdb session. One you have gdb started instructions in the can be used for extracting information.

Analyzing a Core File with gdb on Linux

To analyze the core file on Linux, you can use .

For example, to open a core file with , you could execute the following:

Be sure to replace /usr/sbin/mariadbd with the path to your mariadbd binary (might be mysqld on and earlier) and to also replace /var/lib/mysql/core.932 with the path to your core file.

Once has opened the core file, if you want to , then you could execute the following commands:

If you do not execute set logging file, then the set logging on command creates a gdb.txt in your current working directory. Redirecting the output to a file is useful, because it can make it easier to analyze. It also makes it easier to send the information to a MariaDB developer, if that becomes necessary.

Do any commands that you would like to do. For example, you could .

Once you are done, you can exit by executing the command.

Getting Backtraces with gdb on Linux

On Linux, once you have debugging symbols for your mariadbd binary, you can use the utility to get , which are what gdb calls stack traces. Backtraces can be obtained from a core file or from a running mariadbd process.

Full are preferred and will contain function arguments, which can contain useful information such as query strings, so it can make the information easier to analyze.

To get a full of the main thread, then you could execute the following:

If you want to get a full of all threads, then you could execute the following:

If you want to get a full backtrace to a file to report a bug, the recommended way is to use gdb:

This will write the full backtrace into the file gdb.txt.

Getting Full Backtraces For All Threads From a Core File

Sometimes it can be helpful to get full for all threads. The full will contain function arguments, which can contain useful information such as query strings, so it can make the information easier to analyze.

To get full for all threads from a mariadbd core file, execute a command like the following:

Be sure to replace /usr/sbin/mariadbd with the path to your mariadbd binary and to also replace /var/lib/mysql/core.932 with the path to your core dump.

The will be output to the file mariadbd_full_bt_all_threads.txt.

Getting Full Backtraces For All Threads From a Running mariadbd Process

Sometimes it can be helpful to get full for all threads. The full backtraces will contain function arguments, which can contain useful information such as query strings, so it can make the information easier to analyze.

To get full for all threads from a running mariadbd process, execute a command like the following:

Be sure to replace /usr/sbin/mariadbd with the path to your mariadbd binary.

The will be output to the file mariadbd_full_bt_all_threads.txt.

Sometimes very busy systems are too busy to batch obtain the backtrace. If this is the case, gcore $(pidof mariadbd) can save the core and then obtain the backtrace out of the dumped core.

Getting a Full Backtrace out of a Container

If the crash or assertion is repeatable it could be easiest to run mariadbd under #gdb.

The container image name here can be a prebuilt one from quay.io/mariadb-foundation/mariadb-debug or an explicit version built yourself as above.

In docker-compose.yml form this looks like:

Note, the initialization of data is assumed. Omit command and user if it isn't.

If you wish to attach to and existing process in a container, the container needs to be started with the SYS_PTRACE capability. The sysctl kernel.yama.ptrace_scope that allows this should also be set to 0.

Note: podman has the same arguments and behaviour as docker if you'd rather use that.

or in compose:

The container process is always pid one, and here we use c then thread apply all bt -frame-arguments all full as the pre-loaded gdb commands. When a particular signal like assert or SEGV is triggered, the backtrace will be displayed.

Letting a Container coredump

First, the sysctl kernel.core_pattern needs to be core. If it starts with a pipe character it try to execute this within the container. This is a kernel wide setting and cannot be applied to a specific container. The bottom of a crash will show you want it is set to.

With this set, just running a container with debug info is sufficient. The core should be dumped on the datadir volume. With the core dump here, analysis can occur like:

Running a Copy of the Database Directory

If you are concerned with debuggers running on your production database you can also copy the database to another location.

This is useful when you know which statement crashed the server.

Just start mariadbd with the options--datadir=/copy-of-original-data-directory --core-file --stack-trace --socket=/tmp/mariadbd-alone.sock --skip-networking

Disabling Stack Traces in the Error Log

In order to disable stack traces in the , you can configure the option either on the command-line or in a relevant server in an . For example:

Reporting the Problem

If you encounter some problem in MariaDB, then MariaDB's developers would appreciate if you would at the . Please include the following information:

• Your full stack trace.

• Your .

• Your .

• How to reproduce the problem.

A is available for large and/or sensitive information. Please upload in .tar.gz or .zip archive.

For very difficult or critical errors, you should consider uploading the following information to the the following:

• Your build of mariadbd (if you compiled it), otherwise version information on the mariadb-server package.

• Your core file.

• Your contact information.

• The associated for the bug, if you .

This information will allow MariaDB developers to analyze it and try to create a fix.

_{This page is licensed: CC BY-SA / Gnu FDL}