Enabling Core Dumps

Enabling in an Option File

MariaDB starting with 10.4

In MariaDB 10.4 and later, core dumps are enabled by default on Windows, so this step can be skipped on Windows in those versions. See MDEV-18439 for more information.

In order to enable core dumps, you need to set the core_file system variable either on the command-line or in a relevant server option group in an option file. For example:

[mariadb]
...
core_file

You can check your current value by executing:

my_print_defaults --mysqld
MariaDB starting with 10.1.35

In MariaDB 10.1.35, MariaDB 10.2.17, and MariaDB 10.3.9 and later, core_file has also been made into a system variable. Previously it was just an option. It's value can be checked at runtime by executing the following:

SHOW GLOBAL VARIABLES LIKE 'core_file';

Core Files on Linux

There are some additional details related to using core files on Linux.

Disabling Core File Size Restrictions on Linux

On some systems there is a limit on the sizes of core files that can be dumped. You can check the system's current system-wide limit by executing the following:

ulimit -c

You can check the current limit of the mysqld process specifically by executing the following:

sudo cat /proc/$(pidof mysqld)/limits | grep "core file"

If you need to change the core size limit, the method you use depends on how you start mysqld. See the sections below for more details.

MariaDB starting with 10.2.24

In MariaDB 10.2.24, MariaDB 10.3.15, and MariaDB 10.4.5 and later, the resource limits for the mysqld process are printed to the error log when the mysqld process crashes. That way, users can confirm whether the process may have been allowed to dump a core file. See MDEV-15051 for more information.

Running mysqld Using mysqld_safe

If you are starting MariaDB by running mysqld_safe, then configuring the following in the [mysqld_safe] option group in an option file should allow for unlimited sized core files:

[mysqld_safe]
...
core_file_size=unlimited

You can check your current values by executing:

my_print_defaults mysqld_safe

See mysqld_safe: Configuring the Core File Size for more details.

Note: If you are using mysqld_safe and running mysqld as the root user, then no core file is created on some systems. The solution is to run mysqld as another user.

Running mysqld Manually

If you are starting mysqld manually or in a custom script, then you can allow for unlimited sized core files by executing the following in the same shell or script in which mysqld is executed:

ulimit -c unlimited

Running mysqld Using systemd

If you are starting mysqld using systemd, then you may need to customize the MariaDB service to allow for unlimited size core files. For example, you could execute the following:

sudo tee /etc/systemd/system/mariadb.service.d/limitcore.conf <<EOF
[Service]

LimitCORE=infinity
EOF
sudo systemctl daemon-reload

See systemd: Configuring the Core File Size for more details.

Changing the System-Wide Limit

If you want to change the system-wide limit to allow for unlimited size core files for for the mysql user account, then you can do so by adding the following lines to /etc/security/limits.conf:

mysql soft core unlimited
mysql hard core unlimited

The system would have to be restarted for this change to take effect.

See Configuring Linux for MariaDB: Configuring the Core File Size for more details.

Setting the Path on Linux

If you are using Linux, then it can be helpful to change a few settings to alter where the core files is written and what file name is used. This is done by setting the kernel.core_pattern and kernel.core_uses_pid attributes. You can check the current values by executing the following:

sysctl kernel.core_pattern
sysctl kernel.core_uses_pid

They can also be temporarily altered using the sysctl utility, but it is often more common to alter them via the /proc file system. See the following example:

sudo mkdir /tmp/corefiles
sudo chmod 777 /tmp/corefiles
sudo echo /tmp/corefiles/core > /proc/sys/kernel/core_pattern
sudo echo 1 > /proc/sys/kernel/core_uses_pid

The above commands will tell the system to put core files in /tmp/corefiles, and it also tells the system to put the process ID in the file name.

If you want to make these changes permanent, then you can add the following to /etc/sysctl.conf:

kernel.core_pattern=/tmp/corefiles/core
kernel.core_uses_pid=1
MariaDB starting with 10.2.24

In MariaDB 10.2.24, MariaDB 10.3.15, and MariaDB 10.4.5 and later, the value of kernel.core_pattern is printed to the error log when the mysqld process crashes. That way, users can determine where the process may have dumped a core file. See MDEV-15051 for more information.

MariaDB until 10.2.23

In MariaDB 10.2.23, MariaDB 10.3.14, and MariaDB 10.4.4 and before, the error log contains a message indicating that a core file would be written in the datadir when the mysqld process crashes, even if this was not true because the kernel.core_pattern actually configured the process to write the core file in a different directory.

Note: Ensure that you have enough free disk space in the path pointed to by kernel.core_pattern.

Note: If you are using systemd, then the default value of kernel.core_pattern may redirect core files to be written to the systemd journal. If so, then the core files can be retrieved with the coredumpctl utility. You can change this behavior by changing kernel.core_pattern.

Core Dumps and setuid on Linux

Since mysqld executes setuid, you may have to set fs.suid_dumpable=2 to allow core dumps on Linux. You can check the current fs.suid_dumpable value by using the sysctl utility. For example:

sysctl fs.suid_dumpable

You can temporarily set it to 2 by by using the sysctl utility. For example:

sudo sysctl -w fs.suid_dumpable=2

Or you can temporarily set it to 2 by writing to the /proc file system. For example:

sudo echo 2 > /proc/sys/fs/suid_dumpable

If you want to permanently set it to 2 then you can add the following to /etc/sysctl.conf:

fs.suid_dumpable=2

Note: If you don't want to change fs.suid_dumpable, then another solution is to start mysqld directly as the mysql user, so that the setuid call is not needed.

Forcing a Core File on Linux

To force a core file for mysqld you can send the process the sigabrt signal, which has the signal code 6. This is very useful to get the state of the unresponsive mysqld process. However, this will cause mysqld to crash, and crash recovery will be run on restart.

You can send the signal with the kill command. For example:

sudo kill -6 $(pidof mysqld)

As an alternative to $(pidof mysqld), you can find the process ID either by using the ps utility or by checking the file defined by the pid_file system variable.

Core Files on Windows

There are some additional details related to using core files on Windows.

Minidump Files on Windows

On Windows, the core file is created as a minidump file.

For details on how to configure and read the minidump file, see How to read the small memory dump file that is created by Windows if a crash occurs.

Core Files and Address Sanitizer (ASAN)

If your mysqld binary is built with Address Sanitizer (ASAN) then it will not be able to generate a core file.

What's Included in Core Files

Core files usually contain a dump of all memory in the process's full address space. This means that if a server has some large buffers configured (such as a large InnoDB buffer pool), then the server's core files can get very large.

However, in MariaDB 10.3.7 and later, some large buffers have been excluded from core files on some systems as a way to reduce the size.

The following buffers are excluded:

The buffers are only excluded on Linux when using kernel version 3.4 and above and when using a non-debug build of mysqld. Some Linux kernel versions have a bug which would cause the following warning to be printed to the log:

Sep 25 10:41:19 srv1 mysqld: 2018-09-25 10:41:19 0 [Warning] InnoDB: Failed to set memory to DODUMP: Invalid argument ptr 0x2aaac3400000 size 33554432

In those cases, the core dump may exclude some additional data. If that is not a concern, then the warning can be ignored. The problem can be fixed by upgrading to a Linux kernel version in which the bug is fixed.

See Also

Comments

Comments loading...
Content reproduced on this site is the property of its respective owners, and this content is not reviewed in advance by MariaDB. The views, information and opinions expressed by this content do not necessarily represent those of MariaDB or any other party.