Enabling Core Dumps
- Enabling in an Option File
- Core Files on Linux
- Core Files on Windows
- Core Files on Kubernetes
- Core Files and Address Sanitizer (ASAN)
- What's Included in Core Files
- See Also
Enabling in an Option File
MariaDB starting with 10.4
[mariadb] ... core_file
You can check your current value by executing:
MariaDB starting with 10.1.35
In MariaDB 10.1.35, MariaDB 10.2.17, and MariaDB 10.3.9 and later,
core_file has also been made into a system variable. Previously it was just an option. It's value can be checked at runtime by executing the following:
SHOW GLOBAL VARIABLES LIKE 'core_file';
Core Files on Linux
There are some additional details related to using core files on Linux.
Disabling Core File Size Restrictions on Linux
On some systems there is a limit on the sizes of core files that can be dumped. You can check the system's current system-wide limit by executing the following:
You can check the current limit of the
mysqld process specifically by executing the following:
sudo cat /proc/$(pidof mysqld)/limits | grep "core file"
If you need to change the core size limit, the method you use depends on how you start
mysqld. See the sections below for more details.
MariaDB starting with 10.2.24
In MariaDB 10.2.24, MariaDB 10.3.15, and MariaDB 10.4.5 and later, the resource limits for the
mysqld process are printed to the error log when the
mysqld process crashes. That way, users can confirm whether the process may have been allowed to dump a core file. See MDEV-15051 for more information.
Running mysqld Using mysqld_safe
If you are starting MariaDB by running
mysqld_safe, then configuring the following in the
[mysqld_safe] option group in an option file should allow for unlimited sized core files:
[mysqld_safe] ... core_file_size=unlimited
You can check your current values by executing:
See mysqld_safe: Configuring the Core File Size for more details.
Note: If you are using
mysqld_safe and running
mysqld as the
root user, then no
core file is created on some systems. The solution is to run
mysqld as another user.
Running mysqld Manually
If you are starting mysqld manually or in a custom script, then you can allow for unlimited sized core files by executing the following in the same shell or script in which mysqld is executed:
ulimit -c unlimited
Running mysqld Using systemd
If you are starting
systemd, then you may need to customize the MariaDB service to allow for unlimited size core files. For example, you could execute the following:
sudo systemctl edit mariadb.service add the contents:
See systemd: Configuring the Core File Size for more details.
Changing the System-Wide Limit
If you want to change the system-wide limit to allow for unlimited size core files for for the
mysql user account, then you can do so by adding the following lines to a file in
/etc/security/limits.d/. For example:
sudo tee /etc/security/limits.d/mariadb_core.conf <<EOF mysql soft core unlimited mysql hard core unlimited EOF
The system would have to be restarted for this change to take effect.
See Configuring Linux for MariaDB: Configuring the Core File Size for more details.
Setting the Path on Linux
If you are using Linux, then it can be helpful to change a few settings to alter where the core files is written and what file name is used. This is done by setting the
kernel.core_uses_pid attributes. You can check the current values by executing the following:
sysctl kernel.core_pattern sysctl kernel.core_uses_pid
If you are using
mysql-test-run and want to have the core as part of the test result, the optimal
setting is probably the following (store cores in the current directory as
sudo sysctl kernel.core_pattern=core.%p kernel.core_uses_pid=0
If you are using a production system, you probably want to have the core files in a specific directory,
not in the data directory. They place to store cores can be temporarily altered using the
sysctl utility, but it is often more common to alter them via the
/proc file system. See the following example:
sudo mkdir /tmp/corefiles sudo chmod 777 /tmp/corefiles sudo echo /tmp/corefiles/core > /proc/sys/kernel/core_pattern sudo echo 1 > /proc/sys/kernel/core_uses_pid
The above commands will tell the system to put core files in
/tmp/corefiles, and it also tells the system to put the process ID in the file name.
If you want to make these changes permanent, then you can add the following to a file in
/etc/sysctl.conf.d/. For example:
sudo tee /etc/sysctl.d/mariadb_core.conf <<EOF kernel.core_pattern=/tmp/corefiles/core kernel.core_uses_pid=1 EOF
MariaDB starting with 10.2.24
In MariaDB 10.2.24, MariaDB 10.3.15, and MariaDB 10.4.5 and later, the value of
kernel.core_pattern is printed to the error log when the
mysqld process crashes. That way, users can determine where the process may have dumped a core file. See MDEV-15051 for more information.
MariaDB until 10.2.23
In MariaDB 10.2.23, MariaDB 10.3.14, and MariaDB 10.4.4 and before, the error log contains a message indicating that a core file would be written in the
datadir when the
mysqld process crashes, even if this was not true because the
kernel.core_pattern actually configured the process to write the core file in a different directory.
Note: Ensure that you have enough free disk space in the path pointed to by
Extracting Linux core dumps with systemd-coredump
Core dump management can be automated using
This is enabled per default on Red Hat Enterprise Linux 8 and CentOS 8, and maybe other contemporary Linux distribution releases by now, too. It can be easily checked for by looking at the
kernel.core_pattern setting. If it looks like this systemd-coredump is enabled:
# sysctl kernel.core_pattern kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e
On other distributions like Ubuntu (at least up to 21.10) it is not enabled by default, but can be set up manually.
To see all recent core dumps on the system you can then simply run
# coredumpctl list
Or you can check for MariaDB Server core dumps specifically with:
# coredumpctl list /usr/sbin/mariadbd
If an actual core file got stored you'll see
present in the COREFILE column of the output, you can then extract the core file with:
# coredump dump -o mariadbd.core ...PID...
using the process id number from the PID column, or when you just want to retrieve the latest MariaDB Server related entry:
# coredump dump -o mariadb.core /usr/sbin/mariadbd
systemd 248 it is also possible to invoke the
gdb debugger directly using the new
--debugger-arguments=... option, e.g. making the extraction of all thread backtraces from the most recent MariaDB server crash a one liner without even having to extract the core dump file first (requires
gdb to be installed):
# coredumpctl debug --debugger-arguments="-batch -ex 'thread apply all bt full'" /usr/sbin/mariadbd
So far none of the long term support Linux distribution releases have a new enough
systemd version for this, the (as of this writing) still upcoming Ubuntu 22.04 "Jammy Jellyfish" will probably the first to support this.
Core Dumps and setuid on Linux
You can temporarily set it to
2 by using the
sysctl utility. For example:
sudo sysctl -w fs.suid_dumpable=2
Or you can temporarily set it to
2 by writing to the
/proc file system. For example:
sudo echo 2 > /proc/sys/fs/suid_dumpable
If you want to permanently set it to
2 then you can add the following to a file in
sudo tee /etc/sysctl.d/mariadb_fs_suid_dumpable.conf <<EOF fs.suid_dumpable=2 EOF
Note: If you don't want to change
fs.suid_dumpable, then another solution is to start
mysqld directly as the
mysql user, so that the
setuid call is not needed.
Forcing a Core File on Linux
To force a core file for
mysqld you can send the process the
sigabrt signal, which has the signal code
6. This is very useful to get the state of the unresponsive
mysqld process. However, this will cause
mysqld to crash, and crash recovery will be run on restart.
You can send the signal with the
kill command. For example:
sudo kill -6 $(pidof mysqld)
Core Files on Windows
There are some additional details related to using core files on Windows.
Core Files on Kubernetes
See the IBM Core Dump Handler project.
Minidump Files on Windows
On Windows, the core file is created as a minidump file.
For details on how to configure and read the minidump file, see How to read the small memory dump file that is created by Windows if a crash occurs.
Core Files and Address Sanitizer (ASAN)
mysqld binary is built with Address Sanitizer (ASAN) then it will not be able to generate a core file.
What's Included in Core Files
Core files usually contain a dump of all memory in the process's full address space. This means that if a server has some large buffers configured (such as a large InnoDB buffer pool), then the server's core files can get very large.
However, in MariaDB 10.3.7 and later, some large buffers have been excluded from core files on some systems as a way to reduce the size.
The following buffers are excluded:
The buffers are only excluded on Linux when using kernel version 3.4 and above and when using a non-debug build of
mysqld. Some Linux kernel versions have a bug which would cause the following warning to be printed to the log:
Sep 25 10:41:19 srv1 mysqld: 2018-09-25 10:41:19 0 [Warning] InnoDB: Failed to set memory to DODUMP: Invalid argument ptr 0x2aaac3400000 size 33554432
In those cases, the core dump may exclude some additional data. If that is not a concern, then the warning can be ignored. The problem can be fixed by upgrading to a Linux kernel version in which the bug is fixed.