Enabling Core Dumps
Contents
Enabling in Your Configuration File
Add the following to the end of your MariaDB configuration file:
[mysqld] stack-trace core-file disable-gdb
These options are safe to leave there.
You can check your current values by executing:
my_print_defaults mysqld
core_file has also been made into a system variable which can be checked at runtime:
SHOW GLOBAL VARIABLES LIKE 'core_file';
Disabling Size Restrictions
On some systems there is a limit on the sizes of core files that can be dumped. You can check the system's current system-wide limit by executing the following:
ulimit -c
You can check the current limit of the mysqld process specifically by executing the following:
cat /proc/$(pidof mysqld)/limits | grep "core file"
If you need to change the core size limit, the method you use depends on how you start mysqld. See the sections below for more details.
Running mysqld Using mysqld_safe
If you are starting MariaDB by running mysqld_safe, then configuring the following in your MariaDB configuration file should allow for unlimited sized core dumps:
[mysqld-safe] --core-file-size=unlimited
You can check your current values by executing:
my_print_defaults mysqld-safe
Note: If you are using mysqld_safe and running
mysqld as the root user, then no
core file is created on some systems. The solution
is to run mysqld as another user.
Running mysqld Manually
If you are starting mysqld manually or in a custom script, then you can allow for unlimited sized core dumps by executing the following in the same shell or script in which mysqld is executed:
ulimit -c unlimited
Running mysqld Using systemd
If you are starting mysqld using systemd, then you may need to customize the MariaDB service to allow for unlimited size core dumps. For example, you could execute the following:
tee /etc/systemd/system/mariadb.service.d/core_dumps.conf <<EOF [Service] LimitCORE=infinity EOF systemctl daemon-reload
See customizing MariaDB's systemd service for more details.
Changing the System-Wide Limit
If you want to change the system-wide limit to allow for unlimited size core dumps for all processes, then you can add the following to /etc/security/limits.conf:
* soft core unlimited * hard core unlimited
The system would have to be restarted for this change to take effect.
Setting the Path on Linux
If you are using Linux, then it can be helpful to change a few settings to alter where the core dump is written and what file name is used. This is done by setting the kernel.core_pattern and kernel.core_uses_pid attributes. You can check the current values by executing the following:
sysctl kernel.core_pattern sysctl kernel.core_uses_pid
They can also be temporarily altered using the sysctl utility, but it is often more common to alter them via the /proc file system. See the following example:
mkdir /tmp/corefiles chmod 777 /tmp/corefiles echo “/tmp/corefiles/core” > /proc/sys/kernel/core_pattern echo “1” > /proc/sys/kernel/core_uses_pid
The above commands will tell the system to put core files in /tmp/corefiles, and it also tells the system to put the process ID in the file name.
If you want to make these changes permanent, then you can add the following to /etc/sysctl.conf:
kernel.core_pattern=/tmp/corefiles/core kernel.core_uses_pid=1
Note: Ensure that you have enough free disk space in the path pointed to by kernel.core_pattern.
Note: If you are using systemd, then the default value of kernel.core_pattern may redirect core files to be written to the systemd journal. If so, they can be retrieved with coredumpctl list. You can change this behavior by changing kernel.core_pattern.
Note: When MariaDB is writing a core dump, the error log contains a message indicating that a core file is written in the datadir, but it might actually be written to the directory pointed to by kernel.core_pattern (see MDEV-15775).
Core Dumps and setuid on Linux
Since mysqld executes setuid, you may have to set fs.suid_dumpable=2 to allow core dumps on Linux. You can check the current fs.suid_dumpable value by executing the following:
sysctl fs.suid_dumpable
You can temporarily set it to 2 by executing the following:
sysctl -w fs.suid_dumpable=2
or by executing the following:
echo 2 > /proc/sys/fs/suid_dumpable
If you want to set it to 2 permanently then you can add the following to /etc/sysctl.conf:
fs.suid_dumpable=2
Note: If you don't want to change fs.suid_dumpable, then another solution is to start mysqld directly as the mysql user, so that the setuid call is not needed.
Other Tips
If the mysqld server is built with ASAN (Address Sanitizer) then it will not be able to generate a core file.
If you are still struggling to get a core file, you can test your setup by following the instructions in the "Forcing a Core File" section below.
Forcing a Core File
To force a core file for mysqld you can send the process the sigabrt (6) signal. This is very useful to get the state of the unresponsive mysqld process. Note that this will crash mysqld (like a kill -9) and crash recovery will be run on restart.
kill -6 `pidof mysqld`
As an alternative to `pidof mysqld`, you can find the process number either with ps or in the *.pid file that is in the datadir directory.
What's Included in Core Dumps
Core dumps usually contain a dump of all memory in the process's full address space. This means that if a server has some large buffers configured (such as a large InnoDB buffer pool), then the server's core dump can get very large.
However, in MariaDB 10.3.7 and above, some large buffers have been excluded from core dumps on some systems as a way to reduce the size.
The following buffers are excluded:
- InnoDB buffer pool
- InnoDB log buffer
- Server recv buffer
- Query cache
The buffers are only excluded on Linux when using kernel version 3.4 and above and when using a non-debug build of mysqld. Some Linux kernel versions have a bug which would cause the following warning to be printed to the log:
Sep 25 10:41:19 srv1 mysqld: 2018-09-25 10:41:19 0 [Warning] InnoDB: Failed to set memory to DODUMP: Invalid argument ptr 0x2aaac3400000 size 33554432
In those cases, the core dump may exclude some additional data. If that is not a concern, then the warning can be ignored. The problem can be fixed by upgrading to a Linux kernel version in which the bug is fixed.
