> For the complete documentation index, see [llms.txt](https://mariadb.com/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://mariadb.com/docs/server/reference/product-development/mariadb-fault-finding/enabling-core-dumps.md).

# Enabling Core Dumps

How to configure the operating system and MariaDB to produce a core dump file when a crash occurs for post-mortem analysis.

{% hint style="info" %}
This page is focused on lower-level development: It contains technical "how-to" documentation for engineers who are actually modifying the MariaDB source code (e.g., compiling for debug, using GDB, and memory leak detection in the code).
{% endhint %}

## Enabling in an Option File

Core dumps are enabled by default on **Windows**, so **this step can be skipped on Windows**. See [MDEV-18439](https://jira.mariadb.org/browse/MDEV-18439) for more information.

On Linux, whether a crash produces a core file is decided by the operating system: the core file size limit of the `mariadbd` process and the `kernel.core_pattern` setting (see [Core Files on Linux](#core-files-on-linux)). When `mariadbd` crashes, it raises the fatal signal again after printing its crash report, so the kernel can write a core dump whether or not [core\_file](https://mariadb.com/docs/server/server-management/variables-and-modes/server-system-variables.md#core_file) is set. Enabling [core\_file](https://mariadb.com/docs/server/server-management/variables-and-modes/server-system-variables.md#core_file) also does the following:

* It tries to raise the core file size limit of the `mariadbd` process to unlimited. Unless `mariadbd` runs as `root`, this works only if the hard limit is already unlimited.
* It keeps the process dumpable after `mariadbd` switches from `root` to another user, as it does when [mariadbd-safe](https://mariadb.com/docs/server/server-management/starting-and-stopping-mariadb/mariadbd-safe.md) starts it. Without this, Linux doesn't write a core file of a process that changed its user (see [Core Dumps and setuid on Linux](#core-dumps-and-setuid-on-linux)).

The MariaDB [systemd](https://mariadb.com/docs/server/server-management/starting-and-stopping-mariadb/systemd.md) service starts `mariadbd` as the `mysql` user, so only the first point applies there, and the core file size limit is better set in the service (see [Running mariadbd Using systemd](#running-mariadbd-using-systemd)). On Windows, [core\_file](https://mariadb.com/docs/server/server-management/variables-and-modes/server-system-variables.md#core_file) is what writes the minidump file.

To enable it, set the [core\_file](https://mariadb.com/docs/server/server-management/variables-and-modes/server-system-variables.md#core_file) system variable either on the command-line or in a relevant server [option group](https://mariadb.com/docs/server/server-management/install-and-upgrade-mariadb/configuring-mariadb/configuring-mariadb-with-option-files.md#option-groups) in an [option file](https://mariadb.com/docs/server/server-management/install-and-upgrade-mariadb/configuring-mariadb/configuring-mariadb-with-option-files.md). For example:

```
[mariadb]
...
core_file
```

You can check your current value by executing:

```
my_print_defaults --mariadbd
```

[core\_file](https://mariadb.com/docs/server/server-management/variables-and-modes/server-system-variables.md#core_file) is a system variable. Its value can be checked at runtime by executing the following:

```
SHOW GLOBAL VARIABLES LIKE 'core_file';
```

## Core Files on Linux

There are some additional details related to using core files on Linux.

### Disabling Core File Size Restrictions on Linux

On some systems there is a limit on the sizes of core files that can be dumped. You can check the system's current system-wide limit by executing the following:

```
ulimit -c
```

You can check the current limit of the `mariadbd` process specifically by executing the following:

```
sudo cat /proc/$(pidof mariadbd)/limits | grep "core file"
```

If you need to change the core size limit, the method you use depends on how you start `mariadbd`. See the sections below for more details.

The resource limits for the `mariadbd` process are printed to the [error log](https://mariadb.com/docs/server/server-management/server-monitoring-logs/error-log.md) when the `mariadbd` process crashes. That way, users can confirm whether the process may have been allowed to dump a core file. See [MDEV-15051](https://jira.mariadb.org/browse/MDEV-15051) for more information.

#### Running mariadbd Using mysqld\_safe

If you are starting MariaDB by running [mysqld\_safe](https://mariadb.com/docs/server/server-management/starting-and-stopping-mariadb/mariadbd-safe.md), then configuring the following in the `[mysqld_safe]` option group in an option file should allow for unlimited sized core files:

```
[mysqld_safe]
...
core_file_size=unlimited
```

You can check your current values by executing:

```
my_print_defaults mysqld_safe
```

See [mysqld\_safe: Configuring the Core File Size](https://mariadb.com/docs/server/server-management/starting-and-stopping-mariadb/mariadbd-safe.md) for more details.

**Note:** If you are using [mysqld\_safe](https://mariadb.com/docs/server/server-management/starting-and-stopping-mariadb/mariadbd-safe.md) and running `mariadbd` as the `root` user, then no core file is created on some systems. The solution is to run `mariadbd` as another user.

#### Running mariadbd Manually

If you are starting mariadbd manually or in a custom script, then you can allow for unlimited sized core files by executing the following in the same shell or script in which mariadbd is executed:

```
ulimit -c unlimited
```

#### Running mariadbd Using systemd

If you are starting `mariadbd` using [systemd](https://mariadb.com/docs/server/server-management/starting-and-stopping-mariadb/systemd.md), then you may need to customize the MariaDB service to allow for unlimited size core files. For example, you could execute the following:

Using `sudo systemctl edit mariadb.service` add the contents:

```
[Service]

LimitCORE=infinity
```

Then restart the service with `sudo systemctl restart mariadb.service`.

The `mariadb.service` unit doesn't set `LimitCORE` itself, so unless you add it, the service uses the system-wide systemd default. To check the limit that applies:

```
systemctl show mariadb.service -p LimitCORE -p LimitCORESoft
```

If it shows `0` or a small value, either add `LimitCORE=infinity` for the MariaDB service as above, or raise the default for all services by setting `DefaultLimitCORE=infinity` in `/etc/systemd/system.conf` and running `sudo systemctl daemon-reexec`. A changed default applies to services started afterwards, so restart MariaDB too.

See [systemd: Configuring the Core File Size](https://mariadb.com/docs/server/server-management/starting-and-stopping-mariadb/systemd/configuring.md#configuring-the-core-file-size) for more details.

#### Running MariaDB Containers

To get a core dump in a `mariadb` container requires [setting the path on Linux](#setting-the-path-on-linux) to not include a `sysctl kernel.core_pattern` beginning with a pipe to an executable that doesn't exist in the container. Setting to a straight `core` is recommended.

Also see [Container with Debug Symbols](https://mariadb.com/docs/server/reference/product-development/mariadb-fault-finding/how-to-produce-a-full-stack-trace-for-mariadbd.md#containers-with-debug-symbols).

#### Changing the System-Wide Limit

If you want to change the system-wide limit to allow for unlimited size core files for the `mysql` user account, then you can do so by adding the following lines to a file in [/etc/security/limits.d/](https://linux.die.net/man/5/limits.conf). For example:

```
sudo tee /etc/security/limits.d/mariadb_core.conf <<EOF
mysql soft core unlimited
mysql hard core unlimited
EOF
```

The system would have to be restarted for this change to take effect.

See [Configuring Linux for MariaDB: Configuring the Core File Size](https://mariadb.com/docs/server/server-management/install-and-upgrade-mariadb/configuring-mariadb/mariadb-performance-advanced-configurations/configuring-linux-for-mariadb.md#configuring-the-core-file-size) for more details.

### Setting the Path on Linux

If you are using Linux, then it can be helpful to change a few settings to alter where the core files is written and what file name is used. This is done by setting the `kernel.core_pattern` and `kernel.core_uses_pid` attributes. You can check the current values by executing the following:

```
sysctl kernel.core_pattern
sysctl kernel.core_uses_pid
```

If you are using `mysql-test-run` and want to have the core as part of the test result, the optimal setting is probably the following (store cores in the current directory as `core.number-of-process-id`):

```
sudo sysctl kernel.core_pattern=core.%p kernel.core_uses_pid=0
```

If you are using a production system, you probably want to have the core files in a specific directory, not in the data directory. They place to store cores can be temporarily altered using the [sysctl](https://linux.die.net/man/8/sysctl) utility, but it is often more common to alter them via the [/proc](https://linux.die.net/man/5/proc) file system. See the following example:

```
sudo mkdir /tmp/corefiles
sudo chmod 777 /tmp/corefiles
sudo sysctl kernel.core_pattern=/tmp/corefiles/core
sudo sysctl kernel.core_uses_pid=1
```

The above commands will tell the system to put core files in `/tmp/corefiles`, and it also tells the system to put the process ID in the file name.

If you want to make these changes permanent, then you can add the following to a file in [/etc/sysctl.conf.d/](https://linux.die.net/man/5/sysctl.conf). For example:

```
sudo tee /etc/sysctl.d/mariadb_core.conf <<EOF
kernel.core_pattern=/tmp/corefiles/core
kernel.core_uses_pid=1
EOF
```

Note: if you are using containers, the pid is always going to be 1, so this may not be a useful setting. Appending an identifier like %t to the [kernel.core\_pattern](https://www.kernel.org/doc/html/latest/admin-guide/sysctl/kernel.html#core-pattern) will generate more unique files.

The value of `kernel.core_pattern` is printed to the [error log](https://mariadb.com/docs/server/server-management/server-monitoring-logs/error-log.md) when the `mariadbd` process crashes. That way, users can determine where the process may have dumped a core file. See [MDEV-15051](https://jira.mariadb.org/browse/MDEV-15051) for more information.

**Note:** Ensure that you have enough free disk space in the path pointed to by `kernel.core_pattern`.

#### Extracting Linux Core Dumps with systemd-coredump

Core dump management can be automated using [systemd](https://mariadb.com/docs/server/server-management/starting-and-stopping-mariadb/systemd.md), which then centrally manages all core dump files and provides information about detected core dumps and access to collected core files using the [coredumpctl](https://www.freedesktop.org/software/systemd/man/coredumpctl.html) utility.

This is enabled by default on Red Hat Enterprise Linux 8 and later and on distributions derived from it. To check whether it is enabled, look at the `kernel.core_pattern` setting. If it looks like this, systemd-coredump is enabled:

```
# sysctl kernel.core_pattern
kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e
```

Other distributions can pipe core dumps to a different handler, or have the kernel write core files itself. The `kernel.core_pattern` setting shows which applies.

{% hint style="info" %}
When `kernel.core_pattern` starts with a pipe (`|`), the kernel doesn't write a core file itself. It passes the core to the program named after the pipe. With systemd-coredump, that means:

* No core file appears in the data directory, even with [core\_file](https://mariadb.com/docs/server/server-management/variables-and-modes/server-system-variables.md#core_file) enabled. systemd-coredump stores the core in `/var/lib/systemd/coredump/` instead, compressed, and deletes it after a few days by default.
* Cores larger than the `ProcessSizeMax` and `ExternalSizeMax` limits in [coredump.conf](https://www.freedesktop.org/software/systemd/man/latest/coredump.conf.html) aren't stored. The largest buffers, such as the InnoDB buffer pool, are left out of core files (see [What's Included in Core Files](#whats-included-in-core-files)), but a busy `mariadbd` process can still exceed the limits, so raise both if `coredumpctl list` doesn't show `present` in the `COREFILE` column.

To get a core file, for example to send it to a support engineer, extract it with `coredumpctl dump`, as shown below. You don't need to change `kernel.core_pattern`.

The `Core pattern:` line that `mariadbd` writes to the [error log](https://mariadb.com/docs/server/server-management/server-monitoring-logs/error-log.md) when it crashes shows which case applies.
{% endhint %}

To see all recent core dumps on the system you can then simply run

```
# coredumpctl list
```

Or you can check for MariaDB Server core dumps specifically with:

```
# coredumpctl list /usr/sbin/mariadbd
```

If an actual core file got stored you'll see `present` in the COREFILE column of the output, you can then extract the core file with:

```
# coredumpctl dump -o mariadbd.core ...PID...
```

using the process id number from the PID column, or when you just want to retrieve the latest MariaDB Server related entry:

```
# coredumpctl dump -o mariadb.core /usr/sbin/mariadbd
```

`coredumpctl info` shows the details of the most recent crash, including the `Storage` line with the path of the stored, compressed core file. The path of `mariadbd` depends on the distribution:

```
# coredumpctl info mariadbd
...
      Storage: /var/lib/systemd/coredump/core.mariadbd.<uid>.<boot-id>.<pid>.<timestamp>.zst (present)
...
```

Starting with `systemd` 248 it is also possible to invoke the `gdb` debugger directly using the new `--debugger-arguments=...` option, e.g. making the extraction of all thread backtraces from the most recent MariaDB server crash a one liner without even having to extract the core dump file first (requires `gdb` to be installed):

```
# coredumpctl debug --debugger-arguments="-batch -ex 'thread apply all bt full'" /usr/sbin/mariadbd
```

This requires `systemd` 248 or later, as shipped with Ubuntu 22.04, Debian 12, and RHEL 9 and later.

### Core Dumps and setuid on Linux

When `mariadbd` is started as `root`, for example by [mariadbd-safe](https://mariadb.com/docs/server/server-management/starting-and-stopping-mariadb/mariadbd-safe.md), it switches to the `mysql` user with [setuid](https://linux.die.net/man/2/setuid), and you may have to set `fs.suid_dumpable=2` to allow core dumps on Linux. This doesn't apply to the MariaDB systemd service, which starts `mariadbd` as the `mysql` user. You can check the current `fs.suid_dumpable` value by using the [sysctl](https://linux.die.net/man/8/sysctl) utility. For example:

```
sysctl fs.suid_dumpable
```

You can temporarily set it to `2` by using the [sysctl](https://linux.die.net/man/8/sysctl) utility. For example:

```
sudo sysctl -w fs.suid_dumpable=2
```

Or you can temporarily set it to `2` by writing to the [/proc](https://linux.die.net/man/5/proc) file system. For example:

```
sudo echo 2 > /proc/sys/fs/suid_dumpable
```

If you want to permanently set it to `2` then you can add the following to a file in [/etc/sysctl.conf.d/](https://linux.die.net/man/5/sysctl.conf):

```
sudo tee /etc/sysctl.d/mariadb_fs_suid_dumpable.conf <<EOF
fs.suid_dumpable=2
EOF
```

**Note:** If you don't want to change `fs.suid_dumpable`, then another solution is to start `mariadbd` directly as the `mysql` user, so that the [setuid](https://linux.die.net/man/2/setuid) call is not needed.

### Forcing a Core File on Linux

You can get a core dump from a running server with:

```
sudo gcore -o filename $(pidof mariadbd)
```

This will store a core dump in `filename.pid` where pid is the process ID of `mariadbd`.`mariadbd` will continue to be running and will not be affected by `gcore`.

Another method is to force a core file for `mariadbd` by sending the process the `sigabrt` signal, which has the signal code `6`. This is very useful to get the state of the unresponsive `mariadbd` process. However, this will cause `mariadbd` to crash, and crash recovery will be run on restart.

You can send the signal with the [kill](https://linux.die.net/man/1/kill) command. For example:

```
sudo kill -6 $(pidof mariadbd)
```

As an alternative to `$(pidof mariadbd)`, you can find the process ID either by using the [ps](https://linux.die.net/man/1/ps) utility or by checking the file defined by the [pid\_file](https://mariadb.com/docs/server/server-management/variables-and-modes/server-system-variables.md#pid_file) system variable.

## Core Files on Windows

Core dumps are enabled by default on Windows. See [MDEV-18439](https://jira.mariadb.org/browse/MDEV-18439) for more information.

There are some additional details related to using core files on Windows.

### Minidump Files on Windows

On Windows, the core file is created as a [minidump file](https://docs.microsoft.com/en-us/windows/desktop/debug/minidump-files).

For details on how to configure and read the [minidump file](https://docs.microsoft.com/en-us/windows/desktop/debug/minidump-files), see [How to read the small memory dump file that is created by Windows if a crash occurs](https://support.microsoft.com/en-us/help/315263/how-to-read-the-small-memory-dump-file-that-is-created-by-windows-if-a).

## Core Files in Containers

`kernel.core_pattern` is a host setting, so the host decides what happens to the core dump of a `mariadbd` process in a container. See [Running MariaDB Containers](#running-mariadb-containers). If the host uses systemd-coredump, [systemd-coredump](https://www.freedesktop.org/software/systemd/man/latest/systemd-coredump.html) describes how it handles the core dumps of processes in containers.

## Core Files and Address Sanitizer (ASAN)

If your `mariadbd` binary is built with [Address Sanitizer (ASAN)](https://mariadb.com/docs/server/server-management/install-and-upgrade-mariadb/compiling-mariadb-from-source/compiling-mariadb-from-source-the-master-guide.md) then it will disable core dumps by default. A core dump can be created if `mariadbd` is started with the environment variable `ASAN_OPTIONS` set with `disable_coredump=0`.

## What's Included in Core Files

Core files usually contain a dump of all memory in the process's full address space. This means that if a server has some large buffers configured (such as a large [InnoDB buffer pool](https://mariadb.com/docs/server/server-usage/storage-engines/innodb/innodb-buffer-pool.md)), then the server's core files can get very large.

Some large buffers have been excluded from core files on some systems as a way to reduce the size.

The following buffers are excluded:

* [InnoDB buffer pool](https://mariadb.com/docs/server/server-usage/storage-engines/innodb/innodb-buffer-pool.md)
* [InnoDB log buffer](https://mariadb.com/docs/server/server-usage/storage-engines/innodb/innodb-system-variables.md#innodb_log_buffer_size)
* InnoDB Redo log buffer (fixed 2M)
* [Query cache](https://mariadb.com/docs/server/ha-and-performance/optimization-and-tuning/buffers-caches-and-threads/query-cache.md)

The buffers are only excluded on Linux when using kernel version 3.4 and above and when using a non-debug build of `mariadbd`. Some Linux kernel versions have a [bug](https://lists.launchpad.net/maria-discuss/msg05245.html) which would cause the following warning to be printed to the log:

```
Sep 25 10:41:19 srv1 mysqld: 2018-09-25 10:41:19 0 [Warning] InnoDB: Failed to set memory to DODUMP: Invalid argument ptr 0x2aaac3400000 size 33554432
```

In those cases, the core dump may exclude some additional data. If that is not a concern, then the warning can be ignored. The problem can be fixed by upgrading to a Linux kernel version in which the bug is fixed.

Leaving these buffers out also keeps most table data out of core files ([MDEV-10814](https://jira.mariadb.org/browse/MDEV-10814)), but not all of it: a core file still contains the data that queries were processing at the time of the crash, and memory that was reused can hold earlier data. Treat core files as confidential.

## See Also

* [How to Produce a Full Stack Trace for mariadbd](https://mariadb.com/docs/server/reference/product-development/mariadb-fault-finding/how-to-produce-a-full-stack-trace-for-mariadbd.md)

<sub>*This page is licensed: CC BY-SA / Gnu FDL*</sub>

{% @marketo/form formId="4316" %}
