Debugging Memory Usage

Instructions for monitoring and analyzing memory allocation to identify leaks or excessive consumption within the server.

This page is focused on system-level development: It is geared toward developers or DBAs trying to diagnose why a specific instance is failing or crashing in a production or testing environment (e.g., interpreting error messages, handling crashes).

Troubleshooting Memory Usage

This page describes how to debug MariaDB's memory usage. It uses CentOS 7 but can be applied to other systems as well.

Use Google PerfTools for debugging memory usage: heapprofile.html

On CentOS, install PerfTools like this:

sudo yum install gperftools 
service mariadb stop
systemctl edit mariadb

In the editor that opens, add and save this content:

[Service]
Environment="HEAPPROFILE=/tmp/heap-prof-1"
Environment="HEAP_PROFILE_ALLOCATION_INTERVAL=10737418240"
Environment="HEAP_PROFILE_INUSE_INTERVAL=1073741824"
Environment="LD_PRELOAD=/usr/lib64/libtcmalloc.so.4"

Then run this command:

service mariadb start

Now, run the workload. When memory consumption becomes large enough, run this command:

ls -la /tmp/heap-prof-*

This should show several files. Copy the last file from the list:

cp /tmp/heap-prof-1.0007.heap .

Then, run this command:

pprof --dot /usr/sbin/mysqld heap-prof-1.0007.heap  > 7.dot

This produces a lot of lines, like /bin/addr2line: Dwarf Error: ....

Then, please send us the 7.dot file.

Memory Leak Overview

Operating system memory can be assigned, giving MariaDB a virtual address space, and allocated (paged in), once MariaDB starts to use this memory.

When looking at something that might be a memory leak, pay particular attention between the virtual size and the resident size.

When MariaDB has the same virtual size, but the resident size increases over time, this is a good indication that the memory buffers allocated to MariaDB are slowly being used. In this case its unlikely to be a memory leak.

Because memory leaks are hard to track down, documenting a known good version and a version that is leaking memory can make it easier to examine code changes in those versions against some other criteria like those obtained from the sections below to identify the cause.

Performance Schema

The memory instrumentation of the performance schema can be enable with the following:

performance_schema=on
performance-schema-instrument='memory/%=ON'

This is available since MariaDB 10.5.2. With this enabled, performance_schema.memory_summary_global_by_event_name can start to show where the memory leak is occurring.

Memleak

memleak is a BPF program for Linux that is frequently packaged like bcc-tools or similar (bpfcc-tools on Debian).

The upstream documentation provides this example.

The important aspect on measurement is to let MariaDB startup, and commence an normal workload before measurement. Starting too early or before the caches are populated is likely to result in false recording of memory allocations that will be eventually released.

Increasing the time interval of memleak is important to run longer than the usual query. If your memory is leaking over hours, then a high interval of 10 minutes so will reduce memory temporary allocated for a query. Recording for a longer time or multiple times should provide enough information to narrow down the location of the leak.

Very old Linux kernels like Centos/RHEL 7 might not have sufficient hooks to measure this correctly.

_{This page is licensed: CC BY-SA / Gnu FDL}