# Debugging a Running Server (on Linux)

{% hint style="info" %}
This page is focused on lower-level development: It contains technical "how-to" documentation for engineers who are actually modifying the MariaDB source code (e.g., compiling for debug, using GDB, and memory leak detection in the code).
{% endhint %}

Even if you don't have a server that is [compiled for debugging](/docs/server/reference/product-development/mariadb-fault-finding/compiling-mariadb-for-debugging.md), there are still ways to get more information out of it if things go wrong.

When things go wrong, it's always better to have a version of the mysqld daemon that is not stripped.

```
shell> file /usr/sbin/mysqld
```

If this doesn't say 'stripped', then you are fine. If not, you should either [download a binary with debugging information](https://downloads.mariadb.org) or [compile it, without stripping the binary](/docs/server/reference/product-development/mariadb-fault-finding/compiling-mariadb-for-debugging.md#building-with-debug-symbols).

### Debugging Memory Consumption With tcmalloc

Read the [Profiling Memory Usage](/docs/general-resources/community/community/bug-tracking/profiling-memory-usage.md) page for more information on how to debug high memory consumption.

If you have a problem with a mysqld process that keeps on growing, you can use tcmalloc to find out\
what is allocating memory:

Depending on the system, you have to install the `tcmalloc` (openSUSE) or the `google-perftools-lib` (RedHat, CentOS) package.

The following set of commands starts mysqld with memory profiling, and if you kill it with SIGABRT, you will get a core dump that you can examine:

```
HEAPPROFILE=/tmp/mysqld.prof /usr/sbin/mysqld_safe --malloc-lib=tcmalloc --core-file-size=unlimited --core-file
```

or if you prefer to invoke mysqld directly:

```
ulimit -c unlimited
LD_PRELOAD=/usr/lib64/libtcmalloc.so.4 HEAPPROFILE=/tmp/mysqld.prof /usr/sbin/mysqld --core-file
```

You can, of course, add other [mysqld options](/docs/server/server-management/starting-and-stopping-mariadb/mariadbd-options.md) to the end of the above line.

Now start your client/application that uses MariaDB. You can find where memory is allocated in the `/tmp/mysqld.prof` file. If you find any memory issues, please report them in the [MariaDB bug tracker](https://jira.mariadb.org/secure/Dashboard.jspa)!

### ptrace Protection and Attaching GDB to a mysqld Instance

New Ubuntu releases do not allow one process to examine the memory of an\
arbitrary user's process. As a result, when trying to attach GDB to a running\
MariaDB (or any other process) instance, one gets the following error in GDB:

```
ptrace: Operation not permitted
```

More details are available in the [Ubuntu Wiki](https://wiki.ubuntu.com/SecurityTeam/Roadmap/KernelHardening#ptrace%20Protection).

To allow GDB to attach, one needs to edit the value of the`/proc/sys/kernel/yama/ptrace_scope` sysctl value.

* To change it temporarily, open a root shell and issue:

```
echo 0 > /proc/sys/kernel/yama/ptrace_scope
```

* To change it permanently, edit as root:

```
/etc/sysctl.d/10-ptrace.conf
```

and set the value to `0`.

### Debugging a Server That Hangs

If your mysqld server hangs, you may want to debug it to know what happened.

Preferably, the server should be compiled for debugging, but it's not strictly necessary:

```
cmake -DCMAKE_BUILD_TYPE=Debug -DWITH_VALGRIND=ON .
make -j4
```

To know what the server is doing:

* Find out the process number of mysqld

```
ps -edalf | grep mysqld
```

* Attach to the process and get a back trace:

```
gdb -p 'pid of mysqld' path-to-mysqld
set height 0
set logging file /tmp/mysqld.log
set logging on
thread apply all backtrace full
```

After the above, you have a full backtrace, including all local variables, in the `mysqld.log` file. Note that you will only get all variables if the server is not stripped.

<sub>*This page is licensed: CC BY-SA / Gnu FDL*</sub>

{% @marketo/form formId="4316" %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://mariadb.com/docs/server/reference/product-development/mariadb-fault-finding/debugging-a-running-server-on-linux.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.