Profiling with Linux perf tool

Linux perf tool can be used to do non-intrusive profiling.

Frequency Based Sampling

This mechanism can be used to answer the question, what is MariaDB doing on my CPU.

Recording a sample

Perf records at a high frequency, so only a short recording is sufficient to answer the question. If this is too short adjust the frequency at which perf does its recordings.

The -g option here records the calling stack. Because seeing time in a mutex function isn't particularly interesting without knowing which mutex it is.

Viewing a sample

To view a recording, you need the debug symbols for your executable. See on getting the debug symbols available.

Changing the ownership of the recording means you can run perf report without sudo.

This shows where in the process MariaDB is spending most of its time at the top level. MariaDB uses threads per user connection so this will usually show a significant time in a handle_connection function. There are background threads that also run, so this can quickly show if its connection related time or a background thread.

To see which low level functions are consuming the most time, --no-children means that each function listed include only the time that is being spend it this function and excluding the other functions it calls.

Expanding out the function shows the complete call stack again. Multiple functions may have called the function you are looking at so there may be a different frequency breakdown.

A more complete example of performance analysis using perf is on this .

Dynamic Tracepoints

Adding dynamic tracepoints

One can add tracepoints at function entry/exit (and other locations too):

Viewing the tracepoints

Running the profiler

Something like:

Note: -a means system-wide.

There's also -p $PID option

Examining the trace

_{This page is licensed: CC BY-SA / Gnu FDL}