Compile and Using MariaDB with Sanitizers (ASAN, UBSAN, TSAN, MSAN)

What are Sanitizers?

Sanitizers are open source runtime error detectors developed by Google that are enabled during the compile step. These sanitizers add extra code during compilation that will throw exceptions when certain errors are detected.

Sanitizers find server bugs bug instrumenting the runtime MariaDB during compile time, and performing runtime checks on the executed test. Sanitizers detect a variety of implementation defects:

The sanitizers for clang are:

• AddressSanitizer (aka ASAN)
• UndefinedBehaviourSanitizer (aka UBSAN)
• ThreadSanitizer (aka TSAN)
• MemorySanitizer (aka MSAN)

How to Compile MariaDB with Sanitizers

Before using ASAN locally, please ensure that it is installed on the system:

You can use one of the two following build commands:

cmake  -DWITH_ASAN=ON /source

Additionally, UBSAN, TSAN, and MSAN can be enabled in a similar way:

UBSAN:

cmake -DWITH_UBSAN=ON /source

TSAN:

cmake -DWITH_TSAN=ON /source

MSAN:

cmake -DWITH_MSAN=ON /source

Important to note MSAN requires instrumented c++ and system libraries to have a functioning build per container guide below.

Buildbot's MSAN container

The time consuming aspect of building with MSAN is having instrumented system libraries. To help with this MariaDB Foundation has built the MSAN container for buildbot, but this is reusable locally.

The MariaDB Buildbot MSAN container, quay.io/mariadb-foundation/bb-worker/debian12-msan-clang-20 a container with:

• the instrumented libraries required for MSAN
• a very modern clang version
• the latest stable Debian as a base image
• a compiled rr for debugging

The build of this container isn't hard coded to MSAN so it can be used for:

• A general Debian build container
• Using the gcc/g++ of Debian (by removing the CC and CXX environment variables
• Compiling with a latest clang
• Compiling with the ASAN/UBSAN feature available in the latest clang

Running the MSAN container

First, run the container where your current directory is the source directory:

podman run -v $PWD:/source:z \
  --rm \
  -ti \
  --entrypoint bash \
  --mount=type=tmpfs,tmpfs-size=10G,dst=/build \
  --workdir /build \
   quay.io/mariadb-foundation/bb-worker:debian12-msan-clang-20

The purposes of these, and other options include:

Notes:

• docker can be used instead of the lighter weight podman if you so desired.
• /dev/shm is mounted no-exec so it rr recording here cannot be replayed while in that location
• optionally can make /build a filesystem mount to preserve the build.

There are environment variables that affect the cmake configure options and mtr are:

CXX=clang++
CC=clang
MSAN_LIBDIR=/msan-libs
CMAKE_GENERATOR=Ninja
MSAN_SYMBOLIZER_PATH=/usr/bin/llvm-symbolizer-20
WSREP_PROVIDER=/usr/lib/galera/libgalera_smm.so
MTR_PARALLEL=auto
ASAN_OPTIONS=quarantine_size_mb=512:atexit=0:detect_invalid_pointer_pairs=3:dump_instruction_bytes=1:allocator_may_return_null=1
UBSAN_OPTIONS=print_stacktrace=1:report_error_type=1
MSAN_OPTIONS=abort_on_error=1:poison_in_dtor=0

Note: galera WSREP_PROVIDER isn't instrumented with any sanitizer

Check the latest version of these running env.

Running an MSAN Build

Once you have started the MSAN container this is how you perform a MSAN build.

The time consuming aspect of building with MSAN is having instrumented system libraries.

All the following instructions are executed within the container - there is a document in /etc/motd that contains the latest information. Start by running the configure stage of cmake:

cmake \
      -DUPDATE_SUBMODULES=OFF \
      -DWITH_MSAN=ON  \
      -DCMAKE_{EXE,MODULE}_LINKER_FLAGS="-L${MSAN_LIBDIR} -Wl,-rpath=${MSAN_LIBDIR}" \
      -DHAVE_CXX_NEW=1  \
      -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \
      -DWITH_ZLIB=bundled  \
      -DWITH_NUMA=NO  \
      -DWITH_SYSTEMD=no \
      -DHAVE_LIBAIO_H=0    \
      -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1  \
      -DPLUGIN_{MROONGA,ROCKSDB,OQGRAPH}=NO  \
      -DWITH_EMBEDDED_SERVER=OFF \
      /source

MSAN is a clang only compile options and other compilers will not work.

Run the build stage:

cmake --build . 
...
[100%] Built target mariadbd
[100%] Linking CXX executable mariadb-backup
Creating mariabackup link
[100%] Built target mariadb-backup

As the MSAN has rpath defined in its compile option there is no need for LD_LIBRARY_PATH manipulation.

To run tests:

mysql-test/mtr   (usual mtr options)

As newer versions occur and improvements happen these instructions may change. Look at the execution on the buildbot builder to see if any changes have been made.

Bug Reporting

Use the MSAN in the label when reporting bugs.

Current outstanding MSAN bugs can be found on JIRA by searching for this label.

Running an combined ASAN and UBSAN Build

The clang implemented instrumentation of the MemoryStanitizer (MSAN) conflicts with the AddressSanitizer (ASAN) and UndefinedBehaviour (UBSAN) mechanisms, however ASAN and UBSAN can be combined into the same build.

After starting the MSAN container the following will configure a combined ASAN and UBSAN build:

cmake \
      -DUPDATE_SUBMODULES=OFF \
      -DWITH_ASAN=ON  \
      -DWITH_ASAN_SCOPED=ON \
      -DWITH_UBSAN=ON  \
      -DWITH_UNIT_TESTS=OFF \
     /source

Build and test run are standard.

Note: list of unfixed UBSAN issues by MariaDB Corporation Testing.

Bug Reporting

UBSAN bugs are labelled with UBSAN and also the short form of the undefined behaviour.

For example in the following output, the insufficient-object-size would be used as the additional short form of the label.

SUMMARY: UndefinedBehaviorSanitizer: insufficient-object-size /source/sql/item_strfunc.cc:5256:44 
 /source/sql/item_strfunc.cc:5256:44 

Likewise for ASAN, should use ASAN as the label with the short form of the ASAN type from the output.

Using RR

In the MSAN container there is a build version of the latest rr at the time of the container build.

A recording of the execution can be make with the mariadb-test-run.pl option --rr. Or alternately you can execute:

$ rr record sql/mariadbd ......

To replay a recording adjust per the test worker and instance used in the test:

$ rr replay mysql-test/var/log/mysqld.1.rr/mariadbd-0/

Using curl in the container its possible to save up this directory for another user or developer to use with the same container. This can also be shared with our public ftp server for assistance diagnosing validating a bug report.

ASAN Options

To run mariadbd with instrumentation you have to set the ASAN_OPTIONS environment variable before starting mariadbd including starting mariadbd when running mtr.

export ASAN_OPTIONS=abort_on_error=1

The above command will abort any instrumented executable if any errors are found, which is good for debugging. If you set abort_on_error=0 all server errors are logged to your error log file (mysqld.err).

To catch errors for other processes than the server, you can set more options, like this:

export ASAN_OPTIONS=abort_on_error=1:log_path=/tmp/asan

If you are seeing an incomplete stack trace for a memory allocation, you may rerun the failing test with

export ASAN_OPTIONS=abort_on_error=1:log_path=/tmp/asan:fast_unwind_on_malloc=0

To get core dumps of failures:

export ASAN_OPTIONS=abort_on_error=1:disable_coredump=0

To see all the options (or to check if an executable is instrumented), you may try the following:

ASAN_OPTIONS=help=1 extra/perror 0

Using Valgrind

The MariaDB test system can use Valgrind for finding memory leaks and wrong memory accesses. Valgrind is an instrumentation framework for building dynamic analysis tools. If Valgrind is installed on your system, you can simply use mysql-test-run --valgrind to run the test under Valgrind.

See Also

• Compiling MariaDB for debugging