# Compile and Using MariaDB with Sanitizers (ASAN, UBSAN, TSAN, MSAN)
## What are Sanitizers?
Sanitizers are open source runtime error detectors developed by Google that are enabled during the compile step. These sanitizers add extra code during compilation that will throw exceptions when certain errors are detected.
Sanitizers find server bugs bug instrumenting the runtime MariaDB during compile time, and performing runtime checks on the executed test. Sanitizers detect a variety of implementation defects:
The sanitizers for clang are:
* [AddressSanitizer (aka ASAN)](https://clang.llvm.org/docs/AddressSanitizer.html)
* [UndefinedBehaviourSanitizer (aka UBSAN)](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html)
* [ThreadSanitizer (aka TSAN)](https://clang.llvm.org/docs/ThreadSanitizer.html)
* [MemorySanitizer (aka MSAN)](https://clang.llvm.org/docs/MemorySanitizer.html)
## How to Compile MariaDB with Sanitizers
Before using ASAN locally, please ensure that it is installed on the system:
You can use one of the two following build commands:
```bash
cmake -DWITH_ASAN=ON /source
```
Additionally, UBSAN, TSAN, and MSAN can be enabled in a similar way:
UBSAN:
```
cmake -DWITH_UBSAN=ON /source
```
TSAN:
```
cmake -DWITH_TSAN=ON /source
```
MSAN:
```
cmake -DWITH_MSAN=ON /source
```
Important to note MSAN requires instrumented c++ and system libraries to have a functioning build per container guide below.
## Buildbot's `MSAN` container
The time consuming aspect of building with MSAN is having [instrumented system libraries](https://github.com/MariaDB/buildbot/blob/main/ci_build_images/msan.instrumentedlibs.sh). To help with this MariaDB Foundation has built the MSAN container for `buildbot`, but this is reusable locally.
The MariaDB Buildbot MSAN container, `quay.io/mariadb-foundation/bb-worker:debian12-msan-clang-20` is a container with:
* the instrumented libraries required for MSAN
* a very modern `clang` version
* a stable Debian as a base image
* a compiled [rr](https://rr-project.org/) for debugging
The build of this container isn't hard coded to MSAN so it can be used for:
* A general Debian build container
* Using the `gcc/g++` of Debian (by removing the CC and CXX environment variables
* Compiling with a latest `clang`
* Compiling with the ASAN/UBSAN feature available in the latest `clang`
### Running the MSAN container
First, run the container where your current directory is the source directory.
For Podman:
```bash
podman run -v $PWD:/source:z \
--rm \
-ti \
--entrypoint bash \
--mount=type=tmpfs,tmpfs-size=10G,dst=/build \
--workdir /build \
quay.io/mariadb-foundation/bb-worker:debian12-msan-clang-20
```
The `/build` options for Docker Engine are slightly different as they default to `noexec` and a `root` ownership by default.
```bash
docker run -v $PWD:/source:z \
--rm \
-ti \
--entrypoint bash \
--tmpfs /build:size=10G,exec,uid=1000 \
--workdir /build \
quay.io/mariadb-foundation/bb-worker:debian12-msan-clang-20
```
The purposes of these, and other options include:
| Command Component | Purpose | Notes |
| ---------------------------------------------- | --------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| `podman run / docker run` | run a container | other implementations use docker syntax (or at least close) |
| `--rm` | remove container on termination | |
| `-ti` | a `tty` and a `stdin` are connected | for interactive container use |
| `-v "$PWD":/source:z` | mount current directory as /source inside the container - :z - read selinux label | |
| `--mount=type=tmpfs,tmpfs-size=10G,dst=/build` | a 10G build directory and `mtr` under `podman` | keeps as transient, big enough for some `mtr` tests |
| `--tmpfs /build:size=10G,exec,uid=1000` | a 10G build directory and `mtr` for docker | keeps as transient, big enough for some mtr tests, exec and uid needed by build/tests |
| `--workdir /build` | default directory | |
| `--entrypoint bash` | Ensure the cmd of `buildbot` start isn't executed | |
| `--cap-add=SYS_PTRACE` | capability for tracing used by `gdb` and `rr` | for debugging |
| Extra options | | |
| `--name containername` | useful if running multiple to keep track | Used as a name for a new session in the container (podman exec -ti containername bash) |
| `--shm-size=10g` | Size of `/dev/shm` as alternate for mtr tests | Default is unsable 64k, This is large enough for most `--big-tests` with some parallelism |
| `--privileged` | Allow `rr` recording | Note security impacting, don't run untrusted code as root |
| `-v $DATADIR:/var/lib/mysql:Z` | Mount an existing `datadir` | For testing against some prepared data |
Notes:
* `/dev/shm` is mounted no-exec so it rr recording here cannot be replayed while in that location
* optionally can make `/build` a filesystem mount to preserve the build.
There are environment variables that affect the `cmake` configure options and `mtr` are:
```bash
CXX=clang++
CC=clang
MSAN_LIBDIR=/msan-libs
CMAKE_GENERATOR=Ninja
MSAN_SYMBOLIZER_PATH=/usr/bin/llvm-symbolizer-20
WSREP_PROVIDER=/usr/lib/galera/libgalera_smm.so
MTR_PARALLEL=auto
ASAN_OPTIONS=quarantine_size_mb=512:atexit=0:detect_invalid_pointer_pairs=3:dump_instruction_bytes=1:allocator_may_return_null=1
UBSAN_OPTIONS=print_stacktrace=1:report_error_type=1
MSAN_OPTIONS=abort_on_error=1:poison_in_dtor=0
```
Note: Galera `WSREP_PROVIDER` isn't instrumented with any sanitizer
Check the latest version of these running `env`.
### Running an MSAN Build
Once you have started the MSAN container this is how you perform a MSAN build.
The time consuming aspect of building with MSAN is having [instrumented system libraries](https://github.com/MariaDB/buildbot/blob/main/ci_build_images/msan.instrumentedlibs.sh).
All the following instructions are executed within the container - there is a document in `/etc/motd` that contains the latest information. Start by running the configure stage of `cmake`:
```bash
cmake \
-DUPDATE_SUBMODULES=OFF \
-DWITH_MSAN=ON \
-DCMAKE_{EXE,MODULE}_LINKER_FLAGS="-L${MSAN_LIBDIR} -Wl,-rpath=${MSAN_LIBDIR}" \
-DHAVE_CXX_NEW=1 \
-DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \
-DWITH_ZLIB=bundled \
-DWITH_NUMA=NO \
-DWITH_SYSTEMD=no \
-DHAVE_LIBAIO_H=0 \
-DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \
-DPLUGIN_{MROONGA,ROCKSDB,OQGRAPH}=NO \
-DWITH_EMBEDDED_SERVER=OFF \
/source
```
MSAN is a clang only compile options and other compilers will not work.
| CMake Options | Why |
| ----------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| `UPDATE_SUBMODULES=OFF` | The source directory is read-only so cannot be updated from within the container |
| `WITH_MSAN=ON` | Enables MSAN build in compile options |
| `CMAKE_{EXE,MODULE}_LINKER_FLAGS` | links executables and shared libraries against the instrumented libraries with a `rpath` so a `LD_LIBRARY_PATH` environment variable isn't required |
| `WITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF` | system libraries for these haven't been MSAN instrumented currently |
| `HAVE_CXX_NEW` | Works around a ODR violation |
| `WITH_ZLIB=bundled` | This hasn't been MSAN instrumented currently |
| `WITH_NUMA / WITH_SYSTEMD` | both uninstrumented |
| `HAVE_LIBAIO_H=0 / CMAKE_DISABLE_FIND_PACKAGE_LIBAIO` | AIO currently not MSAN instrumented |
| `CMAKE_DISABLE_FIND_PACKAGE_URING=NO` | Uninstrumented [MDEV-36482](https://jira.mariadb.org/browse/MDEV-36482), and required `seccomp` adjustment or `--privileged` to work |
| `PLUGIN_OQGRAPH (and PLUGIN_COLUMNSTORE)` | Dependency on boost libraries are instrumented |
| `WITH_EMBEDDED_SERVER=OFF` | reduce build time |
| `WITH_DBUG_TRACE=OFF` | reduce runtime overhead if `CMAKE_BUILD_TYPE=Debug` chosen |
Run the build stage:
```bash
cmake --build .
```
```ini
...
[100%] Built target mariadbd
[100%] Linking CXX executable mariadb-backup
Creating mariadb-backup link
[100%] Built target mariadb-backup
```
As the MSAN has `rpath` defined in its compile option there is no need for `LD_LIBRARY_PATH` manipulation.
To run tests:
```bash
mysql-test/mtr (usual mtr options)
```
As newer versions occur and improvements happen these instructions may change. Look at the execution on the [buildbot builder](https://buildbot.mariadb.org/#/builders/amd64-debian-11-msan) to see if any changes have been made.
#### Bug Reporting
Use the `MSAN` in the label when reporting bugs.
Current outstanding MSAN bugs can be found on [JIRA by searching for this label](https://jira.mariadb.org/issues/?jql=labels%20%3D%20MSAN%20and%20status%20!%3D%20Closed).
### Running an combined ASAN and UBSAN Build
The clang implemented instrumentation of the MemorySanitizer (MSAN) conflicts with the AddressSanitizer (ASAN) and UndefinedBehaviour (UBSAN) mechanisms, however ASAN and UBSAN can be combined into the same build.
After starting the MSAN container the following will configure a combined ASAN and UBSAN build:
```bash
cmake \
-DUPDATE_SUBMODULES=OFF \
-DWITH_ASAN=ON \
-DWITH_ASAN_SCOPED=ON \
-DWITH_UBSAN=ON \
-DWITH_UNIT_TESTS=OFF \
/source
```
| CMake Options | Why |
| ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `UPDATE_SUBMODULES=OFF` | The source directory is read-only so cannot be updated from within the container |
| `WITH_ASAN=ON` | Enables Address Sanitizer build in compile options |
| `WITH_ASAN_SCOPED=ON` | Enables Address Sanitizer Use after Scope checking |
| `WITH_UBSAN=ON` | Enables UBSAN build in compile options |
| `WITH_UNIT_TESTS=OFF or PLUGIN_PERFSCHEMA=OFF` | `clang` is incompatible with performance schema unit tests, at least one must be disabled (ref: [MDEV-22940](https://jira.mariadb.org/browse/MDEV-22940)) |
Build and test run are standard.
Note: list of unfixed [UBSAN issues by MariaDB Corporation Testing](https://github.com/mariadb-corporation/mariadb-qa/blob/master/UBSAN.filter).
#### Bug Reporting
UBSAN bugs are labelled with `UBSAN` and also the short form of the undefined behaviour.
For example in the following output, the `insufficient-object-size` would be used as the additional short form of the label.
```ini
SUMMARY: UndefinedBehaviorSanitizer: insufficient-object-size /source/sql/item_strfunc.cc:5256:44
/source/sql/item_strfunc.cc:5256:44
```
Likewise for ASAN, should use `ASAN` as the label with the short form of the ASAN type from the output.
### Using `RR`
In the MSAN container, there is a build version of the latest `rr` at the time of the container build.
A recording of the execution can be make with the [mariadb-test-run.pl option --rr](https://mariadb.com/docs/server/clients-and-utilities/testing-tools/mariadb-test/mariadb-test-run-pl-options). Or alternately you can execute:
```bash
$ rr record sql/mariadbd ......
```
To replay a recording adjust per the test worker and instance used in the test:
```bash
$ rr replay mysql-test/var/log/mysqld.1.rr/mariadbd-0/
```
Using curl in the container its possible to save up this directory for another user or developer to use with the same container. This can also be shared with our public ftp server for assistance diagnosing validating a bug report.
## `ASAN` Options
To run `mariadbd` with instrumentation you have to set the `ASAN_OPTIONS` environment variable before starting `mariadbd` including starting `mariadbd` when running mtr.
```bash
export ASAN_OPTIONS=abort_on_error=1
```
The above command will abort any instrumented executable if any errors are found, which is good for debugging. If you set `abort_on_error=0` all server errors are logged to your error log file (`mysqld.err`).
To catch errors for other processes than the server, you can set more options, like this:
```bash
export ASAN_OPTIONS=abort_on_error=1:log_path=/tmp/asan
```
If you are seeing an incomplete stack trace for a memory allocation, you may rerun the failing test with
```bash
export ASAN_OPTIONS=abort_on_error=1:log_path=/tmp/asan:fast_unwind_on_malloc=0
```
To get core dumps of failures:
```bash
export ASAN_OPTIONS=abort_on_error=1:disable_coredump=0
```
To see all the options (or to check if an executable is instrumented), you may try the following:
```bash
ASAN_OPTIONS=help=1 extra/perror 0
```
### Using Valgrind
The [MariaDB test system](https://mariadb.com/docs/server/clients-and-utilities/testing-tools/mariadb-test) can use [Valgrind](https://www.valgrind.org) for finding memory leaks and wrong memory accesses. Valgrind is an instrumentation framework for building dynamic analysis tools. If Valgrind is installed on your system, you can simply use [mysql-test-run --valgrind](https://mariadb.com/docs/server/clients-and-utilities/testing-tools/mariadb-test/mariadb-test-run-pl-options) to run the test under Valgrind.
## See Also
* [Compiling MariaDB for debugging](https://mariadb.com/docs/server/server-management/install-and-upgrade-mariadb/compiling-mariadb-from-source/legacy-guides/broken-reference)
*This page is licensed: CC BY-SA / Gnu FDL*
{% @marketo/form formId="4316" %}
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://mariadb.com/docs/server/server-management/install-and-upgrade-mariadb/compiling-mariadb-from-source/legacy-guides/compile-and-using-mariadb-with-sanitizers-asan-ubsan-tsan-msan.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.