Recommended Settings for Benchmarks
Contents
Running benchmarks requires a lot of different settings. In this article we collect our best known settings and recommendations.
Hardware and BIOS settings
We have had good experiences with Intel's hyperthreading on newer Xeon CPUs. Please turn on hyperthreading in your BIOS.
NUMA
The NUMA architecture attaches resources (most important: memory) to individual NUMA nodes (typically: NUMA node = cpu socket). This results in a performance penalty when a cpu core from one NUMA node accesses memory from another NUMA node.
The NUMA topology can be checked with the numactl command:
~ $numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 12 13 14 15 16 17 node 0 size: 12278 MB node 0 free: 11624 MB node 1 cpus: 6 7 8 9 10 11 18 19 20 21 22 23 node 1 size: 12288 MB node 1 free: 11778 MB node distances: node 0 1 0: 10 21 1: 21 10
The Linux kernel uses ACPI tables from BIOS to detect if the hardware is NUMA. On NUMA hardware extra optimizations kick in:
- if a task has been scheduled on a certain NUMA node, the scheduler tries to put it on the same node again in the future
- if a task running on a certain NUMA node allocates memory, the kernel tries hard to map physical memory from the same NUMA node
This results in all kinds of weird behavior when you run one big process (mysqld) that consumes most of the memory. In such cases it is recommended to either turn off NUMA (BIOS or kernel command line) or prefix such problem processes with numactl --interleave all
more details can be found here
Linux kernel settings
IO scheduler
For optimal IO performance running a database we are using the noop scheduler. Recommended schedulers are noop and deadline. You can check your scheduler setting with:
cat /sys/block/${DEVICE}/queue/scheduler
For instance, it should look like this output:
cat /sys/block/sda/queue/scheduler [noop] deadline cfq
You can find detailed notes about Linux schedulers here: Linux schedulers in TPCC like benchmark.
Open file limits
The open file limits is not a performance relevant setting, but running a benchmark with a lot of concurrent users can hit the open file limit quite easy.
On most Linux systems the open file limit is at 1024, which can be not enough. Please set your open file limit higher by editing
$EDITOR /etc/security/limits.conf
and adding a line like
#ftp hard nproc 0 #@student - maxlogins 4 * - nofile 16384 # End of file
Your ""ulimit -a"" output should look like this afterwards:
ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 15975 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) 1744200 open files (-n) 16384
InnoDB settings
innodb_buffer_pool_size to about 80% of RAM or leaving <5G RAM free (on large RAM systems). Less if lots of connections are used.
innodb_log_file_size to be larger than the amount of writes in the test run or sufficient to cover several minutes of the test run at least.
MyISAM settings
General settings
threads_cache_size should be the same as max_connections (unless using thread pools)