mariadb-test-run.pl Options
You are viewing an old version of this article. View
the current version here.
Contents
- Syntax
- Options
- Options to Control What Engine/Variation to Run
- Options to Control Directories to Use
- Options to Control What Test Suites or Cases to Run
- Options That Specify Ports
- Options For Test Case Authoring
- Options That Pass On Options
- Options to Run Test On Running Server
- Options For Debugging the Product
- Misc Debugging Related Options
- Misc Options
Syntax
./mariadb-test-run.pl [ OPTIONS ] [ TESTCASE ]
Where the test case can be specified as:
testcase[.test] Runs the test case named 'testcase' from all suits
path-to-testcase [suite.]testcase[,combination]
Examples
alias
main.alias 'main' is the name of the suite for the 't' directory.
rpl.rpl_invoked_features,mix,xtradb_plugin suite/rpl/t/rpl.rpl_invoked_features
Options
Options to Control What Engine/Variation to Run
| Option | Description |
|---|---|
--embedded-server | Use the embedded server, i.e. no mysqld daemons. |
--ps-protocol | Use the binary protocol between client and server. |
--cursor-protocol | Use the cursor protocol between client and server (implies --ps-protocol). |
--view-protocol | Create a view to execute all non updating queries. |
--sp-protocol | Create a stored procedure to execute all queries. |
--compress | Use the compressed protocol between client and server if both support it. |
--ssl | If mariadb-test-run.pl is started with the --ssl option, it sets up a secure connection for all test cases. In this case, if mysqld does not support TLS, mariadb-test-run.pl exits with an error message: Couldn´t find support for SSL. |
--skip-ssl | Dont start server with support for TLS connections. |
--vs-config | Visual Studio configuration used to create executables (default: MTR_VS_CONFIG environment variable). |
--parallel=num | How many parallel tests should be run. Default is 1, use --parallel=auto for auto-setting of num. |
--defaults-file=<config template> | Use fixed config template for all tests. |
--defaults_extra_file=<config template> | Extra config template to add to all generated configs. |
--combination=<opt> | Extra options to pass to mysqld. The value should consist of one or more comma-separated mysqld options. This option is similar to --mysqld but should be given two or more times. mariadb-test-run.pl executes multiple test runs, using the options for each instance of --combination in successive runs. If --combination is given only once, it has no effect. For test runs specific to a given test suite, an alternative to the use of --combination is to create a combinations file in the suite directory. The file should contain a section of options for each test run. |
--dry-run | Don't run any tests, print the list of tests that were selected for execution. |
Options to Control Directories to Use
| Option | Description |
|---|---|
--tmpdir=DIR | The directory where temporary files are stored (default: ./var/tmp). The environment variable MYSQL_TMP_DIR will be set to the path for this directory, whether it has the default value or has been set explicitly. This may be referred to in tests. |
--vardir=DIR | The directory where files generated from the test run is stored (default: ./var). Specifying a ramdisk or tmpfs will speed up tests. The environment variable MYSQLTEST_VARDIR will be set to the path for this directory, whether it has the default value or has been set explicitly. This may be referred to in tests. |
--mem | Run testsuite in "memory" using tmpfs or ramdisk. This can decrease test times significantly, in particular if you would otherwise be running over a remote file system. Attempts to find a suitable location using a builtin list of standard locations for tmpfs (/dev/shm). The option can also be set using environment variable MTR_MEM=[DIR]. If DIR is given, it is added to the beginning of the list of locations to search, so it takes precedence over any built-in locations. Once you have run tests with --mem within a mariadb-testdirectory, a soflink var will have been set up to the temporary directory, and this will be re-used the next time, until the soflink is deleted. Thus, you do not have to repeat the --mem option next time. |
--client-bindir=PATH | Path to the directory where client binaries are located. |
--client-libdir=PATH | Path to the directory where client libraries are located. |
Options to Control What Test Suites or Cases to Run
| Option | Description |
|---|---|
--force | Normally, mariadb-test-run.pl exits if a test case fails. --force causes execution to continue regardless of test case failure. |
--with-ndbcluster-only | Run only tests that include "ndb" in the filename. |
--skip-ndb[cluster] | Skip all tests that need cluster. Default. |
--do-test=PREFIX or REGEX | Run test cases with names prefixed with PREFIX or which fulfil the REGEX. For example, --do-test=testa matches tests that begin with testa, --do-test=main.testa matches tests in the main test suite that begin with testa, and --do-test=main.*testa matches test names that contain main followed by testa with anything in between. In the latter case, the pattern match is not anchored to the beginning of the test name, so it also matches names such as xmainytestz. |
--skip-test=PREFIX or REGEX | Skip test cases with names prefixed with PREFIX or which fulfil the REGEX. See -do-test for examples. |
--start-from=PREFIX | Sorts the list of names of the test cases to be run, and then starts with the test prefixed with PREFIX, where the prefix may be suite.testname or just testname. |
--suite[s]=NAME1,..,NAMEN | Comma separated list of suite names to run. The default, as of MariaDB 10.4.5, is: "main-, archive-, binlog-, binlog_encryption-, csv-, compat/oracle-, encryption-, federated-, funcs_1-, funcs_2-, gcol-, handler-, heap-, innodb-, innodb_fts-, innodb_gis-, innodb_zip-, json-, maria-, mariabackup-, multi_source-, optimizer_unfixed_bugs-, parts-, perfschema-, plugins-, roles-, rpl-, sys_vars-, sql_sequence-, unit-, vcol-, versioning-,period-". |
--skip-rpl | Skip the replication test cases. |
--big-test | Allow tests marked as "big" to run. Tests can be thus marked by including the line --source include/big_test.inc, and they will only be run if this option is given, or if the environment variable BIG_TEST is set to 1. Repeat this option twice to run only "big" tests. This is typically used for tests that take a very long to run, or that use many resources, so that they are not suitable for running as part of a normal test suite run |
--staging-run | Run a limited number of tests (no slow tests). Used for running staging trees with valgrind. |
--enable-disabled | Ignore any disabled.def file, and also run tests marked as disabled. Success or failure of those tests will be reported the same way as other tests. |
--print-testcases | Don't run the tests but print details about all the selected tests, in the order they would be run. |
--skip-test-list=FILE | Skip the tests listed in FILE. Each line in the file is an entry and should be formatted as: <TESTNAME> : <COMMENT> |
Options That Specify Ports
| Option | Description |
|---|---|
--[mtr-]port-base=num | Base for port numbers. Ports from this number to number+9 are reserved. Should be divisible by 10; if not it will be rounded down. May be set with environment variable MTR_PORT_BASE. If this value is set and is not "auto", it overrides build-thread. |
--[mtr-]build-thread=num | Specify unique number to calculate port number(s) from. Can be set in environment variable MTR_BUILD_THREAD. Set MTR_BUILD_THREAD="auto" to automatically acquire a build thread id that is unique to current host. The more logical --port-base is supported as an alternative. |
Options For Test Case Authoring
| Option | Description |
|---|---|
--record TESTNAME | (Re)generate the result file for TESTNAME. |
--check-testcases | Check testcases for side-effects. This is done by checking system state before and after each test case; if there is any difference, a warning to that effect will be written, but the test case will not be marked as failed because of it. This check is enabled by default. Use --nocheck-testcases to disable. |
mark-progress | Log line number and elapsed time to <testname>.progress |
Options That Pass On Options
| Option | Description |
|---|---|
--mysqld=ARGS | Specify additional arguments to "mysqld" |
--mysqltest=ARGS | Specify additional arguments to "mariadb-test". Use additional --mysqld-env options to set more than one variable. |
Options to Run Test On Running Server
| Option | Description |
|---|---|
extern option=value | Use an already running server. The option/value pair is what is needed by the mariadb client to connect to the server. Each --extern option can only take one option/value pair as an argument, so you need to repeat --extern for each pair needed. Example: ./mariadb-test-run.pl --extern socket=var/tmp/mysqld.1.sock alias. Note: If a test case has an .opt file that requires the server to be restarted with specific options, the file will not be used. The test case likely will fail as a result. |
Options For Debugging the Product
In mariadb-test-run.pl there is a concept of a "debugger". A "debugger" is a tool that mariadb-test-run.pl will execute instead of mariadbd. This tool will then start mariadbd and can control its execution as it wants. The following "debuggers" are supported:
Every "debugger" from the list above supports the following set of options (replace XXX below with a debugger name)
| Option | Description |
|---|---|
--XXX | Start mariadbd process under a debugger |
--client-XXX | Start mariadb-test process under a debugger |
--boot-XXX | Before running tests mariadb-test-run executes mariadbd to bootstrap, prepare the datadir. This options causes this bootstrapping mariadbd process to be run under a debugger. |
--manual-XXX | Don't start anything, instead print the command that the user needs to run to start mariadbd under a debugger. Then wait. |
Every option from the above accepts an optional argument. It can be used to specify additional command line options to pass to the tool. Or additional commands that the tool will run on startup. Or both. Commands are separated from each other and from options with a semicolon. For example:
./mtr 1st --strace ./mtr 1st --client-rr=--chaos ./mtr 1st --manual-gdb='b mysql_parse;r' ./mtr 1st --boot-gdb='--quiet --tui;b mysql_parse;r'
Misc Debugging Related Options
| Option | Description |
|---|---|
--debug | Dump trace output for all servers and client programs. |
--debug-common | Same as --debug, but sets the 'd' debug flags to "query,info,error,enter,exit" |
--debug-server | Use debug version of server, but without turning on tracing. |
--max-save-core | Limit the number of core files saved (to avoid filling up disks for heavily crashing server). Defaults to 5, set to 0 for no limit. Set its default with MTR_MAX_SAVE_CORE. |
--max-save-datadir | Limit the number of datadir saved (to avoid filling up disks for heavily crashing server). Defaults to 20, set to 0 for no limit. Set its default with MTR_MAX_SAVE_DATDIR. |
--max-test-fail | Limit the number of test failurs before aborting the current test run. Defaults to 10, set to 0 for no limit. Set its default with MTR_MAX_TEST_FAIL. |
Misc Options
| Option | Description |
|---|---|
--user=USER | User for connecting to mysqld (default: root) |
--comment=STR | Write STR to the output within lines filled with #, as a form of banner. |
--timer | Show test case execution time. Use no-timer to disable. |
--verbose | More verbose output(use multiple times for even more) |
--verbose-restart | Write when and why servers are restarted between test cases. |
--start | Only initialize and start the servers, using the startup settings for the first specified test case Example: ./mariadb-test-run.pl --start alias & start-dirty Only start the servers (without initialization) for the first specified test case |
--start-and-exit | Same as --start, but mariadb-test-run terminates and leaves just the server running. |
--start-dirty | This is similar to --start, but will skip the database initialization phase and assume that database files are already available. Usually this means you must have run another test first. |
--user-args | In combination with start* and no test name, drops arguments to mysqld except those specified with --mysqld (if any). |
--wait-all | If --start or --start-dirty option is used, wait for all servers to exit before finishing the process. Otherwise, it will terminate if one (of several) servers is restarted. |
--fast | Do not perform controlled shutdown when servers need to be restarted or at the end of the test run. This is equivalent to using --shutdown-timeout=0. |
--force-restart | Always restart servers between tests. |
--parallel=N | Run tests in N parallel threads (default 1) Use parallel=auto for auto-setting of N. |
--repeat=N | Run each test N number of times. |
--retry=N | If a test fails, it is retried up to a maximum of N runs (default 1). Retries are also limited by the maximum number of failures before stopping, set with the --retry-failure option. This option has no effect unless --force is also used; without it, test execution will terminate after the first failure. The --retry and --retry-failure options do not affect how many times a test repeated with --repeat may fail in total, as each repetition is considered a new test case, which may in turn be retried if it fails. |
--retry-failure=N | When using the --retry option to retry failed tests, stop when N failures have occured (default 2). Setting it to 0 or 1 effectively turns off retries. |
--reorder | Reorder tests to get fewer server restarts. This is the default behavior. There is no guarantee that a particular set of tests will always end up in the same order. Use -no-reorder to disable. |
--help | Display help text. |
--testcase-timeout=MINUTES | Max test case run time in minutes (default 15). |
--suite-timeout=MINUTES | Max test suite run time in minutes (default 360). |
--shutdown-timeout=SECONDS | Max number of seconds to wait for server shutdown before killing servers (default 10). |
--warnings | Scan the log files for warnings and report any suspicious ones; if any are found, the test will be marked as failed. Use --nowarnings to turn off. |
--stop-file=file | If this file is detected, mariadb-test will not start new tests until the file is removed (also MTR_STOP_FILE environment variable). |
--stop-keep-alive=sec | Works with --stop-file, print messages every sec seconds when mariadb-test is waiting to remove the file (for buildbot) (also MTR_STOP_KEEP_ALIVE environment variable). |
--sleep=SECONDS | Passed to mariadb-test; will be used as fixed sleep time. |
--debug-sync-timeout=NUM | Set default timeout for WAIT_FOR debug sync actions. Disable facility with NUM=0. |
--gcov | Collect coverage information after the test. The result is a dgcov file per source and header file and a last_changes.dgcov file in the vardir with the coverage for the uncommitted changes if any (or the last commit). |
--gprof | Collect profiling information using the gprof profiling tool. |
--experimental=<file> | Specify a file that contains a list of test cases that should be displayed with the [ exp-fail ] code rather than [ fail ] if they fail. For an example of a file that might be specified via this option, see mariadb-test/collections/default.experimental. |
--report-features | First run a "test" that reports MariaDB features, displaying the output of SHOW ENGINES and SHOW VARIABLES. This can be used to verify that binaries are built with all required features. |
--timestamp | Print timestamp before each test report line, showing when the test ended. |
--timediff | Used with --timestamp, also print time passed since the previous test started. |
--max-connections=N | Maximum number of simultaneous server connections that may be used per test. Default is 128. Minimum is 8, maximum is 5120. Corresponds to the same option for mariadb-test. |
--default-myisam | Set default storage engine to MyISAM for non-innodb tests. This is needed after switching default storage engine to InnoDB. |
--report-times | Report how much time has been spent on different phases of test execution. |
--stress=ARGS | Run stress test, providing options to mysql-stress-test.pl. Options are separated by comma. |
xml-report=<file> | Output jUnit xml file of the results. From MariaDB 10.1.45, MariaDB 10.2.32, MariaDB 10.3.23, MariaDB 10.4.13, MariaDB 10.5.3 |
tail-lines=N | Number of lines of the result to include in a failure report. From MariaDB 10.3.4. |
Comments
Comments loading...
Content reproduced on this site is the property of its respective owners,
and this content is not reviewed in advance by MariaDB. The views, information and opinions
expressed by this content do not necessarily represent those of MariaDB or any other party.
