What to Do if MariaDB Doesn't Start

You are viewing an old version of this article. View the current version here.

There could be many reasons that MariaDB fails to start. This page will help troubleshoot some of the more common reasons and provide solutions.

If you have tried everything here, and still need help, you can ask for help on IRC or on the forums - see Where to find other MariaDB users and developers - or ask a question at the Starting and Stopping MariaDB page.

The Error Log and the Data Directory

The reason for the failure will almost certainly be written in the Error Log and, if you are starting MariaDB manually, to the console. By default, the error log is named host-name.err and is written to the data directory.

Common Locations:

  • in the journal (if systemd) - journalctl - run journalctl -u mariadb.service
  • /var/log/
  • /var/log/mysql
  • C:\Program Files\MariaDB x.y\data (x.y refers to the version number)
  • C:\Program Files (x86)\MariaDB x.y\data (32bit version on 64bit Windows)

It's also possible that the error log has been explicitly written to another location, either by changing the datadir system variable, or setting with log-error=filename - see my.cnf below.

In most cases you should be able to find out the place of the error file by doing:

mysqld --help --verbose | grep 'log-error' | tail -1
mysqld --help --verbose | grep 'datadir' | tail -1

my.cnf

The other main file to consider is the my.cnf file. It contains configuration settings, such as the location of the data directory mentioned above. If you're unsure where the my.cnf file is, see configuring MariaDB with my.cnf.

You can see what is in the configuration file for the MariaDB server (mysqld) by executing:

mysqld --print-defaults

or

my_print_defaults mysqld server mysqld-10.0 mariadb mariadb-10.0 client-server

Can't Open Privilege Tables

If you get the following error:

System error 1067 has occurred.
Fatal error: Can't open privilege tables: Table 'mysql.host' doesn't exist

critical files, such as the privilege tables, are either missing or are in the wrong location. By default, MariaDB will create the privilege tables in the correct location, but the above error is quite common after an upgrade, where the configuration files point to a different location. See configuring MariaDB with my.cnf to find the my.cnf (or my.ini) configuration file/s.

Make sure that the basedir and datadir variables are correctly set.

An Incorrect my.cnf Setting

When you upgrade to a new version of MariaDB, in most cases the old my.cnf configuration will work just fine. However, it's possible that one of the variables set there has been removed in the new version, or one of the values is no longer valid. Remember that the options are case sensitive. Examine the specifics of the error, and then remove the invalid setting from the my.cnf file, or change it to the new name, or a valid value. There should be an error like this:

140514 12:19:37 [ERROR] /usr/local/mysql/bin/mysqld: unknown variable 'option=value'

XtraDB/InnoDB

InnoDB is probably the MariaDB component that most frequently causes a crash. In the error log, lines containing InnoDB messages generally start with "InnoDB:".

Cannot Allocate Memory for the Buffer Pool

In a typical installation on a dedicated server, at least 70% of your memory should be assigned to InnoDB buffer pool; sometimes it can even reach 85%. But be very careful: don't assign to the buffer pool more memory than it can allocate. If it cannot allocate memory, InnoDB will use the disk's swap area, which is very bad for performance. If swapping is disabled or the swap area is not big enough, InnoDB will crash. In this case, MariaDB will probably try to restart several times, and each time it will log a message like this:

140124 17:29:01 InnoDB: Fatal error: cannot allocate memory for the buffer pool

In that case, you will need to add more memory to your server/VM or decrease the value of the innodb_buffer_pool_size variables.

Remember that the buffer pool will slightly exceed that limit. Also, remember that MariaDB also needs allocate memory for other storage engines and several per-connection buffers. The operating system also needs memory.

Table Corruption

By default, InnoDB deliberately crashes the server when it detects table corruption. The reason for this behavior is preventing corruption propagation. However, in some situations, server availability is more important than data integrity. For this reason, we can avoid these crashes by changing the value of innodb_corrupt_table_action to 'warn'.

If InnoDB crashes the server after detecting data corruption, it writes a detailed message in the error log. The first lines are similar to the following:

InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 7.
InnoDB: You may have to recover from a backup.

Generally, it is still possible to recover most of the corrupted data. To do so, restart the server in InnoDB recovery mode and try to extract the data that you want to backup. You can save them in a CSV file or in a non-InnoDB table. Then, restart the server in normal mode and restore the data.

MyISAM

Most tables in the mysql database are MyISAM tables. These tables are necessary for MariaDB to properly work, or even start.

A MariaDB crash could cause system tables corruption. With the default settings, MariaDB will simply not start if the system tables are corrupted. With myisam_recover_options, we can force MyISAM to repair damaged tables.

systemd

If you have your datadir under the /home directory, the default systemd file has ProtectHome=true . This prevents access, showing Permission Denied on this directory in the error log.

To work around this without moving your data files, create /etc/systemd/system/mariadb.service.d/homedir.conf containing the following:

[Service]
ProtectHome=false

and then run systemctl daemon-reload.

ProtectSystem=full is also set and places restrictions on writing to other locations. Overwriting this with ProtectSystem=off in the same way as above will restore access.

See systemd documentation for further information on systemd configuration.

SELinux

SELinux is a Mandatory Access Control system. It is enabled by default on some Linux systems, especially on RedHat-derivatives. SELinux prevents programs from accessing files, directories or ports unless it is configured to access those resources.

SELinux could prevent MariaDB from starting with a non-default port, or reading or writing some files (datadir, log files, configuration files...) placed in a non-default path.

To verify if SELinux is running:

getenforce

To temporarily put mysqld_t into permissive mode:

semanage permissive -a mysqld_t

If that helped, it means that current SELinux policy is the culprit. You need to adjust the SELinux policy or labels for MariaDB in one or more of the ways below.

Changing policy and path labels requires the policycoreutils-python package to install semanage and audit2allow tools.

SELinux and MariaDB Path Changes

If you have a mysql error log that looks like:

130321 11:50:51 mysqld_safe Starting mysqld daemon with databases from /datadir
...
2013-03-21 11:50:52 2119 [Warning] Can't create test file /datadir/
2013-03-21 11:50:52 2119 [Warning] Can't create test file /datadir/
...
2013-03-21 11:50:52 2119 [ERROR] /usr/sbin/mysqld: Can't create/write to file 
    '/datadir/boxy.pid' (Errcode: 13 - Permission denied)
2013-03-21 11:50:52 2119 [ERROR] Can't start server: can't create PID file: 
    Permission denied
130321 11:50:52 mysqld_safe mysqld from pid file /datadir/boxy.pid ended

First be sure that the /datadir is owned by the mysql user with read write permissions for that user.

If /var/log/audit/audit.log looks something like:

type=AVC msg=audit(1363866652.030:24): avc:  denied  { write } for  pid=2119 
    comm="mysqld" name="datadir" dev=dm-0 ino=394 
    scontext=unconfined_u:system_r:mysqld_t:s0 
    tcontext=unconfined_u:object_r:default_t:s0 tclass=dir

then you need to adjust your file system labelling so that SELinux knows this is MariaDB's datadir. Check with:

ls -ldaZ /datadir
drwxr-xr-x. root root unconfined_u:object_r:var_t:s0
semanage fcontext -a -t mysqld_db_t "/datadir(/.*)?"
restorecon -Rv /datadir

If you use logrotate, logrotate needs to write to the directory with its logs. Without the right SELinux permissions you might find that logrotate doesn't have the permissions to rotate the log file. Log files need the label mysqld_log_t. It is ideal to configure MariaDB to use a non-datadir for this as two directories can't share the same label. The filesystem label can be changed to mysqld_log_t using:

semanage fcontext -a -t mysqld_log_t "/var/log/mysql(/.*)?"
restorecon -Rv /var/log/mysql

Configuration file permissions use the label mysqld_etc_t so changing from the default will require those to be relabeled the same way.

If your directory is something else, like changing the MariaDB tmpdir to /run/shm (has serious implications if using large temp tables, this isn't a default endorsement), you can't easily change the filesystem label without affecting other programs that use /run/shm. Using the procedure below for other SELinux changes, you should end up with a module like:

MySQL_tmpfs.te:

module MySQL_tmpfs 1.0;

require {
type tmpfs_t;
type mysqld_t;
class dir { write search read remove_name open getattr add_name };
class file { write getattr read lock create unlink open };
}

allow mysqld_t tmpfs_t:dir { write search read remove_name open getattr add_name };

allow mysqld_t tmpfs_t:file { write getattr read lock create unlink open }

To convert this into a loaded module perform the following:

checkmodule -M -m MySQL_tmpfs.te -o MySQL_tmpfs.mod
semodule_package -m MySQL_tmpfs.mod -o MySQL_tmpfs.pp 
semodule -i MySQL_tmpfs.pp

SELinux and MariaDB On a Different Port

TCP and UDP ports are enabled for permission to bind too. If you are using a different port, or some Galera ports, configure SELinux to be able to use those ports:

semanage port -a -t mysqld_port_t -p tcp 3307 

Other SELinux Changes

It could also be the case that a plugin or configuration usages has required a SElinux permission not thought of by the SElinux module developers.

Start by temporarily putting mysqld_t into permissive mode:

semanage permissive -a mysqld_t

Here we record what MariaDB is doing and to create a policy to allow that (but nothing else).

  • Switch SELinux to permissive mode and remove dontaudits from the policy:
semodule -DB
semanage permissive -a mysqld_t
  • start MariaDB
  • use the generated audit log to create a policy:
grep mysqld /var/log/audit/audit.log | audit2allow -M mariadb_local
semodule -i mariadb_local.pp
  • Remove the permissive mode for mysqld_t and restore dontaudits:
setmodule -B
semanage permissive -d mysqld_t

The same procedure can be used if MariaDB starts but SELinux prevents it from functioning correctly. For example, SELinux may prevent PAM plugin from authenticating users. The solution is the same — enable auditing, switch to permissive, do, whatever SELinux didn't allow you to, create a policy from the audit log.

When you discover any needed SELinux permissions, please report the needed permissions to your operating system bug tracking so all users can benefit from your work (e.g. Red Hat Bugzilla https://bugzilla.redhat.com/).

Comments

Comments loading...
Content reproduced on this site is the property of its respective owners, and this content is not reviewed in advance by MariaDB. The views, information and opinions expressed by this content do not necessarily represent those of MariaDB or any other party.