The Ultimate Guide to High Availability with MariaDB

MariaDB troubles, only running after reboot, times out when trying restart, not stable when running.

For a baseline, I have a recently upgraded Ubuntu Server 18.04 LTS from Ubuntu 16.04 LTS. I like to manage it with Virtualmin software. On it, I operate a couple projects, like a Moodle school and a Nextcloud server. I generally like to live in MySQL databases but have been interested in MariaDB for a long time. I got nervous about MySQL when the evil empire of Oracle bought Sun.

At least two new projects I want to run on my server urge the use of MariaDB; PeerTube, and either Frendica or Diaspora. I have a few Drupal sites to develop too.

A few days ago I tried to make the switch from MySQL 5.7 to MariaDB 10. I did it long ago for some reason but had to do a server rebuild recently and didn't try to make MariaDB work before restoring all the stuff I had on it. I remembered the first time I did it on Ubuntu 16.04, I lost all the databases but I didn't have anything valuable at that time, so I was well prepared this time with a dump to restore.

I tried to install at first with a Bourne shell command of sudo apt-get install mariadb-server mariadb-client, but that threw an error message so I looked at the versions Ubuntu 18.04 was supporting and ran, sudo apt-get install mariadb-server10.1 mariadb-client10.1. That seemed to work and did disconnect the data directory cause it was "binary", but MariaDB didn't actually work at that point. I noticed that my Virtualmin System had both mariadb-server and mariadb-client listed in their software package installer so I thought that they would blow over the non-working one. After a reboot, I installed them and still had no working MariaDB.

Since I'm at a #FTW point, I decided to use the Virtualmin System to find MariaDB and MySQL and uninstall with "Purge configuration files" and "Removed unused dependencies as well" flags on. After clearing all those programs, I rebooted and installed only mariadb-server using the Virtualmin System software package installer (it uses APT) and that seemed to go very well. It put on the client, core's and common programs automatically. It was running after a reboot. I was able to restore the previously existing databases after some struggles with utf8mb4 vs. utf8.

I tested two of the sites I run which use MySQL and neither worked so I rebooted again, and later I realized I forgot to put the users back in. I saw MariaDB wasn't running, and nothing I tried to start/restart MariaDB seemed to work. My Virtualmin web interface for the MySQL server just said:

MySQL is not running on your system - database list could not be retrieved.

After clicking the Virtualmin button to restart MariaDB, the full MySQL error message was: 

DBI connect failed : Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

I rebooted and MariaDB was running again. I went to set up the missing users and when I clicked the Virtualmin thing to add a new user, MariaDB stopped running again. Here's the error message after failing to add a user:

Failed to save user : DBI connect failed : Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

Attempting to add a user didn't crash things. On subsequent attempts, I was able to work swiftly and enter the user to get Nextcloud to work fully again. Also, if I do nothing after a reboot the MariaDB server will stop running. This never happened in 16.04 LTS.

I'm not sure if this helps, but It's about a 1-minute window after every reboot where MariaDB is running before it stops running. I don't know much about sockets, but could something be killing the socket for MariaDB within a few minutes of rebooting?

I don't know if this helps, but dmesg | tail -30 produces:

[   28.212731] audit: type=1400 audit(1548120125.668:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-mounting" pid=1034 comm="apparmor_parser"
[   28.212733] audit: type=1400 audit(1548120125.668:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-nesting" pid=1034 comm="apparmor_parser"
[   28.222187] audit: type=1400 audit(1548120125.676:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/man" pid=1038 comm="apparmor_parser"
[   28.222191] audit: type=1400 audit(1548120125.676:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_filter" pid=1038 comm="apparmor_parser"
[   28.222194] audit: type=1400 audit(1548120125.676:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_groff" pid=1038 comm="apparmor_parser"
[   28.239010] audit: type=1400 audit(1548120125.692:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine" pid=1039 comm="apparmor_parser"
[   38.444003] new mount options do not match the existing superblock, will be ignored
[   43.999087] ip6_tables: (C) 2000-2006 Netfilter Core Team
[   44.100012] Ebtables v2.0 registered
[   44.409808] bnx2 0000:02:00.0 enp2s0f0: using MSIX
[   44.410119] IPv6: ADDRCONF(NETDEV_UP): enp2s0f0: link is not ready
[   44.600648] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[   45.280596] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[   45.493559] Netfilter messages via NETLINK v0.30.
[   45.501174] ip_set: protocol 6
[   48.228661] bnx2 0000:02:00.0 enp2s0f0: NIC Copper Link is Up, 1000 Mbps full duplex
[   48.228679] , receive & transmit flow control ON
[   48.228774] IPv6: ADDRCONF(NETDEV_CHANGE): enp2s0f0: link becomes ready
[   64.210462] kauditd_printk_skb: 9 callbacks suppressed
[   64.210463] audit: type=1400 audit(1548120161.662:21): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=2527 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=113 ouid=0
[  144.364055] audit: type=1400 audit(1548120241.595:22): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=2527 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=113 ouid=0
[  144.465883] audit: type=1400 audit(1548120241.699:23): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=2527 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=113 ouid=0
[  144.566363] audit: type=1400 audit(1548120241.799:24): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=2527 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=113 ouid=0
[  144.666722] audit: type=1400 audit(1548120241.899:25): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=2527 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=113 ouid=0
[  144.767069] audit: type=1400 audit(1548120241.999:26): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=2527 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=113 ouid=0
[  144.867432] audit: type=1400 audit(1548120242.099:27): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=2527 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=113 ouid=0
[  144.967801] audit: type=1400 audit(1548120242.199:28): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=2527 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=113 ouid=0
[  145.068178] audit: type=1400 audit(1548120242.299:29): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=2527 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=113 ouid=0
[  145.168519] audit: type=1400 audit(1548120242.399:30): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=2527 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=113 ouid=0
[  145.268887] audit: type=1400 audit(1548120242.499:31): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=2527 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=113 ouid=0

Answer Answered by Faustin Lammler in this comment.

Hi Michael,
Thank you for your question and your help in finding a solution to this.

It seems that as you said in a previous comment, the apparmor profile coming from MySQL is still loaded and this prevents MariaDB to start after upgrade. For the record, we do not provide an Apparmor profile (it is commented).

Here is an explication why:
https://salsa.debian.org/mariadb-team/mariadb-10.1/blob/stretch/debian/apparmor-profile

A first solution is to disable only the problematic Apparmor profile.

$ sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
$ sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld

Verify:

$ sudo aa-status

Finalize mariadb-server upgrade:

$ sudo dpkg --configure mariadb-server

Another solution (provided by Michael) could be to disable momentarily Apparmor during the upgrade:

$ sudo systemctl stop apparmor.service
$ sudo update-rc.d -f apparmor remove

$ sudo apt-get remove --purge mysql-server mysql-client mysql-common
$ sudo apt-get autoremove
$ sudo apt-get autoclean
$ sudo apt-get install mariadb-server

$ sudo systemctl start apparmor.service
$ sudo update-rc.d apparmor defaults

Faustin

Comments

Comments loading...
Content reproduced on this site is the property of its respective owners, and this content is not reviewed in advance by MariaDB. The views, information and opinions expressed by this content do not necessarily represent those of MariaDB or any other party.