The Ultimate Guide to High Availability with MariaDB

MariaDB 10.4 / 10.5 crashing and restarting intermittently - status=11/SEGV

Hello,

I am installing a new server with MariaDB 10.4 or 10.5, but I am facing on ubuntu syslog the "systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV" being reported a lot of times. Then applications loss the DB connections and some transactions are broken. I tried to install a new server, MariaDB, and restore a existing dump, however the same error happened.

My system: - AWS EC2 running ubuntu 18.04 instances; - MariaDB 10.4.14 / 10.5 (error on both versions); - Client applications (nodejs using mysql or mariadb dependency);

Any comments are welcome.

Best regards,

This is a sample of my syslog:

Aug 28 22:58:25 ip-172--- systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV Aug 28 22:58:25 ip-172--- systemd[1]: mariadb.service: Failed with result 'signal'. Aug 28 22:58:30 ip-172--- systemd[1]: mariadb.service: Service hold-off time over, scheduling restart. Aug 28 22:58:30 ip-172--- systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 2. Aug 28 22:58:30 ip-172--- systemd[1]: Stopped MariaDB 10.4.14 database server. Aug 28 22:58:30 ip-172--- systemd[1]: Starting MariaDB 10.4.14 database server... Aug 28 22:58:30 ip-172--- mysqld[17019]: 2020-08-28 22:58:30 0 [Note] /usr/sbin/mysqld (mysqld 10.4.14-MariaDB-1:10.4.14+mariabionic-log) starting as process 17019 ... Aug 28 22:58:30 ip-172--- mysqld[17019]: 2020-08-28 22:58:30 0 [Note] InnoDB: Using Linux native AIO Aug 28 22:58:30 ip-172--- mysqld[17019]: 2020-08-28 22:58:30 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins Aug 28 22:58:30 ip-172--- mysqld[17019]: 2020-08-28 22:58:30 0 [Note] InnoDB: Uses event mutexes Aug 28 22:58:30 ip-172--- mysqld[17019]: 2020-08-28 22:58:30 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 Aug 28 22:58:30 ip-172--- mysqld[17019]: 2020-08-28 22:58:30 0 [Note] InnoDB: Number of pools: 1 Aug 28 22:58:30 ip-172--- mysqld[17019]: 2020-08-28 22:58:30 0 [Note] InnoDB: Using SSE2 crc32 instructions Aug 28 22:58:30 ip-172--- mysqld[17019]: 2020-08-28 22:58:30 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts) Aug 28 22:58:30 ip-172--- mysqld[17019]: 2020-08-28 22:58:30 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M Aug 28 22:58:30 ip-172--- mysqld[17019]: 2020-08-28 22:58:30 0 [Note] InnoDB: Completed initialization of buffer pool Aug 28 22:58:30 ip-172--- mysqld[17019]: 2020-08-28 22:58:30 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). Aug 28 22:58:30 ip-172--- mysqld[17019]: 2020-08-28 22:58:30 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1136196476 Aug 28 22:58:30 ip-172--- mysqld[17019]: 2020-08-28 22:58:30 0 [Note] InnoDB: Starting final batch to recover 2 pages from redo log. Aug 28 22:58:31 ip-172--- mysqld[17019]: 2020-08-28 22:58:31 0 [Note] InnoDB: Last binlog file '/var/log/mysql/mariadb-bin.000011', position 71237387 Aug 28 22:58:31 ip-172--- mysqld[17019]: 2020-08-28 22:58:31 0 [Note] InnoDB: 128 out of 128 rollback segments are active. Aug 28 22:58:31 ip-172--- mysqld[17019]: 2020-08-28 22:58:31 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" Aug 28 22:58:31 ip-172--- mysqld[17019]: 2020-08-28 22:58:31 0 [Note] InnoDB: Creating shared tablespace for temporary tables Aug 28 22:58:31 ip-172--- mysqld[17019]: 2020-08-28 22:58:31 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... Aug 28 22:58:31 ip-172--- mysqld[17019]: 2020-08-28 22:58:31 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. Aug 28 22:58:31 ip-172--- mysqld[17019]: 2020-08-28 22:58:31 0 [Note] InnoDB: Waiting for purge to start Aug 28 22:58:31 ip-172--- mysqld[17019]: 2020-08-28 22:58:31 0 [Note] InnoDB: 10.4.14 started; log sequence number 1136197921; transaction id 4009 Aug 28 22:58:31 ip-172--- mysqld[17019]: 2020-08-28 22:58:31 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool Aug 28 22:58:31 ip-172--- mysqld[17019]: 2020-08-28 22:58:31 0 [Note] Plugin 'FEEDBACK' is disabled. Aug 28 22:58:31 ip-172--- mysqld[17019]: 2020-08-28 22:58:31 0 [Note] Recovering after a crash using /var/log/mysql/mariadb-bin Aug 28 22:58:31 ip-172--- mysqld[17019]: 2020-08-28 22:58:31 0 [Note] Starting crash recovery... Aug 28 22:58:31 ip-172--- mysqld[17019]: 2020-08-28 22:58:31 0 [Note] Crash recovery finished. Aug 28 22:58:31 ip-172--- mysqld[17019]: 2020-08-28 22:58:31 0 [Note] Server socket created on IP: '::'. Aug 28 22:58:31 ip-172--- mysqld[17019]: 2020-08-28 22:58:31 0 [Note] Reading of all Master_info entries succeeded Aug 28 22:58:31 ip-172--- mysqld[17019]: 2020-08-28 22:58:31 0 [Note] Added new Master_info '' to hash table Aug 28 22:58:31 ip-172--- mysqld[17019]: 2020-08-28 22:58:31 0 [Note] /usr/sbin/mysqld: ready for connections. Aug 28 22:58:31 ip-172--- mysqld[17019]: Version: '10.4.14-MariaDB-1:10.4.14+mariabionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution Aug 28 22:58:31 ip-172--- systemd[1]: Started MariaDB 10.4.14 database server. Aug 28 22:58:31 ip-172--- /etc/mysql/debian-start[17062]: Upgrading MySQL tables if necessary. Aug 28 22:58:31 ip-172--- /etc/mysql/debian-start[17065]: Looking for 'mysql' as: /usr/bin/mysql Aug 28 22:58:31 ip-172--- /etc/mysql/debian-start[17065]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck Aug 28 22:58:31 ip-172--- /etc/mysql/debian-start[17065]: This installation of MariaDB is already upgraded to 10.4.14-MariaDB, use --force if you still need to run mysql_upgrade Aug 28 22:58:31 ip-172--- /etc/mysql/debian-start[17073]: Checking for insecure root accounts. Aug 28 22:58:31 ip-172--- /etc/mysql/debian-start[17077]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables Aug 28 22:58:32 ip-172--- mysqld[17019]: 2020-08-28 22:58:32 32 [Warning] Host name 'bd6446e1.virtua.com.br' could not be resolved: Name or service not known Aug 28 22:58:33 ip-172--- mysqld[17019]: 2020-08-28 22:58:33 0 [Note] InnoDB: Buffer pool(s) load completed at 200828 22:58:33

Answer Answered by Daniel Black in this comment.

Hi,

I'm sorry you have found a bug.

Can you please report this on https://jira.mariadb.org/?

In your output I can only see the crash recovery, not the case of the crash itself. The following information will assist in making progress in resolving your difficulties:

Can you install the debug info packages with:

  1. add-apt-repository 'deb [arch=amd64,arm64,ppc64el] https://mirrors.dotsrc.org/mariadb/repo/10.4/ubuntu bionic main/debug'
  2. apt-get update
  3. apt-get install -y mariadb-server-10.4-dbgsym

And then follow the following to get a stack trace out of your coredump.

https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/#analyzing-a-core-file-with-gdb-on-linux

Comments

Comments loading...
Content reproduced on this site is the property of its respective owners, and this content is not reviewed in advance by MariaDB. The views, information and opinions expressed by this content do not necessarily represent those of MariaDB or any other party.