mariadb server mysteriously seems to be crashing

hello - the last couple of days my server seems to have mysteriously stopped. simply doing:

systemctl stop mariadb.service ; systemctl start mariadb.service ;

seems to temporarily fix it. i am tempted to have something like this run in a cron job every hour just to be safe.

Server version: 10.2.27-MariaDB MariaDB Server

# systemctl  status  mariadb.service  ;  
● mariadb.service - MariaDB 10.2.27 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─migrated-from-my.cnf-settings.conf
   Active: failed (Result: timeout) since Sun 2019-10-06 15:28:21 EDT; 2h 27min ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 20375 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 20343 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=killed, signal=KILL)
  Process: 14181 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemct                                              l set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=killed, signal=TERM)
  Process: 14154 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
 Main PID: 20343 (code=killed, signal=KILL)
   CGroup: /system.slice/mariadb.service
 

EDIT:

here is what i am seeing now:

# systemctl  status  -l  mariadb.service  ;   ## restart mariadb
● mariadb.service - MariaDB 10.2.27 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─migrated-from-my.cnf-settings.conf
   Active: activating (start) since Sun 2019-10-06 18:31:14 EDT; 43s ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 10240 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 13399 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
  Process: 13320 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
 Main PID: 13455 (mysqld)
   CGroup: /system.slice/mariadb.service
           └─13455 /usr/sbin/mysqld

Oct 06 18:31:37 edwardsmark.info mysqld[13455]: 2019-10-06 18:31:32 140379062454464 [Note] InnoDB: Using Linux native AIO
Oct 06 18:31:37 edwardsmark.info mysqld[13455]: 2019-10-06 18:31:32 140379062454464 [Note] InnoDB: Number of pools: 1
Oct 06 18:31:37 edwardsmark.info mysqld[13455]: 2019-10-06 18:31:32 140379062454464 [Note] InnoDB: Using SSE2 crc32 instructions
Oct 06 18:31:37 edwardsmark.info mysqld[13455]: 2019-10-06 18:31:32 140379062454464 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
Oct 06 18:31:37 edwardsmark.info mysqld[13455]: 2019-10-06 18:31:32 140379062454464 [Note] InnoDB: Completed initialization of buffer pool
Oct 06 18:31:37 edwardsmark.info mysqld[13455]: 2019-10-06 18:31:32 140378200946432 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
Oct 06 18:31:37 edwardsmark.info mysqld[13455]: 2019-10-06 18:31:34 140379062454464 [Note] InnoDB: Highest supported file format is Barracuda.
Oct 06 18:31:37 edwardsmark.info mysqld[13455]: 2019-10-06 18:31:35 140379062454464 [Note] InnoDB: Starting crash recovery from checkpoint LSN=38062514322
Oct 06 18:31:13 edwardsmark.info systemd[1]: Stopped MariaDB 10.2.27 database server.
Oct 06 18:31:14 edwardsmark.info systemd[1]: Starting MariaDB 10.2.27 database server...

AND ANOTHER CRASH!

# systemctl  status  -l  mariadb.service  ;  
● mariadb.service - MariaDB 10.2.27 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─migrated-from-my.cnf-settings.conf
   Active: failed (Result: timeout) since Sun 2019-10-06 18:38:58 EDT; 6min ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 10240 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 14236 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 14026 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
  Process: 13898 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
 Main PID: 14236 (code=exited, status=0/SUCCESS)
   Status: "MariaDB server is down"

Oct 06 18:38:53 edwardsmark.info mysqld[14236]: 2019-10-06 18:38:53 139696513103616 [Note] InnoDB: FTS optimize thread exiting.
Oct 06 18:38:54 edwardsmark.info mysqld[14236]: 191006 18:38:54 server_audit: STOPPED
Oct 06 18:38:56 edwardsmark.info mysqld[14236]: 2019-10-06 18:38:56 139696652347136 [Note] InnoDB: Starting shutdown...
Oct 06 18:38:56 edwardsmark.info mysqld[14236]: 2019-10-06 18:38:56 139696403367680 [Note] InnoDB: Buffer pool(s) load completed at 191006 18:38:56
Oct 06 18:38:56 edwardsmark.info mysqld[14236]: 2019-10-06 18:38:56 139696403367680 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
Oct 06 18:38:56 edwardsmark.info mysqld[14236]: 2019-10-06 18:38:56 139696403367680 [Note] InnoDB: Buffer pool(s) dump completed at 191006 18:38:56
Oct 06 18:38:58 edwardsmark.info mysqld[14236]: 2019-10-06 18:38:57 139696652347136 [Note] InnoDB: Shutdown completed; log sequence number 38062514359
Oct 06 18:38:58 edwardsmark.info systemd[1]: Failed to start MariaDB 10.2.27 database server.
Oct 06 18:38:58 edwardsmark.info systemd[1]: Unit mariadb.service entered failed state.
Oct 06 18:38:58 edwardsmark.info systemd[1]: mariadb.service failed.

Comments

Comments loading...
Content reproduced on this site is the property of its respective owners, and this content is not reviewed in advance by MariaDB. The views, information and opinions expressed by this content do not necessarily represent those of MariaDB or any other party.