Mariadb crashed during deleting a database - Source

Revision: 141134
User: Rich Ayoade
Date: 2024-06-25 13:51

Hello everyone,

I need your help to deal with this problem on a production server.

I was trying to delete a seemingly corrupted crm database when mariadb crashed.
Since then mariadb.service has been unavailable.

.....server details......
openSUSE Leap 15.1
mariadb 10.2.31-lp151.2.12.1 x86_64 from opensuse
php 7.2.5
php7-MySQL Mysql client for php (mysql 5.0.2)

..some commands executed...........
#systemctl stop mariadb.service  (No error thrown probably because mariadb wasn't running anyway)
.......
#systemctl start mariadb.service
Job for mariadb.service failed because a fatal signal was delivered to the control process.
.......
#systemctl status  mariadb.service
● mariadb.service - MySQL server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disa>
   Active: failed (Result: protocol) since Sun 2024-06-23 16:35:52 CEST; 19min ago
  Process: 2895 ExecStart=/usr/lib/mysql/mysql-systemd-helper start (code=killed, signal>
  Process: 2888 ExecStartPre=/usr/lib/mysql/mysql-systemd-helper upgrade (code=exited, s>
  Process: 2882 ExecStartPre=/usr/lib/mysql/mysql-systemd-helper install (code=exited, s>
 Main PID: 2895 (code=killed, signal=SEGV)
   Status: "Starting final batch to recover 87 pages from redo log"

Jun 23 16:35:46 smooths systemd[1]: Starting MySQL server...
Jun 23 16:35:47 smooths mysql-systemd-helper[2895]: 2024-06-23 16:35:47 14071001006>
Jun 23 16:35:52 smooths systemd[1]: mariadb.service: Main process exited, code=kill>
Jun 23 16:35:52 smooths systemd[1]: Failed to start MySQL server.
Jun 23 16:35:52 smooths systemd[1]: mariadb.service: Unit entered failed state.
Jun 23 16:35:52 smooths systemd[1]: mariadb.service: Failed with result 'protocol'.

[4]+  Stopped                 systemctl status mariadb.service

..........
#journalctl -xe
-- Unit UNIT has finished shutting down.
Jun 24 10:20:02 smooths systemd[13641]: Reached target Shutdown.
-- Subject: Unit UNIT has finished start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit UNIT has finished starting up.
--
-- The start-up result is done.
Jun 24 10:20:02 smooths systemd[13641]: Starting Exit the Session...
-- Subject: Unit UNIT has begun start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit UNIT has begun starting up.
Jun 24 10:20:02 smooths systemd[13641]: Received SIGRTMIN+24 from PID 13648 (kill).
Jun 24 10:20:02 smooths systemd[13642]: pam_unix(systemd-user:session): session clo>
Jun 24 10:20:02 smooths systemd[1]: Stopped User Manager for UID 72.
-- Subject: Unit user@72.service has finished shutting down
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit user@72.service has finished shutting down.
Jun 24 10:20:02 smooths systemd[1]: Removed slice User Slice of UID 72.
-- Subject: Unit user-72.slice has finished shutting down
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit user-72.slice has finished shutting down.

[6]+  Stopped                 journalctl -xe
............

#ps auxw |grep mysql
root     14238  0.0  0.0   8692   920 pts/0    S+   11:11   0:00 grep --color=auto mysql

#mysql -u root -p
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/run/mysql/mysql.sock' (111)

.....last bits in mysql.log...(I could post more from the log if required)........

2024-06-24 10:19:37 140452141463296 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
2024-06-24 10:19:37 140453265478208 [Note] Plugin 'FEEDBACK' is disabled.
2024-06-24 10:19:37 140453265478208 [Note] Recovering after a crash using tc.log
2024-06-24 10:19:37 140453265478208 [Note] Starting crash recovery...
2024-06-24 10:19:37 140453265478208 [Note] Crash recovery finished.
2024-06-24 10:19:37 140452141463296 [ERROR] InnoDB: Page [page id: space=86, page number=102] log sequence number 75258578838 is in the future! Current system log sequence number 75111066143.
2024-06-24 10:19:37 140452141463296 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
2024-06-24 10:19:37 140452141463296 [ERROR] InnoDB: Page [page id: space=86, page number=103] log sequence number 75769951403 is in the future! Current system log sequence number 75111066143.
2024-06-24 10:19:37 140452141463296 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
2024-06-24 10:19:37 140452141463296 [ERROR] InnoDB: Page [page id: space=160, page number=3] log sequence number 75133061380 is in the future! Current system log sequence number 75111066143.
2024-06-24 10:19:37 140452141463296 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
/lib64/libc.so.6(abort+0x151)[0x7fbdd0279b01]
/usr/sbin/mysqld(+0x44616a)[0x55c5418b916a]
/usr/sbin/mysqld(+0x8fd0b8)[0x55c541d700b8]
/usr/sbin/mysqld(+0x8fd85a)[0x55c541d7085a]
/usr/sbin/mysqld(+0x8fe1bb)[0x55c541d711bb]
/usr/sbin/mysqld(+0x8e1693)[0x55c541d54693]
/lib64/libpthread.so.0(+0x84f9)[0x7fbdd0edc4f9]
2024-06-24 10:19:37 140453265478208 [Note] Server socket created on IP: '0.0.0.0'.
/lib64/libc.so.6(clone+0x3f)[0x7fbdd033af2f]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): is an invalid pointer
Connection ID (thread ID): 1
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

We think the query pointer is invalid, but we will try to print it anyway. 
Query:

Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             35758                35758                processes 
Max open files            4184                 4184                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       35758                35758                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: |/bin/false
.................

Please note the following:
- I renamed both ib_logfile0 and ib_logfile1 and they both got recreated when service.mysql command was ran.
- There is mysql.sock in /run/mysql
- The corrupt database I was trying to delete without success is still in /var/lib/mysql/

What do I need to do to get mariadb working again? Thank you.

Content reproduced on this site is the property of its respective owners, and this content is not reviewed in advance by MariaDB. The views, information and opinions expressed by this content do not necessarily represent those of MariaDB or any other party.

Mariadb crashed during deleting a database - Source

Products

Services

Pricing

Resources

About Us

Download MariaDB