MariaDB service fails to start after InnoDB/XtraDB crash

Hi, I've been running a script, inserting a large amount of data to several MySQL tables. Apparently, InnoDB crashed at some point (when ibdata1 reached 1.6 Gb). I had to restart the computer and when I tried to restart MariaDB service, I got the following errors:

Jun 11 08:58:02 honolulu.bi.technion.ac.il mysqld_safe[7970]: 150611  8:58:02  InnoDB: Operating system error number 5 in a file operation.
Jun 11 08:58:02 honolulu.bi.technion.ac.il mysqld_safe[7970]: InnoDB: Error number 5 means 'Input/output error'.
Jun 11 08:58:02 honolulu.bi.technion.ac.il mysqld_safe[7970]: InnoDB: Some operating system error numbers are described at
Jun 11 08:58:02 honolulu.bi.technion.ac.il mysqld_safe[7970]: InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html
Jun 11 08:58:02 honolulu.bi.technion.ac.il mysqld_safe[7970]: InnoDB: File operation call: 'read'.
Jun 11 08:58:02 honolulu.bi.technion.ac.il mysqld_safe[7970]: InnoDB: Cannot continue operation.'
Jun 11 08:58:02 honolulu.bi.technion.ac.il systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
Jun 11 08:58:02 honolulu.bi.technion.ac.il systemd[1]: mariadb.service: control process exited, code=exited status=1
Jun 11 08:58:02 honolulu.bi.technion.ac.il systemd[1]: Failed to start MariaDB database server.
Jun 11 08:58:02 honolulu.bi.technion.ac.il systemd[1]: Unit mariadb.service entered failed state.

I checked /var/log/mariadb/mariadb.log to find out what might have caused the crash:

140911 10:40:38  InnoDB: Error: Write to file ./ibdata1 failed at offset 0 220200960.
InnoDB: 1048576 bytes should have been written, only 0 were written.
InnoDB: Operating system error number 28.
InnoDB: Check that your OS and file system support files of this size.
InnoDB: Check also that the disk is not full or a disk quota exceeded.
InnoDB: Error number 28 means 'No space left on device'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html
140911 10:40:38  InnoDB: Assertion failure in thread 139942049851136 in file os0file.c line 4382
InnoDB: Failing assertion: ret
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately a150505 07:59:38 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150505 07:59:38 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.SFCcCu' --pid-file='/var/lib/mysql/honolulu.bi.technion.ac.il-recover.pid'
150505 07:59:41 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
150505  7:59:41 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'
150505  7:59:41 InnoDB: The InnoDB memory heap is disabled
150505  7:59:41 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150505  7:59:41 InnoDB: Compressed tables use zlib 1.2.8
150505  7:59:41 InnoDB: Using Linux native AIO
150505  7:59:41 InnoDB: Initializing buffer pool, size = 128.0M
150505  7:59:41 InnoDB: Completed initialization of buffer pool
150505  7:59:41 InnoDB: highest supported file format is Barracuda.
150505  7:59:41  InnoDB: Waiting for the background threads to start
150505  7:59:42 Percona XtraDB (http://www.percona.com) 5.5.37-MariaDB-35.0 started; log sequence number 458740332
150505  7:59:42 [Note] Plugin 'FEEDBACK' is disabled.
150505  7:59:42 [Note] Server socket created on IP: '0.0.0.0'.
150505  7:59:42 [ERROR] mysqld: Table './mysql/user' is marked as crashed and should be repaired
150505  7:59:42 [Warning] Checking table:   './mysql/user'
150505  7:59:42 [ERROR] mysql.user: 1 client is using or hasn't closed the table properly
150505  7:59:42 [ERROR] mysqld: Table './mysql/db' is marked as crashed and should be repaired
150505  7:59:42 [Warning] Checking table:   './mysql/db'
150505  7:59:42 [ERROR] mysql.db: 1 client is using or hasn't closed the table properly
150505  7:59:42 [Note] Event Scheduler: Loaded 0 events
150505  7:59:42 [Note] WSREP: Read nil XID from storage engines, skipping position init
150505  7:59:42 [Note] WSREP: wsrep_load(): loading provider library 'none'
150505  7:59:42 [Note] [Debug] WSREP: dummy_init
150505  7:59:42 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.38-MariaDB-wsrep'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server, wsrep_25.10.r3997
150608 04:43:00 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150608 04:43:00 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.7Yb3bw' --pid-file='/var/lib/mysql/honolulu.bi.technion.ac.il-recover.pid'
150608 04:43:13 mysqld_safe WSREP: Failed to recover position: 
'150608  4:43:01 InnoDB: The InnoDB memory heap is disabled
150608  4:43:01 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150608  4:43:01 InnoDB: Compressed tables use zlib 1.2.8
150608  4:43:01 InnoDB: Using Linux native AIO
150608  4:43:01 InnoDB: Initializing buffer pool, size = 128.0M
150608  4:43:01 InnoDB: Completed initialization of buffer pool
150608  4:43:01 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 4676403586
150608  4:43:01  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Error: tried to read 1048576 bytes at offset 0 1048576.
InnoDB: Was only able to read 655360.
150608  4:43:13  InnoDB: Operating system error number 5 in a file operation.
InnoDB: Error number 5 means 'Input/output error'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html
InnoDB: File operation call: 'read'.
InnoDB: Cannot continue operation.'

I have to say that there shouldn't be any disk space problem:

df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5       819G   27G  750G   4% /
devtmpfs         7.9G       0   7.9G    0% /dev
tmpfs              7.9G        0   7.9G    0% /dev/shm
tmpfs              7.9G  8.9M   7.9G    1% /run
tmpfs              7.9G        0   7.9G    0% /sys/fs/cgroup
tmpfs              7.9G   12K    7.9G   1% /tmp
/dev/sda3       1.9T  123M   1.8T    1% /home
/dev/sda2      477M  112M  336M  25% /boot

The MariaDB is currently installed in default configuration.

I really need help on this issue... 1. What might have cause the problem? Do I have to configure some InnoDB parameters? Maybe the ibdata file size is a problem? 2. How to recover my data and restart the MariaDB service? (I tried all kind of tips I found in other forums, but I couldn't backup the ibdata1 file or start InnoDB in recovery mode).

Answer

Note these lines in the log:

InnoDB: Operating system error number 28.
InnoDB: Check that your OS and file system support files of this size.
InnoDB: Check also that the disk is not full or a disk quota exceeded.
InnoDB: Error number 28 means 'No space left on device'.

May be it's, indeed, disk quota. Or may be the disk was full, but when you run your df -h it got some free space. Either way, error 28 can pretty much mean only one thing — disk full.

Comments

Comments loading...
Content reproduced on this site is the property of its respective owners, and this content is not reviewed in advance by MariaDB. The views, information and opinions expressed by this content do not necessarily represent those of MariaDB or any other party.