Migrating from MySql: MariaDB server closing client connections unexpectedly
We in the process of migrating from MySql to MariaDB due to licensing/commercial usage reasons.
We have successfully replaced the MySql connector jar with MariaDB client jar (first change) and are now trying to replace MySql server with MariaDB server without changing the data files.
All our applications run perfectly for about 8-12 hours after which we see the following exception:
org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Cannot open connection Caused by: org.hibernate.exception.JDBCConnectionException: Cannot open connection at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:74) at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43) at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29) at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:426) at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144) at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:119) at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:57) at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1326) at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:494) at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:315) at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:257) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:102) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:209) at $Proxy4.getMessageCountByStatus(Unknown Source) at com.onmobile.cmfweb.monitoring.CmfMessagesMonitor.getMessageCounts(CmfMessagesMonitor.java:56) at sun.reflect.GeneratedMethodAccessor625.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:270) at com.onmobile.cmfshare.MethodInvockingBean.invoke(MethodInvockingBean.java:28) at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:212) at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:79) at org.quartz.core.JobRunShell.run(JobRunShell.java:203) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520) Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to localhost:3306: unexpected end of stream, read 0bytes from 4 at org.mariadb.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:136) at org.mariadb.jdbc.internal.SQLExceptionMapper.throwException(SQLExceptionMapper.java:106) at org.mariadb.jdbc.Driver.connect(Driver.java:114) at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37) at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290) at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:840) at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95) at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:544) at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81) at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423) ... 21 more Caused by: org.mariadb.jdbc.internal.common.QueryException: Could not connect to localhost:3306: unexpected end of stream, read 0bytes from 4 at org.mariadb.jdbc.internal.mysql.MySQLProtocol.connect(MySQLProtocol.java:509) at org.mariadb.jdbc.internal.mysql.MySQLProtocol.connect(MySQLProtocol.java:669) at org.mariadb.jdbc.internal.mysql.MySQLProtocol.<init>(MySQLProtocol.java:264) at org.mariadb.jdbc.Driver.connect(Driver.java:110) ... 28 more Caused by: java.io.EOFException: unexpected end of stream, read 0bytes from 4 at org.mariadb.jdbc.internal.common.packet.buffer.ReadUtil.readFully(ReadUtil.java:84) at org.mariadb.jdbc.internal.common.packet.buffer.ReadUtil.readFully(ReadUtil.java:92) at org.mariadb.jdbc.internal.common.packet.RawPacket.nextPacket(RawPacket.java:77) at org.mariadb.jdbc.internal.common.packet.SyncPacketFetcher.getRawPacket(SyncPacketFetcher.java:67) at org.mariadb.jdbc.internal.mysql.MySQLProtocol.connect(MySQLProtocol.java:467) ... 31 more
The exceptions disappear if MySql server is used again.
From what I have debugged so far MariaDB server is closing the client connections for some reason. I initially suspected of idle connections but we are using Hibernate configurations like 'testOnBorrow' so it shouldn't be the case.
Can anyone help us to find out the cause for this and help us fix the problem? Is there any specific configuration in MariaDB that I should be doing?
I've even run our applications by increasing the value of the param 'interactive_timeout' in the MariaDB server but it did not help.
Btw, we are using Spring-Hibernate and pooling our connections using Commons-pool jar.
Any kind of help will be deeply appreciated.
PS: The same question is also posted in 'stackoverflow'.
Answer Answered by Vladislav Vaintroub in this comment.
You might be running into wait_timeout.
Increasing that, rather than interactive_timeout might help. MariaDB does not close client connections just for the fun of it, there is always a reason (of those reasons, there are not many - client closes connection, or timeout, or kill)
