MaxScale Troubleshooting
Contents
SystemD Watchdog Kills MaxScale
This can occur if a reverse DNS name lookup takes a long time. To disable name lookups of clients, add skip_name_resolve=true under the [maxscale]
section.
High Memory Usage
MaxScale starting with 22.08.4
The default value of writeq_high_water was lowered to 64KiB to reduce excessive memory usage. This change should result in a net decrease in memory usage and possibly a small improvement in performance.
Set writeq_high_water and writeq_low_water to lower values, for example writeq_high_water=512
and writeq_low_water=128
. The default is to buffer a maximum of 16MB in memory before network throttling begins which under intensive loads can result in a large amount of memory being used per client.
The query classifier cache in MaxScale by default takes up to 15% of memory to cache query classification data. This value can be lowered using the query_classifier_cache_size parameter.
The retain_last_statements and session_trace debugging parameters can cause memory usage to increase. Disabling them under intensive loads is recommended if they are not needed. Note that the maxctrl list queries
requires that retain_last_statements=1
is set.
Authentication Errors
Access Denied
If you are receiving authentication errors like this:
ERROR 1045 (28000): Access denied for user 'bob'@'office' (using password: YES)
Make sure you create users for both 'bob'@'office'
and 'bob'@'maxscale'
. The host 'office'
is where the client is attempting to connect from and 'maxscale'
is the host where MaxScale is installed.
If you do not want to create a second set of users, you can enable proxy_protocol in MaxScale and configure the MariaDB server to allow proxied connections from the MaxScale host.
Verifying that a user is allowed to connect
- MaxScale connection
- SSH to the server where MaxScale is installed
- Connect to MariaDB
- Check output of
SHOW GRANTS
- Client connection
- SSH to theserver where client is connecting from
- Connect to MariaDB
- Check output of
SHOW GRANTS
Checking MaxScale has correct grants
Service Grants
Make sure that the MaxScale services have a user configured and that it has the correct grants. Refer to the MariaDB protocol documentation on what grants are required for services.
Monitor Grants
The monitor user requires different grants than the service user and each monitor type requires different grants.
Other Errors
For all authentication and permission related errors, add debug=enable-statement-logging
under the [maxscale]
section of your MaxScale configuration file. This will cause all SQL statements to be logged on the notice level which will help you figure out what the problem is.
Access denied errors for user root!
If you want to connect as root, you'll need to add enable_root_user=true to the service.
Access denied on databases/tables containing underscores
There seems to be a bug for databases containing underscores. Connect as root and use "SHOW GRANTS FOR user".
GRANT SELECT ON `my\_database`.* TO 'user'@'%' <-- bad
GRANT SELECT ON `my_database`.* TO 'user'@'%' <-- good
If you got a grant containing a escaped underscore, you can add the strip_db_esc=true parameter to the service to automatically strip escape characters or just replace the grant with a unescaped one.
System Errors
Failed to write message: 11, Resource temporarily unavailable
MaxScale starting with 22.08.0
MaxScale 22.08 no longer uses pipes for internal communication. This means that this error is never logged and the pipe size no longer needs to be adjusted.
Starting with MaxScale 2.1 and until MaxScale 6, MaxScale can log the Failed to write message: 11, Resource temporarily unavailable
message under extremely intensive workloads (see MXS-1983).
The first action to take when these messages are encountered is to upgrade your MaxScale installation to the latest version. Whenever this message is seen, it means that something is causing the internal message queue in MaxScale to fill up. More often than not it is a sign of a possible bug in MaxScale and most likely has been fixed in the most recent release of MaxScale.
To correct it increase the pipe buffer size from the default 1MB to a higher value. At least 8MB is recommended and should be increased until the message stops appearing.
To set the pipe buffer size, execute the following command.
sudo sysctl -w fs.pipe-max-size=8388608
Before MaxScale 6.4.5, messages in the queue would end up taking 4096 bytes of memory which translated to a maximum of 256 messages with a 1MiB pipe size. In MaxScale 6.4.5, the size is 24 bytes which causes the maximum limit to be increased to about 43k messages.
If after all these actions you still see these warnings, please open a bug report on the MariaDB Jira under the MaxScale project.
Error 23: Too many open files
This is a common error when system limits for open files is too low. The fix to this is to increase the limits.
Systemd
Edit or add LimitNOFILE=<number of files>
under the [Service]
section in /usr/lib/systemd/system/maxscale.service
.
Listeners not starting
While having multiple listeners, it takes time to load all user data from the backend servers. The default value of TimeoutStartSec in /usr/lib/systemd/system/maxscale.service
for MaxScale is 120. If all listeners do not start before Systemd times out, increase the value from 120 to 360.
Binlogrouter
Commands not Working
Make sure you are connecting on the port where the binlogrouter is listening. A common mistake is to connect to a readwritesplit or readconnroute port and execute the replication configuration commands there.
MaxScale CDC: Avrorouter
For most problems, resetting the conversion state is the solution. If the conversion repeatedly stops at a certain point, please open a bug report.
Resetting conversion state
- Stop MaxScale
- Remove the
avro.index
andavro-conversion.ini
files along with any generated.avro
files from the director where the Avro files are stored - Start MaxScale
Binlog files are not found
Make sure the start_index
parameter is set to the lowest binlog file number. For example, to start from mariadb-bin-000005
, set start_index=5
.
Access denied to CDC interface
Create the user with maxadmin call command cdc add_user <service name> <user> <password>
or maxctrl call command cdc add_user <service name> <user> <password>
.