- SystemD Watchdog Kills MaxScale
- High Memory Usage
- Authentication Errors
- System Errors
- MaxScale CDC: Avrorouter
SystemD Watchdog Kills MaxScale
This can occur if a reverse DNS name lookup takes a long time. To disable name lookups of clients, add skip_name_resolve=true under the
High Memory Usage
Set writeq_high_water and writeq_low_water to lower values, for example
writeq_low_water=128. The default is to buffer a maximum of 16MB in memory before network throttling begins which under intensive loads can result in a large amount of memory being used per client.
The query classifier cache in MaxScale by default takes up to 15% of memory to cache query classification data. This value can be lowered using the query_classifier_cache_size parameter.
The retain_last_statements and session_trace debugging parameters can cause memory usage to increase. Disabling them under intensive loads is recommended if they are not needed. Note that the
maxctrl list queries requires that
retain_last_statements=1 is set.
If you are receiving authentication errors like this:
ERROR 1045 (28000): Access denied for user 'bob'@'office' (using password: YES)
Make sure you create users for both
'bob'@'maxscale'. The host
'office' is where the client is attempting to connect from and
'maxscale' is the host where MaxScale is installed.
Verifying that a user is allowed to connect
- MaxScale connection
- SSH to the server where MaxScale is installed
- Connect to MariaDB
- Check output of
- Client connection
- SSH to theserver where client is connecting from
- Connect to MariaDB
- Check output of
Checking MaxScale has correct grants
Make sure that the MaxScale services have a user configured and that it has the correct grants. Refer to the MariaDB protocol documentation on what grants are required for services.
The monitor user requires different grants than the service user and each monitor type requires different grants.
For all authentication and permission related errors, add
debug=enable-statement-logging under the
[maxscale] section of your MaxScale configuration file. This will cause all SQL statements to be logged on the notice level which will help you figure out what the problem is.
Access denied errors for user root!
If you want to connect as root, you'll need to add enable_root_user=true to the service.
Access denied on databases/tables containing underscores
There seems to be a bug for databases containing underscores. Connect as root and use "SHOW GRANTS FOR user".
GRANT SELECT ON `my\_database`.* TO 'user'@'%' <-- bad
GRANT SELECT ON `my_database`.* TO 'user'@'%' <-- good
If you got a grant containing a escaped underscore, you can add the strip_db_esc=true parameter to the service to automatically strip escape characters or just replace the grant with a unescaped one.
Failed to write message: 11, Resource temporarily unavailable
Starting with MaxScale 2.1 and until MaxScale 6, MaxScale can log the
Failed to write message: 11, Resource temporarily unavailable message under extremely intensive workloads (see MXS-1983). MaxScale 22.08 no longer uses pipes for internal communication which means this problem will not appear there.
To correct it increase the pipe buffer size from the default 1MB to a higher value. At least 8MB is recommended and should be increased until the message stops appearing.
To set the pipe buffer size, execute the following command.
sudo sysctl -w fs.pipe-max-size=8388608
Error 23: Too many open files
This is a common error when system limits for open files is too low. The fix to this is to increase the limits.
Edit or add
LimitNOFILE=<number of files> under the
[Service] section in
Listeners not starting
While having multiple listeners, it takes time to load all user data from the backend servers. The default value of TimeoutStartSec in
/usr/lib/systemd/system/maxscale.service for MaxScale is 120. If all listeners do not start before Systemd times out, increase the value from 120 to 360.
Commands not Working
Make sure you are connecting on the port where the binlogrouter is listening. A common mistake is to connect to a readwritesplit or readconnroute port and execute the replication configuration commands there.
MaxScale CDC: Avrorouter
For most problems, resetting the conversion state is the solution. If the conversion repeatedly stops at a certain point, please open a bug report.
Resetting conversion state
- Stop MaxScale
- Remove the
avro-conversion.inifiles along with any generated
.avrofiles from the director where the Avro files are stored
- Start MaxScale
Binlog files are not found
Make sure the
start_index parameter is set to the lowest binlog file number. For example, to start from
Access denied to CDC interface
Create the user with
maxadmin call command cdc add_user <service name> <user> <password> or
maxctrl call command cdc add_user <service name> <user> <password>.