When 2nd node attempts to add to galera, donor gets: --innobackupex: unknown option '--wsrep-new-cluster'

Hi community,

I am standing up a galera cluster like others have set up, so I am confident in the basic galera.cnf, access, users, etc. I start the first node with "galera_new_cluster" as usual and it stays up and show a cluster size of 1. I add the 2nd node, 172.20.7.212, with "systemctl start mariadb". The first node, being the donor, responds with this in the log:

WSREP_SST: [INFO] Streaming the backup to joiner at 172.20.7.212 4444 (20210428 15:02:25.997)

WSREP_SST: [INFO] Evaluating /usr/bin/mariabackup --innobackupex --no-version-check $tmpopts $INNOEXTRA --galera-info --stream=$sfmt $itmpdir 2> /var/lib/mysqlmariabackup.backup.log | socat -u stdio TCP:172.20.7.212:4444; RC=( ${PIPESTATUS[@]} ) (20210428 15:02:25.999)

WSREP_SST: [ERROR] /usr/bin/mariabackup finished with error: 2. Check syslog or /var/lib/mysql/mariabackup.backup.log for details (20210428 15:02:26.013)

WSREP_SST: [ERROR] Cleanup after exit with status:22 (20210428 15:02:26.014)

And that 2nd node fails to start. So I look at /var/lib/mysql/mariabackup.backup.log on the donor. That file is strange, it appears to be getting created and deleted very frequently. And it is called/var/lib/mysqlmariabackup.backup.log with two forward slashes as shown. It contains:

--innobackupex: unknown option '--wsrep-new-cluster'

Similar errors show up in the log:

2021-04-28 16:07:58 0 [ERROR] WSREP: Failed to read from: wsrep_sst_mariabackup --role 'donor' --address '172.20.7.212:4444/xtrabackup_sst1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --gtid '374ac72d-a79e-11eb-8dc7-6f2622c213bd:811' --gtid-domain-id '0' --mysqld-args --wsrep-new-cluster --wsrep_start_position=374ac72d-a79e-11eb-8dc7-6f2622c213bd:809

2021-04-28 16:07:58 0 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'donor' --address '172.20.7.212:4444/xtrabackup_sst1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --gtid '374ac72d-a79e-11eb-8dc7-6f2622c213bd:811' --gtid-domain-id '0' --mysqld-args --wsrep-new-cluster --wsrep_start_position=374ac72d-a79e-11eb-8dc7-6f2622c213bd:809: 1 (Operation not permitted)

If I run those commands on the command line I get the same error in the mariabackup.backup.log file

wsrep_sst_mariabackup --role 'donor' --address '172.20.7.212:4444/xtrabackup_sst1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --gtid '374ac72d-a79e-11eb-8dc7-6f2622c213bd:811' --gtid-domain-id '0' --mysqld-args --wsrep-new-cluster --wsrep_start_position=374ac72d-a79e-11eb-8dc7-6f2622c213bd:809

WSREP_SST: [INFO] Streaming with xbstream (20210428 16:57:09.815)

WSREP_SST: [INFO] Using socat as streamer (20210428 16:57:09.817)

WSREP_SST: [INFO] Using /tmp/tmp.e4YEwI3vbA as xtrabackup temporary directory (20210428 16:57:09.833)

WSREP_SST: [INFO] Using /tmp/tmp.adYYlhKsqj as innobackupex temporary directory (20210428 16:57:09.836)

WSREP_SST: [INFO] Streaming GTID file before SST (20210428 16:57:09.839)

WSREP_SST: [INFO] Evaluating mbstream -c ${INFO_FILE} | socat -u stdio TCP:172.20.7.212:4444; RC=( ${PIPESTATUS[@]} ) (20210428 16:57:09.842)

WSREP_SST: [INFO] Sleeping before data transfer for SST (20210428 16:57:09.848)

WSREP_SST: [INFO] Streaming the backup to joiner at 172.20.7.212 4444 (20210428 16:57:19.850)

WSREP_SST: [INFO] Evaluating /bin/mariabackup --innobackupex --no-version-check $tmpopts $INNOEXTRA --galera-info --stream=$sfmt $itmpdir 2> /var/lib/mysqlmariabackup.backup.log | socat -u stdio TCP:172.20.7.212:4444; RC=( ${PIPESTATUS[@]} ) (20210428 16:57:19.852) 2021/04/28 16:57:19 socat[25372] E connect(5, AF=2 172.20.7.212:4444, 16): Connection refused

WSREP_SST: [ERROR] /bin/mariabackup finished with error: 2. Check syslog or /var/lib/mysqlmariabackup.backup.log for details (20210428 16:57:19.865)

WSREP_SST: [ERROR] Cleanup after exit with status:22 (20210428 16:57:19.867)

WSREP_SST: [INFO] Cleaning up temporary directories (20210428 16:57:19.869)

Indeed, there is no --wsrep-new-cluster option for mariabackup. Maybe it could be in the $tmpopts or $INNOEXTRA vars?

What could result in this situation and how can a fix or work around it?

Answer Answered by Dennis Ordiway in this comment.

I figured this out. The version of MariaDB-backup was out of sync. MariaDB-backup was 10.4.18, everything else was 10.4.8. This was due to backup being installed later than the rest of MariaDB but from the same repo. The repos points to 10.4, but the patch level can change. Once I updated everything to 10.4.18 the other nodes joined the cluster.

Comments

Comments loading...
Content reproduced on this site is the property of its respective owners, and this content is not reviewed in advance by MariaDB. The views, information and opinions expressed by this content do not necessarily represent those of MariaDB or any other party.