Comments - Manual SST of Galera Cluster Node With Mariabackup

I prefer to use pure scp, it is faster and straightforward, no need to play with grastate.dat. Something like this:

Here we assume that we have one node (node1) running, started with galera_new_cluster. All other nodes are stopped. And MaxScales stopped

Perform this on all stopped nodes 2-5: Wipe datadirs on those stopped nodes (nodes 2-5), NOT on that running node! (node1). Also on slave if existing.

find /var/lib/mysql -type f -not -path "/var/lib/mysql/example/*" -delete

Perform this on all stopped nodes 2-5: Give permissions for your account to /var/lib/mysql. This is needed because root cannot scp files between the nodes. Use id <loginname> too see your group. Also on slave if existing.

chown -R myid:mygroup /var/lib/mysql/

Stop running node (node1)

systemctl stop mariadb

Start screens as user root for the scp’s on that node (node1) which was running Galera. You may want to utilise 4 terminals.

screen -S fornode2 
screen -S fornode3 
screen -S fornode4 
screen -S fornode5 
screen -S fornode6 (for possible existing slave)

And on every screen:

scp -pr /var/lib/mysql/* myid@node2IP:/var/lib/mysql/ && touch /tmp/scp_transfer_done_node2

scp -pr /var/lib/mysql/* myid@node3IP:/var/lib/mysql/ && touch /tmp/scp_transfer_done_node3

scp -pr /var/lib/mysql/* myid@node4IP:/var/lib/mysql/ && touch /tmp/scp_transfer_done_node4

scp -pr /var/lib/mysql/* myid@node5IP:/var/lib/mysql/ && touch /tmp/scp_transfer_done_node5

scp -pr /var/lib/mysql/* myid@node6IP:/var/lib/mysql/ && touch /tmp/scp_transfer_done_node6

Wait until scp’s are finished. (watch -n 30 -d /tmp/)

Clear logs on node 2-5 if writing logs to /var/lib/mysql. Also on slave if existing

 :> /var/lib/mysql/mariadb.err /var/lib/mysql/slow-queries.log

Set permissions back for mysql on nodes 2-5, also on slave if existing

chown -R mysql:mysql /var/lib/mysql/

Bootstrap Galera Cluster from node (node1)

galera_new_cluster & 
tail -f /var/lib/mysql/mariadb.err

Start other nodes (2-5) one-by-one. Wait that node has joined to the cluster with IST before proceeding to start next node.

systemctl start mariadb & 
tail -f /var/lib/mysql/mariadb.err

Start MaxScales Start replication

SST is a big headache. Although i've been able to follow the entire procedure (after some modifications because you assume that people use standard folders, standard hosts and standard ports) joiners kept asking data from donor, erasing data folder. I probably had strange problems with network. At the end I went back to rsync sst restarting servers one by one. One node is retriving data (I hope it wont stop in the middle of the phase)

i write here the problem that i had because i'm pretty sure someone could have the same problem and it can be useful: When you have a big database, invoking a service restart on joiner nodes from a remote ssh shell can cause internally a timeout during the sync phase. I've solved using a remote desktop service