Local state seqno out of sync with group
We have MariaDB and Galera working in our lab on a three node cluster. When setting them up the wsrep options for each node were as follows:
node1: wsrep_cluster_address="gcomm://" node2: wsrep_cluster_address="gcomm://172.30.1.3" node3: wsrep_cluster_address="gcomm://172.30.1.1"
Once all the nodes were in sync I shutdown node1 and set its wsrep_cluster_address to be 172.30.1.2. This I thought would allow the three nodes to keep in sync. What happened though was when I started MariaDB backup I received the error:
130702 8:35:51 [ERROR] WSREP: Local state seqno (16053) is greater than group seqno (15898): states diverged. Aborting to avoid potential data loss. Remove '/var/lib/mysql//grastate.dat' file and restart if you wish to continue. (FATAL) at galera/src/replicator_str.cpp:state_transfer_required():34
Why has this happened ? Is it safe to proceed with the recommendation of removing grastate.dat ?
Answer Answered by Alex yurchenko in this comment.
It is impossible to diagnose what happened here without full logs from the node in question and at least one node from the cluster. Most likely you did not exactly what you described, maybe in a subtle and non-obvious way.
It is safe to proceed with removing grastate.dat in a sense that the node will join the cluster and synchronize with it by copying the state with seqno 15898. However you will lose the 155 extra transactions that you have there now.