10.4.14 cluster + rsync + ssl
Hello! I am asking to assist to setup MariaDB cluster + State Snapshot Transfers (SSTs) rsync + SSL. There are two servers Mariadb: Centos6 10.4.14 + Centos 7 10.4.14 with identical config files: server.cnf: [server] max_connections=350 log-bin=/opt/mysql/binlog/mysql-bin log-bin-index=/opt/mysql/binlog/mysql-bin.index max_binlog_size=33554432 binlog-format = 'ROW' default-storage-engine=innodb innodb_autoinc_lock_mode=2 innodb_rollback_on_timeout=1 innodb_lock_wait_timeout=600 datadir=/opt/mysql/datadir tmpdir=/opt/mysql/tmp log_error=/opt/mysql/log/mysql.err
[mysqld] ssl_cert = /etc/my.cnf.d/certs/server-cert.pem ssl_key = /etc/my.cnf.d/certs/server-key.pem ssl_ca = /etc/my.cnf.d/certs/ca.pem
[embedded]
[mariadb]
[mariadb-10.4]
- [sst]
- tkey = /etc/my.cnf.d/certs/server-key.pem
- tcert = /etc/my.cnf.d/certs/server-cert.pem
galera.cnf: [galera] innodb_doublewrite=1 innodb_flush_log_at_trx_commit=1 bind-address=0.0.0.0 wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so wsrep_cluster_address="gcomm:1.1.1.16,1.1.1.15" wsrep_on=ON wsrep_cluster_name="cluster" wsrep_sst_method=rsync wsrep_node_address="1.1.1.16" wsrep_node_name="node16" wsrep_provider_options="socket.ssl_cert=/etc/my.cnf.d/certs/server-cert.pem;socket.ssl_key=/etc/my.cnf.d/certs/server-key.pem;socket.ssl_ca=/etc/my.cnf.d/certs/ca.pem"
The cluster starts with these configuration files.
But if you uncomment the lines
[sst]
tkey = /etc/my.cnf.d/certs/server-key.pem
tcert = /etc/my.cnf.d/certs/server-cert.pem
the cluster does not start with message:
Centos6:
2020-09-25 7:41:06 1 [Note] WSREP: State transfer required:
Group state: e330214a-fe78-11ea-9fef-9779040d909c:30
Local state: 00000000-0000-0000-0000-000000000000:-1
2020-09-25 7:41:06 1 [Note] WSREP: Server status change connected -> joiner
2020-09-25 7:41:06 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-09-25 7:41:06 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '1.1.1.16' --datadir '/opt/mysql/datadir/' --parent '31813' --binlog '/opt/mysql/binlog/mysql-bin' --binlog-index '/opt/mysql/binlog/mysql-bin.index' --mysqld-args --basedir=/usr --datadir=/opt/mysql/datadir --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so --wsrep_on=ON --log-error=/opt/mysql/log/mysql.err --pid-file=/opt/mysql/datadir/centos610.pid --socket=/opt/mysql/mysql.sock --wsrep_start_position=e330214a-fe78-11ea-9fef-9779040d909c:24'
2020-09-25 7:41:06 0 [Note] WSREP: Joiner monitor thread started to monitor
2020.09.25 07:41:06 LOG4[31882:139836024436672]: Diffie-Hellman initialization failed
2020-09-25 7:41:06 1 [Note] WSREP: Prepared SST request: rsync|1.1.1.16:4444/rsync_sst
2020-09-25 7:41:06 1 [Note] WSREP: # IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 30, STRv: 3
2020-09-25 7:41:06 1 [Note] WSREP: IST receiver addr using ssl:1.1.1.16:4568
2020-09-25 7:41:06 1 [Note] WSREP: IST receiver using ssl
2020-09-25 7:41:06 1 [Note] WSREP: Prepared IST receiver for 0-30, listening at: ssl:1.1.1.16:4568
2020-09-25 7:41:06 0 [Note] WSREP: Member 0.0 (node16) requested state transfer from '*any*'. Selected 1.0 (node15)(SYNCED) as donor.
2020-09-25 7:41:06 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 30)
2020-09-25 7:41:06 1 [Note] WSREP: Requesting state transfer: success, donor: 1
2020-09-25 7:41:06 1 [Note] WSREP: Resetting GCache seqno map due to different histories.
2020-09-25 7:41:06 1 [Note] WSREP: GCache history reset: e330214a-fe78-11ea-9fef-9779040d909c:24 -> e330214a-fe78-11ea-9fef-9779040d909c:30
2020-09-25 7:41:06 1 [Note] WSREP: GCache DEBUG: RingBuffer::seqno_reset(): discarded 712 bytes
2020-09-25 7:41:06 1 [Note] WSREP: GCache DEBUG: RingBuffer::seqno_reset(): found 1/2 locked buffers
2020-09-25 7:41:06 0 [Warning] WSREP: 1.0 (node15): State transfer to 0.0 (node16) failed: -255 (Unknown error 255)
2020-09-25 7:41:06 0 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():1178: Will never receive state. Need to abort.
2020-09-25 7:41:06 0 [Note] WSREP: gcomm: terminating thread
2020-09-25 7:41:06 0 [Note] WSREP: gcomm: joining thread
2020-09-25 7:41:06 0 [Note] WSREP: gcomm: closing backend
2020-09-25 7:41:07 0 [Note] WSREP: view(view_id(NON_PRIM,527183aa-a894,2) memb {
527183aa-a894,0
} joined {
} left {
} partitioned {
890ce1c9-a7d9,0
})
2020-09-25 7:41:07 0 [Note] WSREP: PC protocol downgrade 1 -> 0
2020-09-25 7:41:07 0 [Note] WSREP: view((empty))
2020-09-25 7:41:07 0 [Note] WSREP: gcomm: closed
Answer
Fixed. On donor side: in the file /usr/bin/wsrep_sst_rsync (Centos7) change eval rsync
${STUNNEL:+--rsh="$STUNNEL"} \
to
eval rsync ${STUNNEL:+--rsh=\"$STUNNEL\"} \
