10.4.14 cluster + rsync + ssl
Hello! I am asking to assist to setup MariaDB cluster + State Snapshot Transfers (SSTs) rsync + SSL. There are two servers Mariadb: Centos6 10.4.14 + Centos 7 10.4.14 with identical config files: server.cnf:
[server] max_connections=350 log-bin=/opt/mysql/binlog/mysql-bin log-bin-index=/opt/mysql/binlog/mysql-bin.index max_binlog_size=33554432 binlog-format = 'ROW' default-storage-engine=innodb innodb_autoinc_lock_mode=2 innodb_rollback_on_timeout=1 innodb_lock_wait_timeout=600 datadir=/opt/mysql/datadir tmpdir=/opt/mysql/tmp log_error=/opt/mysql/log/mysql.err [mysqld] ssl_cert = /etc/my.cnf.d/certs/server-cert.pem ssl_key = /etc/my.cnf.d/certs/server-key.pem ssl_ca = /etc/my.cnf.d/certs/ca.pem [embedded] [mariadb] [mariadb-10.4] # [sst] #tkey = /etc/my.cnf.d/certs/server-key.pem #tcert = /etc/my.cnf.d/certs/server-cert.pem
galera.cnf: <</code>> [galera] innodb_doublewrite=1 innodb_flush_log_at_trx_commit=1 bind-address=0.0.0.0 wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so wsrep_cluster_address="gcomm:1.1.1.16,1.1.1.15" wsrep_on=ON wsrep_cluster_name="cluster" wsrep_sst_method=rsync wsrep_node_address="1.1.1.16" wsrep_node_name="node16" wsrep_provider_options="socket.ssl_cert=/etc/my.cnf.d/certs/server-cert.pem;socket.ssl_key=/etc/my.cnf.d/certs/server-key.pem;socket.ssl_ca=/etc/my.cnf.d/certs/ca.pem" <</code>>
The cluster starts with these configuration files. But if you uncomment the lines
[sst] tkey = /etc/my.cnf.d/certs/server-key.pem tcert = /etc/my.cnf.d/certs/server-cert.pem
the cluster does not start with message: Centos6:
2020-09-25 7:41:06 1 [Note] WSREP: State transfer required: Group state: e330214a-fe78-11ea-9fef-9779040d909c:30 Local state: 00000000-0000-0000-0000-000000000000:-1 2020-09-25 7:41:06 1 [Note] WSREP: Server status change connected -> joiner 2020-09-25 7:41:06 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2020-09-25 7:41:06 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '1.1.1.16' --datadir '/opt/mysql/datadir/' --parent '31813' --binlog '/opt/mysql/binlog/mysql-bin' --binlog-index '/opt/mysql/binlog/mysql-bin.index' --mysqld-args --basedir=/usr --datadir=/opt/mysql/datadir --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so --wsrep_on=ON --log-error=/opt/mysql/log/mysql.err --pid-file=/opt/mysql/datadir/centos610.pid --socket=/opt/mysql/mysql.sock --wsrep_start_position=e330214a-fe78-11ea-9fef-9779040d909c:24' 2020-09-25 7:41:06 0 [Note] WSREP: Joiner monitor thread started to monitor 2020.09.25 07:41:06 LOG4[31882:139836024436672]: Diffie-Hellman initialization failed 2020-09-25 7:41:06 1 [Note] WSREP: Prepared SST request: rsync|1.1.1.16:4444/rsync_sst 2020-09-25 7:41:06 1 [Note] WSREP: ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 30, STRv: 3 2020-09-25 7:41:06 1 [Note] WSREP: IST receiver addr using ssl://1.1.1.16:4568 2020-09-25 7:41:06 1 [Note] WSREP: IST receiver using ssl 2020-09-25 7:41:06 1 [Note] WSREP: Prepared IST receiver for 0-30, listening at: ssl://1.1.1.16:4568 2020-09-25 7:41:06 0 [Note] WSREP: Member 0.0 (node16) requested state transfer from '*any*'. Selected 1.0 (node15)(SYNCED) as donor. 2020-09-25 7:41:06 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 30) 2020-09-25 7:41:06 1 [Note] WSREP: Requesting state transfer: success, donor: 1 2020-09-25 7:41:06 1 [Note] WSREP: Resetting GCache seqno map due to different histories. 2020-09-25 7:41:06 1 [Note] WSREP: GCache history reset: e330214a-fe78-11ea-9fef-9779040d909c:24 -> e330214a-fe78-11ea-9fef-9779040d909c:30 2020-09-25 7:41:06 1 [Note] WSREP: GCache DEBUG: RingBuffer::seqno_reset(): discarded 712 bytes 2020-09-25 7:41:06 1 [Note] WSREP: GCache DEBUG: RingBuffer::seqno_reset(): found 1/2 locked buffers 2020-09-25 7:41:06 0 [Warning] WSREP: 1.0 (node15): State transfer to 0.0 (node16) failed: -255 (Unknown error 255) 2020-09-25 7:41:06 0 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():1178: Will never receive state. Need to abort. 2020-09-25 7:41:06 0 [Note] WSREP: gcomm: terminating thread 2020-09-25 7:41:06 0 [Note] WSREP: gcomm: joining thread 2020-09-25 7:41:06 0 [Note] WSREP: gcomm: closing backend 2020-09-25 7:41:07 0 [Note] WSREP: view(view_id(NON_PRIM,527183aa-a894,2) memb { 527183aa-a894,0 } joined { } left { } partitioned { 890ce1c9-a7d9,0 }) 2020-09-25 7:41:07 0 [Note] WSREP: PC protocol downgrade 1 -> 0 2020-09-25 7:41:07 0 [Note] WSREP: view((empty)) 2020-09-25 7:41:07 0 [Note] WSREP: gcomm: closed
Answer
Fixed. On donor side: in the file /usr/bin/wsrep_sst_rsync (Centos7) change eval rsync
${STUNNEL:+--rsh="$STUNNEL"} \
to
eval rsync ${STUNNEL:+--rsh=\"$STUNNEL\"} \