IST replication failing on 2 node galera mariadb setup
We have a galera cluster setup with 2 nodes (master-slave setup) in a docker environment on 2 Rhel 7 hosts . All neccesary ports (3306,4444,4567,4568) are mounted and are opened in the firewalld. When starting the joiner an SST is completed successfully but when the joiner needs an IST due to network connectivity timeout this always fails with the error "Warning] WSREP: Failed to prepare for incremental state transfer: Failed to open IST listener at tcp:192.168.60.123:4568', asio error 'bind: Cannot assign requested address': 99 (Cannot assign requested address)". Does anybody have any clue as to what is the problem here? Below are some relevant logs and my.cnf settings
Joiner Log:
2017-06-01 19:40:56 140261471205312 [Note] mysqld (mysqld 10.1.21-MariaDB-1~jessie) starting as process 1 ... mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: Read nil XID from storage engines, skipping position init mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: wsrep_load(): Galera 25.3.19(r3667) by Codership Oy <[email protected]> loaded successfully. mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: CRC-32C: using hardware acceleration. mysql_1 | 2017-06-01 19:40:56 140261471205312 [Warning] WSREP: Could not open state file for reading: '/var/lib/mysql//grastate.dat' mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootsrap: 1 mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.60.123; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) -> new(00000000-0000-0000-0000-000000000000:-1) mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1 mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: wsrep_sst_grab() mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: Start replication mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1 mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: protonet asio version 0 mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: Using CRC-32C for message checksums. mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: backend: asio mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: gcomm thread scheduling priority set to other:0 mysql_1 | 2017-06-01 19:40:56 140261471205312 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory) mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: restore pc from disk failed mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: GMCast version 0 mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: (15b3ea58, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: (15b3ea58, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: EVS version 0 mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: gcomm: connecting to group 'mariadb-cluster-nmb', peer '192.168.114.39:,192.168.60.123:' mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: (15b3ea58, 'tcp://0.0.0.0:4567') connection established to 15b3ea58 tcp://172.18.0.1:4567 mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: (15b3ea58, 'tcp://0.0.0.0:4567') connection established to ce3e5068 tcp://192.168.114.39:4567 mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: (15b3ea58, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: declaring ce3e5068 at tcp://192.168.114.39:4567 stable mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: Node ce3e5068 state prim mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: view(view_id(PRIM,15b3ea58,14) memb { mysql_1 | 15b3ea58,0 mysql_1 | ce3e5068,0 mysql_1 | } joined { mysql_1 | } left { mysql_1 | } partitioned { mysql_1 | }) mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: save pc into disk mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: discarding pending addr without UUID: tcp://192.168.60.123:4567 mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: discarding pending addr proto entry 0x7f91279eafc0 mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: gcomm: connected mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: Opened channel 'mariadb-cluster-nmb' mysql_1 | 2017-06-01 19:40:56 140260113639168 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2 mysql_1 | 2017-06-01 19:40:56 140261471205312 [Note] WSREP: Waiting for SST to complete. mysql_1 | 2017-06-01 19:40:56 140260113639168 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 16019753-46e9-11e7-ba5e-f721bd4bc828 mysql_1 | 2017-06-01 19:40:56 140260113639168 [Note] WSREP: STATE EXCHANGE: sent state msg: 16019753-46e9-11e7-ba5e-f721bd4bc828 mysql_1 | 2017-06-01 19:40:56 140260113639168 [Note] WSREP: STATE EXCHANGE: got state msg: 16019753-46e9-11e7-ba5e-f721bd4bc828 from 0 (nmb_backup) mysql_1 | 2017-06-01 19:40:56 140260113639168 [Note] WSREP: STATE EXCHANGE: got state msg: 16019753-46e9-11e7-ba5e-f721bd4bc828 from 1 (nmb_main) mysql_1 | 2017-06-01 19:40:56 140260113639168 [Note] WSREP: Quorum results: mysql_1 | version = 4, mysql_1 | component = PRIMARY, mysql_1 | conf_id = 13, mysql_1 | members = 1/2 (joined/total), mysql_1 | act_id = 528895, mysql_1 | last_appl. = -1, mysql_1 | protocols = 0/7/3 (gcs/repl/appl), mysql_1 | group UUID = 625893a6-4554-11e7-b41d-dfe527fde688 mysql_1 | 2017-06-01 19:40:56 140260113639168 [Note] WSREP: Flow-control interval: [23, 23] mysql_1 | 2017-06-01 19:40:56 140260113639168 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 528895) mysql_1 | 2017-06-01 19:40:56 140261470890752 [Note] WSREP: State transfer required: mysql_1 | Group state: 625893a6-4554-11e7-b41d-dfe527fde688:528895 mysql_1 | Local state: 00000000-0000-0000-0000-000000000000:-1 mysql_1 | 2017-06-01 19:40:56 140261470890752 [Note] WSREP: New cluster view: global state: 625893a6-4554-11e7-b41d-dfe527fde688:528895, view# 14: Primary, number of nodes: 2, my index: 0, protocol version 3 mysql_1 | 2017-06-01 19:40:56 140261470890752 [Warning] WSREP: Gap in state sequence. Need state transfer. mysql_1 | 2017-06-01 19:40:56 140260084283136 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.60.123' --datadir '/var/lib/mysql/' --parent '1' '' ' mysql_1 | WSREP_SST: [INFO] Logging all stderr of SST/Innobackupex to syslog (20170601 19:40:57.289) mysql_1 | 2017-06-01 19:40:57 140261470890752 [Note] WSREP: Prepared SST request: xtrabackup-v2|192.168.60.123:4444/xtrabackup_sst//1 mysql_1 | 2017-06-01 19:40:57 140261470890752 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. mysql_1 | 2017-06-01 19:40:57 140261470890752 [Note] WSREP: REPL Protocols: 7 (3, 2) mysql_1 | 2017-06-01 19:40:57 140261470890752 [Note] WSREP: Assign initial position for certification: 528895, protocol version: 3 mysql_1 | 2017-06-01 19:40:57 140260172363520 [Note] WSREP: Service thread queue flushed. mysql_1 | 2017-06-01 19:40:57 140261470890752 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (625893a6-4554-11e7-b41d-dfe527fde688): 1 (Operation not permitted) mysql_1 | at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable. mysql_1 | 2017-06-01 19:40:57 140260113639168 [Note] WSREP: Member 0.0 (nmb_backup) requested state transfer from '*any*'. Selected 1.0 (nmb_main)(SYNCED) as donor. mysql_1 | 2017-06-01 19:40:57 140260113639168 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 528895) mysql_1 | 2017-06-01 19:40:57 140261470890752 [Note] WSREP: Requesting state transfer: success, donor: 1 mysql_1 | 2017-06-01 19:40:57 140261470890752 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) -> new(625893a6-4554-11e7-b41d-dfe527fde688:528895) mysql_1 | 2017-06-01 19:40:59 140260122031872 [Note] WSREP: (15b3ea58, 'tcp://0.0.0.0:4567') turning message relay requesting off mysql_1 | 2017-06-01 22:02:14 140260113639168 [Note] WSREP: 1.0 (nmb_main): State transfer to 0.0 (nmb_backup) complete. mysql_1 | 2017-06-01 22:02:14 140260113639168 [Note] WSREP: Member 1.0 (nmb_main) synced with group. mysql_1 | 2017-06-01 22:02:37 140261471205312 [Note] WSREP: SST complete, seqno: 539477 mysql_1 | 2017-06-01 22:02:37 7f912b2ab7c0 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator. mysql_1 | 2017-06-01 22:02:37 7f912b2ab7c0 InnoDB: Warning: Using innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be removed in future releases. Please use READ COMMITTED transaction isolation level instead, see http://dev.mysql.com/doc/refman/5.6/en/set-transaction.html. mysql_1 | 2017-06-01 22:02:37 140261471205312 [Note] InnoDB: Using mutexes to ref count buffer pool pages mysql_1 | 2017-06-01 22:02:37 140261471205312 [Note] InnoDB: The InnoDB memory heap is disabled mysql_1 | 2017-06-01 22:02:37 140261471205312 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins mysql_1 | 2017-06-01 22:02:37 140261471205312 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier mysql_1 | 2017-06-01 22:02:37 140261471205312 [Note] InnoDB: Compressed tables use zlib 1.2.8 mysql_1 | 2017-06-01 22:02:37 140261471205312 [Note] InnoDB: Using Linux native AIO mysql_1 | 2017-06-01 22:02:37 140261471205312 [Note] InnoDB: Using SSE crc32 instructions mysql_1 | 2017-06-01 22:02:37 140261471205312 [Note] InnoDB: Initializing buffer pool, size = 7.0G mysql_1 | 2017-06-01 22:02:39 140261471205312 [Note] InnoDB: Completed initialization of buffer pool mysql_1 | 2017-06-01 22:02:39 140261471205312 [Note] InnoDB: Highest supported file format is Barracuda. mysql_1 | 2017-06-01 22:03:05 140261471205312 [Note] InnoDB: 128 rollback segment(s) are active. mysql_1 | 2017-06-01 22:03:05 140261471205312 [Note] InnoDB: Waiting for purge to start mysql_1 | 2017-06-01 22:03:05 140261471205312 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.34-79.1 started; log sequence number 740320502294 mysql_1 | 2017-06-01 22:03:05 140241654482688 [Note] InnoDB: Dumping buffer pool(s) not yet started mysql_1 | 2017-06-01 22:03:05 140261471205312 [Note] Plugin 'FEEDBACK' is disabled. mysql_1 | 2017-06-01 22:03:05 140261471205312 [Note] Server socket created on IP: '0.0.0.0'. mysql_1 | 2017-06-01 22:03:06 140261471205312 [Warning] 'proxies_priv' entry '@% root@mysql' ignored in --skip-name-resolve mode. mysql_1 | 2017-06-01 22:03:06 140261471205312 [Note] WSREP: Signalling provider to continue. mysql_1 | 2017-06-01 22:03:06 140261471205312 [Note] WSREP: SST received: 625893a6-4554-11e7-b41d-dfe527fde688:539477 mysql_1 | 2017-06-01 22:03:06 140260113639168 [Note] WSREP: 0.0 (nmb_backup): State transfer from 1.0 (nmb_main) complete. mysql_1 | 2017-06-01 22:03:06 140260113639168 [Note] WSREP: Shifting JOINER -> JOINED (TO: 539502) mysql_1 | 2017-06-01 22:03:06 140261471205312 [Note] Reading of all Master_info entries succeded mysql_1 | 2017-06-01 22:03:06 140261471205312 [Note] Added new Master_info '' to hash table mysql_1 | 2017-06-01 22:03:06 140261471205312 [Note] mysqld: ready for connections. mysql_1 | Version: '10.1.21-MariaDB-1~jessie' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution mysql_1 | 2017-06-01 22:03:06 140260113639168 [Note] WSREP: Member 0.0 (nmb_backup) synced with group. mysql_1 | 2017-06-01 22:03:06 140260113639168 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 539502) mysql_1 | 2017-06-01 22:03:06 140261418715904 [Note] WSREP: Synchronized with group, ready for connections mysql_1 | 2017-06-01 22:03:06 140261418715904 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. mysql_1 | 2017-06-02 12:19:22 140260122031872 [Note] WSREP: (15b3ea58, 'tcp://0.0.0.0:4567') connection to peer ce3e5068 with addr tcp://192.168.114.39:4567 timed out, no messages seen in PT3S mysql_1 | 2017-06-02 12:19:22 140260122031872 [Note] WSREP: (15b3ea58, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.114.39:4567 mysql_1 | 2017-06-02 12:19:23 140260122031872 [Note] WSREP: (15b3ea58, 'tcp://0.0.0.0:4567') reconnecting to ce3e5068 (tcp://192.168.114.39:4567), attempt 0 mysql_1 | 2017-06-02 12:19:24 140260122031872 [Note] WSREP: evs::proto(15b3ea58, OPERATIONAL, view_id(REG,15b3ea58,14)) suspecting node: ce3e5068 mysql_1 | 2017-06-02 12:19:24 140260122031872 [Note] WSREP: evs::proto(15b3ea58, OPERATIONAL, view_id(REG,15b3ea58,14)) suspected node without join message, declaring inactive mysql_1 | 2017-06-02 12:19:25 140260122031872 [Note] WSREP: view(view_id(NON_PRIM,15b3ea58,14) memb { mysql_1 | 15b3ea58,0 mysql_1 | } joined { mysql_1 | } left { mysql_1 | } partitioned { mysql_1 | ce3e5068,0 mysql_1 | }) mysql_1 | 2017-06-02 12:19:25 140260122031872 [Note] WSREP: view(view_id(NON_PRIM,15b3ea58,15) memb { mysql_1 | 15b3ea58,0 mysql_1 | } joined { mysql_1 | } left { mysql_1 | } partitioned { mysql_1 | ce3e5068,0 mysql_1 | }) mysql_1 | 2017-06-02 12:19:25 140260113639168 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 mysql_1 | 2017-06-02 12:19:25 140260113639168 [Note] WSREP: Flow-control interval: [16, 16] mysql_1 | 2017-06-02 12:19:25 140260113639168 [Note] WSREP: Received NON-PRIMARY. mysql_1 | 2017-06-02 12:19:25 140260113639168 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 680188) mysql_1 | 2017-06-02 12:19:25 140260113639168 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 mysql_1 | 2017-06-02 12:19:25 140260113639168 [Note] WSREP: Flow-control interval: [16, 16] mysql_1 | 2017-06-02 12:19:25 140260113639168 [Note] WSREP: Received NON-PRIMARY. mysql_1 | 2017-06-02 12:19:25 140261419019008 [Note] WSREP: New cluster view: global state: 625893a6-4554-11e7-b41d-dfe527fde688:680188, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3 mysql_1 | 2017-06-02 12:19:25 140261419019008 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. mysql_1 | 2017-06-02 12:19:25 140261419019008 [Note] WSREP: New cluster view: global state: 625893a6-4554-11e7-b41d-dfe527fde688:680188, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3 mysql_1 | 2017-06-02 12:19:25 140261419019008 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. mysql_1 | 2017-06-02 12:19:26 140260122031872 [Note] WSREP: (15b3ea58, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://192.168.114.39:4567 timed out, no messages seen in PT3S mysql_1 | 2017-06-02 12:19:30 140260122031872 [Note] WSREP: (15b3ea58, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://192.168.114.39:4567 timed out, no messages seen in PT3S mysql_1 | 2017-06-02 12:19:32 140260122031872 [Note] WSREP: (15b3ea58, 'tcp://0.0.0.0:4567') connection established to ce3e5068 tcp://192.168.114.39:4567 mysql_1 | 2017-06-02 12:19:33 140260122031872 [Note] WSREP: view(view_id(NON_PRIM,15b3ea58,15) memb { mysql_1 | 15b3ea58,0 mysql_1 | } joined { mysql_1 | } left { mysql_1 | } partitioned { mysql_1 | ce3e5068,0 mysql_1 | }) mysql_1 | 2017-06-02 12:19:33 140260122031872 [Note] WSREP: declaring ce3e5068 at tcp://192.168.114.39:4567 stable mysql_1 | 2017-06-02 12:19:33 140260113639168 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 mysql_1 | 2017-06-02 12:19:33 140260113639168 [Note] WSREP: Flow-control interval: [16, 16] mysql_1 | 2017-06-02 12:19:33 140260113639168 [Note] WSREP: Received NON-PRIMARY. mysql_1 | 2017-06-02 12:19:33 140261419019008 [Note] WSREP: New cluster view: global state: 625893a6-4554-11e7-b41d-dfe527fde688:680188, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3 mysql_1 | 2017-06-02 12:19:33 140261419019008 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. mysql_1 | 2017-06-02 12:19:33 140260122031872 [Note] WSREP: Node ce3e5068 state prim mysql_1 | 2017-06-02 12:19:33 140260122031872 [Note] WSREP: view(view_id(PRIM,15b3ea58,16) memb { mysql_1 | 15b3ea58,0 mysql_1 | ce3e5068,0 mysql_1 | } joined { mysql_1 | } left { mysql_1 | } partitioned { mysql_1 | }) mysql_1 | 2017-06-02 12:19:33 140260122031872 [Note] WSREP: save pc into disk mysql_1 | 2017-06-02 12:19:33 140260113639168 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2 mysql_1 | 2017-06-02 12:19:33 140260113639168 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 96f8c457-4774-11e7-bb3f-2f97b0c8026b mysql_1 | 2017-06-02 12:19:33 140260113639168 [Note] WSREP: STATE EXCHANGE: sent state msg: 96f8c457-4774-11e7-bb3f-2f97b0c8026b mysql_1 | 2017-06-02 12:19:33 140260113639168 [Note] WSREP: STATE EXCHANGE: got state msg: 96f8c457-4774-11e7-bb3f-2f97b0c8026b from 0 (nmb_backup) mysql_1 | 2017-06-02 12:19:33 140260113639168 [Note] WSREP: STATE EXCHANGE: got state msg: 96f8c457-4774-11e7-bb3f-2f97b0c8026b from 1 (nmb_main) mysql_1 | 2017-06-02 12:19:33 140260113639168 [Note] WSREP: Quorum results: mysql_1 | version = 4, mysql_1 | component = PRIMARY, mysql_1 | conf_id = 15, mysql_1 | members = 1/2 (joined/total), mysql_1 | act_id = 680590, mysql_1 | last_appl. = 680152, mysql_1 | protocols = 0/7/3 (gcs/repl/appl), mysql_1 | group UUID = 625893a6-4554-11e7-b41d-dfe527fde688 mysql_1 | 2017-06-02 12:19:33 140260113639168 [Note] WSREP: Flow-control interval: [23, 23] mysql_1 | 2017-06-02 12:19:33 140260113639168 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 680590) mysql_1 | 2017-06-02 12:19:33 140261419019008 [Note] WSREP: State transfer required: mysql_1 | Group state: 625893a6-4554-11e7-b41d-dfe527fde688:680590 mysql_1 | Local state: 625893a6-4554-11e7-b41d-dfe527fde688:680188 mysql_1 | 2017-06-02 12:19:33 140261419019008 [Note] WSREP: New cluster view: global state: 625893a6-4554-11e7-b41d-dfe527fde688:680590, view# 16: Primary, number of nodes: 2, my index: 0, protocol version 3 mysql_1 | 2017-06-02 12:19:33 140261419019008 [Warning] WSREP: Gap in state sequence. Need state transfer. mysql_1 | 2017-06-02 12:19:33 140261419019008 [Note] WSREP: You have configured 'xtrabackup-v2' state snapshot transfer method which cannot be performed on a running server. Wsrep provider won't be able to fall back to it if other means of state transfer are unavailable. In that case you will need to restart the server. mysql_1 | 2017-06-02 12:19:33 140261419019008 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. mysql_1 | 2017-06-02 12:19:33 140261419019008 [Note] WSREP: REPL Protocols: 7 (3, 2) mysql_1 | 2017-06-02 12:19:33 140261419019008 [Note] WSREP: Assign initial position for certification: 680590, protocol version: 3 mysql_1 | 2017-06-02 12:19:33 140260172363520 [Note] WSREP: Service thread queue flushed. mysql_1 | 2017-06-02 12:19:33 140261419019008 [Note] WSREP: IST receiver addr using tcp://192.168.60.123:4568 mysql_1 | 2017-06-02 12:19:33 140261419019008 [Warning] WSREP: Failed to prepare for incremental state transfer: Failed to open IST listener at tcp://192.168.60.123:4568', asio error 'bind: Cannot assign requested address': 99 (Cannot assign requested address) mysql_1 | at galera/src/ist.cpp:prepare():325. IST will be unavailable. mysql_1 | 2017-06-02 12:19:33 140260113639168 [Note] WSREP: Member 0.0 (nmb_backup) requested state transfer from '*any*'. Selected 1.0 (nmb_main)(SYNCED) as donor. mysql_1 | 2017-06-02 12:19:33 140260113639168 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 680590) mysql_1 | 2017-06-02 12:19:33 140261419019008 [Note] WSREP: Requesting state transfer: success, donor: 1 mysql_1 | 2017-06-02 12:19:33 140261419019008 [Note] WSREP: GCache history reset: old(625893a6-4554-11e7-b41d-dfe527fde688:680188) -> new(625893a6-4554-11e7-b41d-dfe527fde688:680590) mysql_1 | 2017-06-02 12:19:33 140261419019008 [Note] WSREP: GCache DEBUG: RingBuffer::seqno_reset(): full reset mysql_1 | 2017-06-02 12:19:33 140260113639168 [Warning] WSREP: 1.0 (nmb_main): State transfer to 0.0 (nmb_backup) failed: -125 (Operation canceled) mysql_1 | 2017-06-02 12:19:33 140260113639168 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():736: Will never receive state. Need to abort. mysql_1 | 2017-06-02 12:19:33 140260113639168 [Note] WSREP: gcomm: terminating thread mysql_1 | 2017-06-02 12:19:33 140260113639168 [Note] WSREP: gcomm: joining thread mysql_1 | 2017-06-02 12:19:33 140260113639168 [Note] WSREP: gcomm: closing backend mysql_1 | 2017-06-02 12:19:36 140260113639168 [Note] WSREP: (15b3ea58, 'tcp://0.0.0.0:4567') turning message relay requesting off mysql_1 | 2017-06-02 12:19:36 140260113639168 [Note] WSREP: (15b3ea58, 'tcp://0.0.0.0:4567') connection to peer ce3e5068 with addr tcp://192.168.114.39:4567 timed out, no messages seen in PT3S mysql_1 | 2017-06-02 12:19:36 140260113639168 [Note] WSREP: (15b3ea58, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.114.39:4567 mysql_1 | 2017-06-02 12:19:38 140260113639168 [Note] WSREP: (15b3ea58, 'tcp://0.0.0.0:4567') reconnecting to ce3e5068 (tcp://192.168.114.39:4567), attempt 0 mysql_1 | 2017-06-02 12:19:38 140260113639168 [Note] WSREP: evs::proto(15b3ea58, LEAVING, view_id(REG,15b3ea58,16)) suspecting node: ce3e5068 mysql_1 | 2017-06-02 12:19:38 140260113639168 [Note] WSREP: evs::proto(15b3ea58, LEAVING, view_id(REG,15b3ea58,16)) suspected node without join message, declaring inactive mysql_1 | 2017-06-02 12:19:38 140260113639168 [Note] WSREP: view(view_id(NON_PRIM,15b3ea58,16) memb { mysql_1 | 15b3ea58,0 mysql_1 | } joined { mysql_1 | } left { mysql_1 | } partitioned { mysql_1 | ce3e5068,0 mysql_1 | }) mysql_1 | 2017-06-02 12:19:38 140260113639168 [Note] WSREP: view((empty)) mysql_1 | 2017-06-02 12:19:38 140260113639168 [Note] WSREP: gcomm: closed mysql_1 | 2017-06-02 12:19:38 140260113639168 [Note] WSREP: mysqld: Terminated. mysql_1 | 170602 12:19:38 [ERROR] mysqld got signal 11 ; mysql_1 | This could be because you hit a bug. It is also possible that this binary mysql_1 | or one of the libraries it was linked against is corrupt, improperly built, mysql_1 | or misconfigured. This error can also be caused by malfunctioning hardware. mysql_1 | mysql_1 | To report this bug, see https://mariadb.com/kb/en/reporting-bugs mysql_1 | mysql_1 | We will try our best to scrape up some info that will hopefully help mysql_1 | diagnose the problem, but since we have already crashed, mysql_1 | something is definitely wrong and this may fail. mysql_1 | mysql_1 | Server version: 10.1.21-MariaDB-1~jessie mysql_1 | key_buffer_size=8589934592 mysql_1 | read_buffer_size=2097152 mysql_1 | max_used_connections=0 mysql_1 | max_threads=253 mysql_1 | thread_count=9 mysql_1 | It is possible that mysqld could use up to mysql_1 | key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 9948215 K bytes of memory mysql_1 | Hope that's ok; if not, decrease some variables in the equation. mysql_1 | mysql_1 | Thread pointer: 0x0 mysql_1 | Attempting backtrace. You can use the following information to find out mysql_1 | where mysqld died. If you see no messages after this, something went mysql_1 | terribly wrong... mysql_1 | 2017-06-02 12:19:38 140261419019008 [Warning] WSREP: 0x7f9127841c48 down context(s) not set mysql_1 | 2017-06-02 12:19:38 140261419019008 [Warning] WSREP: Sending JOIN failed: -107 (Transport endpoint is not connected). Will retry in new primary component. mysql_1 | stack_bottom = 0x0 thread_stack 0x48400 mysql_1 | mysqld(my_print_stacktrace+0x2e)[0x7f912bd28ade] mysql_1 | mysqld(handle_fatal_signal+0x2fd)[0x7f912b86412d] mysql_1 | /lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7f912ae8d890] mysql_1 | /lib/x86_64-linux-gnu/libc.so.6(abort+0x232)[0x7f9128f7e532] mysql_1 | /usr/lib/galera/libgalera_smm.so(+0x77527)[0x7f90e63d0527] mysql_1 | /usr/lib/galera/libgalera_smm.so(_Z13gcs_core_recvP8gcs_coreP12gcs_act_rcvdx+0x692)[0x7f90e650b6b2] mysql_1 | /usr/lib/galera/libgalera_smm.so(+0x1b939e)[0x7f90e651239e] mysql_1 | /lib/x86_64-linux-gnu/libpthread.so.0(+0x8064)[0x7f912ae86064] mysql_1 | /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f912903062d] mysql_1 | The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains mysql_1 | information that should help you find out what is causing the crash.
Donor log:
2017-06-01 19:41:30 140421086836480 [Note] WSREP: (ce3e5068, 'tcp://0.0.0.0:4567') connection established to 15b3ea58 tcp://172.20.0.1:4567
mysql_1 | 2017-06-01 19:41:30 140421086836480 [Note] WSREP: (ce3e5068, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
mysql_1 | 2017-06-01 19:41:31 140421086836480 [Note] WSREP: declaring 15b3ea58 at tcp://172.20.0.1:4567 stable
mysql_1 | 2017-06-01 19:41:31 140421086836480 [Note] WSREP: Node ce3e5068 state prim
mysql_1 | 2017-06-01 19:41:31 140421086836480 [Note] WSREP: view(view_id(PRIM,15b3ea58,14) memb {
mysql_1 | 15b3ea58,0
mysql_1 | ce3e5068,0
mysql_1 | } joined {
mysql_1 | } left {
mysql_1 | } partitioned {
mysql_1 | })
mysql_1 | 2017-06-01 19:41:31 140421086836480 [Note] WSREP: save pc into disk
mysql_1 | 2017-06-01 19:41:31 140421078443776 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
mysql_1 | 2017-06-01 19:41:31 140421078443776 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
mysql_1 | 2017-06-01 19:41:31 140421078443776 [Note] WSREP: STATE EXCHANGE: sent state msg: 16019753-46e9-11e7-ba5e-f721bd4bc828
mysql_1 | 2017-06-01 19:41:31 140421078443776 [Note] WSREP: STATE EXCHANGE: got state msg: 16019753-46e9-11e7-ba5e-f721bd4bc828 from 0 (nmb_backup)
mysql_1 | 2017-06-01 19:41:31 140421078443776 [Note] WSREP: STATE EXCHANGE: got state msg: 16019753-46e9-11e7-ba5e-f721bd4bc828 from 1 (nmb_main)
mysql_1 | 2017-06-01 19:41:31 140421078443776 [Note] WSREP: Quorum results:
mysql_1 | version = 4,
mysql_1 | component = PRIMARY,
mysql_1 | conf_id = 13,
mysql_1 | members = 1/2 (joined/total),
mysql_1 | act_id = 528895,
mysql_1 | last_appl. = 528812,
mysql_1 | protocols = 0/7/3 (gcs/repl/appl),
mysql_1 | group UUID = 625893a6-4554-11e7-b41d-dfe527fde688
mysql_1 | 2017-06-01 19:41:31 140421078443776 [Note] WSREP: Flow-control interval: [23, 23]
mysql_1 | 2017-06-01 19:41:31 140422385650432 [Note] WSREP: New cluster view: global state: 625893a6-4554-11e7-b41d-dfe527fde688:528895, view# 14: Primary, number of nodes: 2, my index: 1, protocol version 3
mysql_1 | 2017-06-01 19:41:31 140422385650432 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
mysql_1 | 2017-06-01 19:41:31 140422385650432 [Note] WSREP: REPL Protocols: 7 (3, 2)
mysql_1 | 2017-06-01 19:41:31 140422385650432 [Note] WSREP: Assign initial position for certification: 528895, protocol version: 3
mysql_1 | 2017-06-01 19:41:31 140421137168128 [Note] WSREP: Service thread queue flushed.
mysql_1 | 2017-06-01 19:41:32 140421078443776 [Note] WSREP: Member 0.0 (nmb_backup) requested state transfer from '*any*'. Selected 1.0 (nmb_main)(SYNCED) as donor.
mysql_1 | 2017-06-01 19:41:32 140421078443776 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 528895)
mysql_1 | 2017-06-01 19:41:32 140422385650432 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
mysql_1 | 2017-06-01 19:41:32 140401654617856 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.60.123:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' '' --gtid '625893a6-4554-11e7-b41d-dfe527fde688:528895' --gtid-domain-id '0''
mysql_1 | 2017-06-01 19:41:32 140422385650432 [Note] WSREP: sst_donor_thread signaled with 0
mysql_1 | WSREP_SST: [INFO] Logging all stderr of SST/Innobackupex to syslog (20170601 19:41:32.454)
mysql_1 | 2017-06-01 19:41:34 140421086836480 [Note] WSREP: (ce3e5068, 'tcp://0.0.0.0:4567') turning message relay requesting off
mysql_1 | 2017-06-01 21:45:22 140421291137792 [Note] WSREP: Provider paused at 625893a6-4554-11e7-b41d-dfe527fde688:539477 (164624)
mysql_1 | 2017-06-01 22:02:48 140421291137792 [Note] WSREP: resuming provider at 164624
mysql_1 | 2017-06-01 22:02:48 140421291137792 [Note] WSREP: Provider resumed.
mysql_1 | 2017-06-01 22:02:48 140421078443776 [Note] WSREP: 1.0 (nmb_main): State transfer to 0.0 (nmb_backup) complete.
mysql_1 | 2017-06-01 22:02:48 140421078443776 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 539497)
mysql_1 | 2017-06-01 22:02:48 140421078443776 [Note] WSREP: Member 1.0 (nmb_main) synced with group.
mysql_1 | 2017-06-01 22:02:48 140421078443776 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 539498)
mysql_1 | 2017-06-01 22:02:48 140422385347328 [Note] WSREP: Synchronized with group, ready for connections
mysql_1 | 2017-06-01 22:02:48 140422385347328 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
mysql_1 | 2017-06-01 22:03:40 140421078443776 [Note] WSREP: 0.0 (nmb_backup): State transfer from 1.0 (nmb_main) complete.
mysql_1 | 2017-06-01 22:03:41 140421078443776 [Note] WSREP: Member 0.0 (nmb_backup) synced with group.
mysql_1 | 2017-06-01 23:04:12 140402029284096 [Warning] Aborted connection 19167 to db: 'unconnected' user: 'root' host: 'localhost' (Got timeout reading communication packets)
mysql_1 | 2017-06-01 23:20:24 140402028677888 [Warning] Aborted connection 19169 to db: 'unconnected' user: 'root' host: 'localhost' (Got timeout reading communication packets)
mysql_1 | 2017-06-02 10:33:38 140421069744896 [Warning] Aborted connection 73934 to db: 'nmb_base' user: 'root' host: '172.20.0.4' (Got an error writing communication packets)
mysql_1 | 2017-06-02 10:33:51 140421142137600 [Warning] Aborted connection 73933 to db: 'nmb_base' user: 'root' host: '172.20.0.4' (Got an error writing communication packets)
mysql_1 | 2017-06-02 10:37:40 140402037467904 [Warning] Aborted connection 74229 to db: 'nmb_base' user: 'root' host: '172.20.0.4' (Got an error writing communication packets)
mysql_1 | 2017-06-02 10:42:55 140402449107712 [Warning] Aborted connection 74231 to db: 'nmb_base' user: 'root' host: '172.20.0.4' (Got an error writing communication packets)
mysql_1 | 2017-06-02 10:48:01 140402035649280 [Warning] Aborted connection 74260 to db: 'nmb_base' user: 'root' host: '172.20.0.4' (Got an error writing communication packets)
mysql_1 | 2017-06-02 12:19:57 140421086836480 [Note] WSREP: (ce3e5068, 'tcp://0.0.0.0:4567') connection to peer 15b3ea58 with addr tcp://172.20.0.1:4567 timed out, no messages seen in PT3S
mysql_1 | 2017-06-02 12:19:57 140421086836480 [Note] WSREP: (ce3e5068, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.20.0.1:4567
mysql_1 | 2017-06-02 12:19:58 140421086836480 [Note] WSREP: (ce3e5068, 'tcp://0.0.0.0:4567') reconnecting to 15b3ea58 (tcp://172.20.0.1:4567), attempt 0
mysql_1 | 2017-06-02 12:19:58 140421086836480 [Note] WSREP: (ce3e5068, 'tcp://0.0.0.0:4567') connection established to ce3e5068 tcp://172.20.0.1:4567
mysql_1 | 2017-06-02 12:19:59 140421086836480 [Note] WSREP: evs::proto(ce3e5068, OPERATIONAL, view_id(REG,15b3ea58,14)) suspecting node: 15b3ea58
mysql_1 | 2017-06-02 12:19:59 140421086836480 [Note] WSREP: evs::proto(ce3e5068, OPERATIONAL, view_id(REG,15b3ea58,14)) suspected node without join message, declaring inactive
mysql_1 | 2017-06-02 12:20:00 140421086836480 [Note] WSREP: Node ce3e5068 state prim
mysql_1 | 2017-06-02 12:20:00 140421086836480 [Note] WSREP: view(view_id(PRIM,ce3e5068,15) memb {
mysql_1 | ce3e5068,0
mysql_1 | } joined {
mysql_1 | } left {
mysql_1 | } partitioned {
mysql_1 | 15b3ea58,0
mysql_1 | })
mysql_1 | 2017-06-02 12:20:00 140421086836480 [Note] WSREP: save pc into disk
mysql_1 | 2017-06-02 12:20:00 140421086836480 [Note] WSREP: forgetting 15b3ea58 (tcp://172.20.0.1:4567)
mysql_1 | 2017-06-02 12:20:00 140421086836480 [Note] WSREP: deleting entry tcp://172.20.0.1:4567
mysql_1 | 2017-06-02 12:20:00 140421086836480 [Note] WSREP: (ce3e5068, 'tcp://0.0.0.0:4567') turning message relay requesting off
mysql_1 | 2017-06-02 12:20:00 140421078443776 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
mysql_1 | 2017-06-02 12:20:00 140421078443776 [Note] WSREP: STATE_EXCHANGE: sent state UUID: a701f7e0-4774-11e7-b7cf-76bbe711775a
mysql_1 | 2017-06-02 12:20:00 140421078443776 [Note] WSREP: STATE EXCHANGE: sent state msg: a701f7e0-4774-11e7-b7cf-76bbe711775a
mysql_1 | 2017-06-02 12:20:00 140421078443776 [Note] WSREP: STATE EXCHANGE: got state msg: a701f7e0-4774-11e7-b7cf-76bbe711775a from 0 (nmb_main)
mysql_1 | 2017-06-02 12:20:00 140421078443776 [Note] WSREP: Quorum results:
mysql_1 | version = 4,
mysql_1 | component = PRIMARY,
mysql_1 | conf_id = 14,
mysql_1 | members = 1/1 (joined/total),
mysql_1 | act_id = 680205,
mysql_1 | last_appl. = 680177,
mysql_1 | protocols = 0/7/3 (gcs/repl/appl),
mysql_1 | group UUID = 625893a6-4554-11e7-b41d-dfe527fde688
mysql_1 | 2017-06-02 12:20:00 140421078443776 [Note] WSREP: Flow-control interval: [16, 16]
mysql_1 | 2017-06-02 12:20:00 140422437219072 [Note] WSREP: New cluster view: global state: 625893a6-4554-11e7-b41d-dfe527fde688:680205, view# 15: Primary, number of nodes: 1, my index: 0, protocol version 3
mysql_1 | 2017-06-02 12:20:00 140422437219072 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
mysql_1 | 2017-06-02 12:20:00 140422437219072 [Note] WSREP: REPL Protocols: 7 (3, 2)
mysql_1 | 2017-06-02 12:20:00 140422437219072 [Note] WSREP: Assign initial position for certification: 680205, protocol version: 3
mysql_1 | 2017-06-02 12:20:00 140421137168128 [Note] WSREP: Service thread queue flushed.
mysql_1 | 2017-06-02 12:20:02 140421086836480 [Note] WSREP: cleaning up 15b3ea58 (tcp://172.20.0.1:4567)
mysql_1 | 2017-06-02 12:20:07 140421086836480 [Note] WSREP: (ce3e5068, 'tcp://0.0.0.0:4567') connection established to 15b3ea58 tcp://172.20.0.1:4567
mysql_1 | 2017-06-02 12:20:07 140421086836480 [Note] WSREP: (ce3e5068, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
mysql_1 | 2017-06-02 12:20:08 140421086836480 [Note] WSREP: declaring 15b3ea58 at tcp://172.20.0.1:4567 stable
mysql_1 | 2017-06-02 12:20:08 140421086836480 [Note] WSREP: Node ce3e5068 state prim
mysql_1 | 2017-06-02 12:20:08 140421086836480 [Note] WSREP: view(view_id(PRIM,15b3ea58,16) memb {
mysql_1 | 15b3ea58,0
mysql_1 | ce3e5068,0
mysql_1 | } joined {
mysql_1 | } left {
mysql_1 | } partitioned {
mysql_1 | })
mysql_1 | 2017-06-02 12:20:08 140421086836480 [Note] WSREP: save pc into disk
mysql_1 | 2017-06-02 12:20:08 140421078443776 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
mysql_1 | 2017-06-02 12:20:08 140421078443776 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
mysql_1 | 2017-06-02 12:20:08 140421078443776 [Note] WSREP: STATE EXCHANGE: sent state msg: 96f8c457-4774-11e7-bb3f-2f97b0c8026b
mysql_1 | 2017-06-02 12:20:08 140421078443776 [Note] WSREP: STATE EXCHANGE: got state msg: 96f8c457-4774-11e7-bb3f-2f97b0c8026b from 0 (nmb_backup)
mysql_1 | 2017-06-02 12:20:08 140421078443776 [Note] WSREP: STATE EXCHANGE: got state msg: 96f8c457-4774-11e7-bb3f-2f97b0c8026b from 1 (nmb_main)
mysql_1 | 2017-06-02 12:20:08 140421078443776 [Note] WSREP: Quorum results:
mysql_1 | version = 4,
mysql_1 | component = PRIMARY,
mysql_1 | conf_id = 15,
mysql_1 | members = 1/2 (joined/total),
mysql_1 | act_id = 680590,
mysql_1 | last_appl. = 680572,
mysql_1 | protocols = 0/7/3 (gcs/repl/appl),
mysql_1 | group UUID = 625893a6-4554-11e7-b41d-dfe527fde688
mysql_1 | 2017-06-02 12:20:08 140421078443776 [Note] WSREP: Flow-control interval: [23, 23]
mysql_1 | 2017-06-02 12:20:08 140422385347328 [Note] WSREP: New cluster view: global state: 625893a6-4554-11e7-b41d-dfe527fde688:680590, view# 16: Primary, number of nodes: 2, my index: 1, protocol version 3
mysql_1 | 2017-06-02 12:20:08 140422385347328 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
mysql_1 | 2017-06-02 12:20:08 140422385347328 [Note] WSREP: REPL Protocols: 7 (3, 2)
mysql_1 | 2017-06-02 12:20:08 140422385347328 [Note] WSREP: Assign initial position for certification: 680590, protocol version: 3
mysql_1 | 2017-06-02 12:20:08 140421137168128 [Note] WSREP: Service thread queue flushed.
mysql_1 | 2017-06-02 12:20:08 140421078443776 [Note] WSREP: Member 0.0 (nmb_backup) requested state transfer from '*any*'. Selected 1.0 (nmb_main)(SYNCED) as donor.
mysql_1 | 2017-06-02 12:20:08 140421078443776 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 680590)
mysql_1 | 2017-06-02 12:20:08 140422385347328 [Warning] WSREP: SST request is null, SST canceled.
mysql_1 | 2017-06-02 12:20:08 140421078443776 [Warning] WSREP: 1.0 (nmb_main): State transfer to 0.0 (nmb_backup) failed: -125 (Operation canceled)
mysql_1 | 2017-06-02 12:20:08 140421078443776 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 680590)
mysql_1 | 2017-06-02 12:20:08 140421078443776 [Note] WSREP: Member 1.0 (nmb_main) synced with group.
mysql_1 | 2017-06-02 12:20:08 140421078443776 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 680590)
mysql_1 | 2017-06-02 12:20:08 140422385650432 [Note] WSREP: Synchronized with group, ready for connections
mysql_1 | 2017-06-02 12:20:08 140422385650432 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
mysql_1 | 2017-06-02 12:20:09 140421086836480 [Note] WSREP: forgetting 15b3ea58 (tcp://172.20.0.1:4567)
mysql_1 | 2017-06-02 12:20:09 140421086836480 [Note] WSREP: Node ce3e5068 state prim
mysql_1 | 2017-06-02 12:20:09 140421086836480 [Note] WSREP: view(view_id(PRIM,ce3e5068,17) memb {
mysql_1 | ce3e5068,0
mysql_1 | } joined {
mysql_1 | } left {
mysql_1 | } partitioned {
mysql_1 | 15b3ea58,0
mysql_1 | })
mysql_1 | 2017-06-02 12:20:09 140421086836480 [Note] WSREP: save pc into disk
mysql_1 | 2017-06-02 12:20:09 140421086836480 [Note] WSREP: forgetting 15b3ea58 (tcp://172.20.0.1:4567)
mysql_1 | 2017-06-02 12:20:09 140421078443776 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
mysql_1 | 2017-06-02 12:20:09 140421078443776 [Note] WSREP: STATE_EXCHANGE: sent state UUID: ac951928-4774-11e7-bd9f-56a82fb24364
mysql_1 | 2017-06-02 12:20:09 140421078443776 [Note] WSREP: STATE EXCHANGE: sent state msg: ac951928-4774-11e7-bd9f-56a82fb24364
mysql_1 | 2017-06-02 12:20:09 140421078443776 [Note] WSREP: STATE EXCHANGE: got state msg: ac951928-4774-11e7-bd9f-56a82fb24364 from 0 (nmb_main)
mysql_1 | 2017-06-02 12:20:09 140421078443776 [Note] WSREP: Quorum results:
mysql_1 | version = 4,
mysql_1 | component = PRIMARY,
mysql_1 | conf_id = 16,
mysql_1 | members = 1/1 (joined/total),
mysql_1 | act_id = 680590,
mysql_1 | last_appl. = 680572,
mysql_1 | protocols = 0/7/3 (gcs/repl/appl),
mysql_1 | group UUID = 625893a6-4554-11e7-b41d-dfe527fde688
mysql_1 | 2017-06-02 12:20:09 140421078443776 [Note] WSREP: Flow-control interval: [16, 16]
mysql_1 | 2017-06-02 12:20:09 140422437219072 [Note] WSREP: New cluster view: global state: 625893a6-4554-11e7-b41d-dfe527fde688:680590, view# 17: Primary, number of nodes: 1, my index: 0, protocol version 3
mysql_1 | 2017-06-02 12:20:09 140422437219072 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
mysql_1 | 2017-06-02 12:20:09 140422437219072 [Note] WSREP: REPL Protocols: 7 (3, 2)
mysql_1 | 2017-06-02 12:20:09 140422437219072 [Note] WSREP: Assign initial position for certification: 680590, protocol version: 3
mysql_1 | 2017-06-02 12:20:09 140421137168128 [Note] WSREP: Service thread queue flushed.
mysql_1 | 2017-06-02 12:20:11 140421086836480 [Note] WSREP: (ce3e5068, 'tcp://0.0.0.0:4567') turning message relay requesting off
mysql_1 | 2017-06-02 12:20:13 140421086836480 [Note] WSREP: (ce3e5068, 'tcp://0.0.0.0:4567') connection established to 15b3ea58 tcp://172.20.0.1:4567
mysql_1 | 2017-06-02 12:20:13 140421086836480 [Warning] WSREP: discarding established (time wait) 15b3ea58 (tcp://172.20.0.1:4567)
mysql_1 | 2017-06-02 12:20:14 140421086836480 [Note] WSREP: cleaning up 15b3ea58 (tcp://172.20.0.1:4567)
Joiner my.cnf # MariaDB database server configuration file. # # You can copy this file to one of: # - "/etc/mysql/my.cnf" to set global options, # - "~/.my.cnf" to set user-specific options. # # One can use all long options that the program supports. # Run program with --help to get a list of available options and with # --print-defaults to see which it would actually understand and use. # # For explanations see # http://dev.mysql.com/doc/mysql/en/server-system-variables.html # This will be passed to all mysql clients # It has been reported that passwords should be enclosed with ticks/quotes # escpecially if they contain "#" chars... # Remember to edit /etc/mysql/debian.cnf when changing the socket location. [client] port = 3306 socket = /var/run/mysqld/mysqld.sock # Here is entries for some specific programs # The following values assume you have at least 32M ram # This was formally known as [safe_mysqld]. Both versions are currently parsed. [mysqld_safe] socket = /var/run/mysqld/mysqld.sock nice = 0 [mysqld] # Settings for replication server_id=2 bind-address=0.0.0.0 skip-host-cache skip-name-resolve # # * Basic Settings # #user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock port = 3306 basedir = /usr datadir = /var/lib/mysql tmpdir = /tmp lc_messages_dir = /usr/share/mysql lc_messages = en_US skip-external-locking character-set-server=utf8 default-storage-engine=INNODB sql-mode="NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION" # # Instead of skip-networking the default is now to listen only on # localhost which is more compatible and is not less secure. #bind-address = 127.0.0.1 # # * Fine Tuning # max_connections = 251 connect_timeout = 10000 wait_timeout = 6000 max_allowed_packet = 16M thread_cache_size = 128 sort_buffer_size = 4M bulk_insert_buffer_size = 16M tmp_table_size = 256M max_heap_table_size = 32M # # * MyISAM # # This replaces the startup script and checks MyISAM tables if needed # the first time they are touched. On error, make copy and try a repair. myisam_max_sort_file_size=100G myisam_recover_options = BACKUP key_buffer_size = 8G #open-files-limit = 2000 table_open_cache = 2000 myisam_sort_buffer_size = 222M concurrent_insert = 2 read_buffer_size = 2M read_rnd_buffer_size = 1M # # * Query Cache Configuration # # Cache only tiny result sets, so we can fit more in the query cache. query_cache_limit = 100M query_cache_size = 1G # for more write intensive setups, set to DEMAND or OFF #query_cache_type = DEMAND # # * Logging and Replication # # Both location gets rotated by the cronjob. # Be aware that this log type is a performance killer. # As of 5.1 you can enable the log at runtime! #general_log_file = /var/log/mysql/mysql.log #general_log = 1 # # Error logging goes to syslog due to /etc/mysql/conf.d/mysqld_safe_syslog.cnf. # # we do want to know about network errors and such log_warnings = 2 # # Enable the slow query log to see queries with especially long duration slow_query_log=1 slow_query_log_file = /var/lib/mysql/mariadb-slow.log long_query_time = 4 #log_slow_rate_limit = 1000 #log_slow_verbosity = query_plan relay-log-space-limit=6G #log-queries-not-using-indexes #log_slow_admin_statements # # The following can be used as easy to replay backup logs or for replication. # note: if you are setting up a replication slave, see README.Debian about # other settings you may need to change. #server-id = 1 #report_host = master1 #auto_increment_increment = 2 #auto_increment_offset = 1 #log_bin = /var/log/mysql/mariadb-bin #log_bin_index = /var/log/mysql/mariadb-bin.index # not fab for performance, but safer #sync_binlog = 1 expire_logs_days = 10 max_binlog_size = 100M # slaves #relay_log = /var/log/mysql/relay-bin #relay_log_index = /var/log/mysql/relay-bin.index #relay_log_info_file = /var/log/mysql/relay-bin.info #log_slave_updates #read_only # # If applications support it, this stricter sql_mode prevents some # mistakes like inserting invalid dates etc. #sql_mode = NO_ENGINE_SUBSTITUTION,TRADITIONAL # # * InnoDB # # InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/. # Read the manual for more InnoDB related options. There are many! default_storage_engine = InnoDB # you can't just change log file size, requires special procedure innodb_log_file_size = 2G innodb_buffer_pool_size = 7G innodb_log_buffer_size = 8M innodb_file_per_table = 1 innodb_open_files = 400 innodb_io_capacity = 400 innodb_flush_method = O_DIRECT innodb_additional_mem_pool_size=2G innodb_flush_log_at_trx_commit=1 innodb_thread_concurrency=0 innodb_read_io_threads=64 innodb_write_io_threads=64 innodb_autoextend_increment=64 innodb_buffer_pool_instances=8 innodb_concurrency_tickets=5000 innodb_old_blocks_time=1000 innodb_stats_on_metadata=0 innodb_checksum_algorithm=0 innodb_locks_unsafe_for_binlog = 1 innodb_autoinc_lock_mode = 2 back_log=80 flush_time=0 max_connect_errors=100 open_files_limit=4161 table_definition_cache=1400 sync_master_info=1000 sync_relay_log=10000 sync_relay_log_info=10000 # # * Security Features # # Read the manual, too, if you want chroot! # chroot = /var/lib/mysql/ # # For generating SSL certificates I recommend the OpenSSL GUI "tinyca". # # ssl-ca=/etc/mysql/cacert.pem # ssl-cert=/etc/mysql/server-cert.pem # ssl-key=/etc/mysql/server-key.pem # # * Galera-related settings # [galera] # Mandatory settings wsrep_on=ON wsrep_provider=/usr/lib/galera/libgalera_smm.so wsrep_provider_options="gcache.size=2G" wsrep_provider_options="gcache.recover=yes" #the fc_limit setting determines how many transactions a node should fall behind #before flow control is activated (which blocks whole cluster until node catches up #default is 16 but for master-slave setup this can be increased a lot #100000 value to prevent a lockdown after a full SST (measured around 60000 max) wsrep_provider_options="gcs.fc_limit=100000" wsrep_cluster_name=mariadb-cluster-nmb wsrep_cluster_address=gcomm://192.168.114.39,192.168.60.123 wsrep_node_address="192.168.60.123" wsrep_node_name="nmb_backup" wsrep_sst_method=xtrabackup-v2 wsrep_sst_auth=root:<correct password> binlog_format=row #important setting to prevent cluster not reaching quorum when 1 node fails #by setting DR to 0 (this config) live will retain quorum if DR fails and DR will continue as non-primary if Live fails wsrep_provider_options="pc.weight=0" #wsrep_provider_options="ist.recv_addr=192.168.114.39" #wsrep_sst_receive_address=192.168.114.39 # Optional setting wsrep_slave_threads=8 #innodb_flush_log_at_trx_commit=0 [mysqldump] quick quote-names max_allowed_packet = 64M [mysql] #no-auto-rehash # faster start of mysql but no tab completion [isamchk] key_buffer = 16M # # * IMPORTANT: Additional settings that can override those from this file! # The files must end with '.cnf', otherwise they'll be ignored. # !includedir /etc/mysql/conf.d/
Donor my.cnf
# MariaDB database server configuration file. # # You can copy this file to one of: # - "/etc/mysql/my.cnf" to set global options, # - "~/.my.cnf" to set user-specific options. # # One can use all long options that the program supports. # Run program with --help to get a list of available options and with # --print-defaults to see which it would actually understand and use. # # For explanations see # http://dev.mysql.com/doc/mysql/en/server-system-variables.html # This will be passed to all mysql clients # It has been reported that passwords should be enclosed with ticks/quotes # escpecially if they contain "#" chars... # Remember to edit /etc/mysql/debian.cnf when changing the socket location. [client] port = 3306 socket = /var/run/mysqld/mysqld.sock # Here is entries for some specific programs # The following values assume you have at least 32M ram # This was formally known as [safe_mysqld]. Both versions are currently parsed. [mysqld_safe] socket = /var/run/mysqld/mysqld.sock nice = 0 [mysqld] # Settings for replication server_id=1 bind-address=0.0.0.0 skip-host-cache skip-name-resolve # # * Basic Settings # #user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock port = 3306 basedir = /usr datadir = /var/lib/mysql tmpdir = /tmp lc_messages_dir = /usr/share/mysql lc_messages = en_US skip-external-locking character-set-server=utf8 default-storage-engine=INNODB sql-mode="NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION" # # Instead of skip-networking the default is now to listen only on # localhost which is more compatible and is not less secure. #bind-address = 127.0.0.1 # # * Fine Tuning # max_connections = 251 connect_timeout = 10000 wait_timeout = 6000 max_allowed_packet = 16M thread_cache_size = 128 sort_buffer_size = 4M bulk_insert_buffer_size = 16M tmp_table_size = 256M max_heap_table_size = 32M # # * MyISAM # # This replaces the startup script and checks MyISAM tables if needed # the first time they are touched. On error, make copy and try a repair. myisam_max_sort_file_size=100G myisam_recover_options = BACKUP key_buffer_size = 8G #open-files-limit = 2000 table_open_cache = 2000 myisam_sort_buffer_size = 222M concurrent_insert = 2 read_buffer_size = 2M read_rnd_buffer_size = 1M # # * Query Cache Configuration # # Cache only tiny result sets, so we can fit more in the query cache. query_cache_limit = 100M query_cache_size = 1G # for more write intensive setups, set to DEMAND or OFF #query_cache_type = DEMAND # # * Logging and Replication # # Both location gets rotated by the cronjob. # Be aware that this log type is a performance killer. # As of 5.1 you can enable the log at runtime! #general_log_file = /var/log/mysql/mysql.log #general_log = 1 # # Error logging goes to syslog due to /etc/mysql/conf.d/mysqld_safe_syslog.cnf. # # we do want to know about network errors and such log_warnings = 2 # # Enable the slow query log to see queries with especially long duration slow_query_log=1 slow_query_log_file = /var/lib/mysql/mariadb-slow.log long_query_time = 2 #log_slow_rate_limit = 1000 #log_slow_verbosity = query_plan relay-log-space-limit=6G #log-queries-not-using-indexes #log_slow_admin_statements # # The following can be used as easy to replay backup logs or for replication. # note: if you are setting up a replication slave, see README.Debian about # other settings you may need to change. #server-id = 1 #report_host = master1 #auto_increment_increment = 2 #auto_increment_offset = 1 #log_bin = /var/log/mysql/mariadb-bin #log_bin_index = /var/log/mysql/mariadb-bin.index # not fab for performance, but safer #sync_binlog = 1 expire_logs_days = 10 max_binlog_size = 100M # slaves #relay_log = /var/log/mysql/relay-bin #relay_log_index = /var/log/mysql/relay-bin.index #relay_log_info_file = /var/log/mysql/relay-bin.info #log_slave_updates #read_only # # If applications support it, this stricter sql_mode prevents some # mistakes like inserting invalid dates etc. #sql_mode = NO_ENGINE_SUBSTITUTION,TRADITIONAL # # * InnoDB # # InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/. # Read the manual for more InnoDB related options. There are many! default_storage_engine = InnoDB # you can't just change log file size, requires special procedure innodb_log_file_size = 2G innodb_buffer_pool_size = 7G innodb_log_buffer_size = 8M innodb_file_per_table = 1 innodb_open_files = 400 innodb_io_capacity = 400 innodb_flush_method = O_DIRECT innodb_additional_mem_pool_size=2G innodb_flush_log_at_trx_commit=1 innodb_thread_concurrency=0 innodb_read_io_threads=64 innodb_write_io_threads=64 innodb_autoextend_increment=64 innodb_buffer_pool_instances=8 innodb_concurrency_tickets=5000 innodb_old_blocks_time=1000 innodb_stats_on_metadata=0 innodb_checksum_algorithm=0 innodb_locks_unsafe_for_binlog = 1 innodb_autoinc_lock_mode = 2 back_log=80 flush_time=0 max_connect_errors=100 open_files_limit=4161 table_definition_cache=1400 sync_master_info=1000 sync_relay_log=10000 sync_relay_log_info=10000 # # * Security Features # # Read the manual, too, if you want chroot! # chroot = /var/lib/mysql/ # # For generating SSL certificates I recommend the OpenSSL GUI "tinyca". # # ssl-ca=/etc/mysql/cacert.pem # ssl-cert=/etc/mysql/server-cert.pem # ssl-key=/etc/mysql/server-key.pem # # * Galera-related settings # [galera] # Mandatory settings wsrep_on=ON wsrep_provider=/usr/lib/galera/libgalera_smm.so wsrep_provider_options="gcache.size=2G" wsrep_provider_options="gcache.recover=yes" #the fc_limit setting determines how many transactions a node should fall behind #before flow control is activated (which blocks whole cluster until node catches up #default is 16 but for master-slave setup this can be increased a lot #100000 value to prevent a lockdown after a full SST (measured around 60000 max) wsrep_provider_options="gcs.fc_limit=100000" wsrep_cluster_name=mariadb-cluster-nmb wsrep_cluster_address=gcomm:// wsrep_node_address="192.168.114.39" wsrep_node_name="nmb_main" wsrep_sst_method=xtrabackup-v2 binlog_format=row wsrep_sst_auth=root:<correct password> #important setting to prevent cluster to go down if 1 of the nodes fails #if DR fails live server retains quorum, if Live fails DR will continue as non-primary component wsrep_provider_options="pc.weight=2" # Optional setting wsrep_slave_threads=8 #innodb_flush_log_at_trx_commit=0 [mysqldump] quick quote-names max_allowed_packet = 64M [mysql] #no-auto-rehash # faster start of mysql but no tab completion [isamchk] key_buffer = 16M # # * IMPORTANT: Additional settings that can override those from this file! # The files must end with '.cnf', otherwise they'll be ignored. # !includedir /etc/mysql/conf.d/ The docker containers are started with their own network (not in network_mode = host). As stated the ports are configured in firewalld on both nodes. nmap and telnet tests succeed, although on ports 4568 the telnet session is immediately closed (but not refused) . Below is the output of the active zone from firewalld to be sure: public (active) target: default icmp-block-inversion: no interfaces: ens192 sources: services: dhcpv6-client mysql ssh ports: 4567/tcp 80/tcp 4568/tcp 4567/udp 5601/udp 8025/tcp 5601/tcp 8025/udp 4444/udp 4568/udp 4444/tcp protocols: masquerade: no forward-ports: port=4444:proto=tcp:toport=4444:toaddr= port=4567:proto=tcp:toport=4567:toaddr= port=4568:proto=tcp:toport=4568:toaddr= port=3306:proto=tcp:toport=3306:toaddr= port=5601:proto=tcp:toport=5601:toaddr= port=80:proto=tcp:toport=80:toaddr= port=8025:proto=tcp:toport=8025:toaddr= sourceports: icmp-blocks: rich rules:
The only weird thing is that we had to add forward ports to the same ports to make the ports accessible which should not be neccesary.
Comments
Comments loading...
Content reproduced on this site is the property of its respective owners,
and this content is not reviewed in advance by MariaDB. The views, information and opinions
expressed by this content do not necessarily represent those of MariaDB or any other party.
