Percona xtradb cluster on CentOS 6

Another simple reminder on how to install PXC (Percona xtradb cluster) on CentOS 6.
Mostly a reminder for iptables, SELinux and socat [3] [4] [5] [6]
Documentation is well done. [0] [1] [2]

yum install compat-readline5.x86_64
wget "https://dl.fedoraproject.org/pub/epel/6/x86_64/socat-1.7.2.3-1.el6.x86_64.rpm"
rpm -ivh socat-1.7.2.3-1.el6.x86_64.rpm
yum install http://www.percona.com/downloads/percona-release/redhat/0.1-3/percona-release-0.1-3.noarch.rpm
yum install Percona-XtraDB-Cluster-56

Then, follow the Percona documentation until

/etc/init.d/mysql bootstrap-pxc

Bootstrapping PXC (Percona XtraDB Cluster)Starting MySQL (Percona XtraDB Cluster).. ERROR!
ERROR! MySQL (Percona XtraDB Cluster) server startup failed!

WELL, SELinux, dear friend.

Disable SELinux, IS NOT a good option.
So we need to build some rules.

yum provides \*/audit2allow
yum install policycoreutils-python

Find SELinux status

sestatus

SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   enforcing
Mode from config file:          enforcing
Policy version:                 24
Policy from config file:        targeted
setenforce 0
/etc/init.d/mysql bootstrap-pxc

Bootstrapping PXC (Percona XtraDB Cluster)Starting MySQL (Percona XtraDB Cluster). SUCCESS!

Then the same for node 2 and node 3.

Enable them and re-activate SELinux.

grep mysql audit.log | audit2allow -M percona && semodule -i percona.pp
setenforce 1

iptables

 iptables-save
# Generated by iptables-save v1.4.7 on Thu Jun 11 17:13:24 2015
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [251565:16788875]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 3306 -j ACCEPT
-A INPUT -s $IP/32 -p tcp -m state --state NEW -m tcp --dport 4444 -j ACCEPT
-A INPUT -s $IP/32 -p tcp -m state --state NEW -m tcp --dport 4567 -j ACCEPT
-A INPUT -s $IP/32 -p tcp -m state --state NEW -m tcp --dport 4568 -j ACCEPT
-A INPUT -s $IP/32 -p tcp -m state --state NEW -m tcp --dport 4444 -j ACCEPT
-A INPUT -s $IP/32 -p tcp -m state --state NEW -m tcp --dport 4567 -j ACCEPT
-A INPUT -s $IP/32 -p tcp -m state --state NEW -m tcp --dport 4568 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited


PCX Error not very explicit.

2015-06-12 12:32:04 90736 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (86f838ba-104a-11e5-b590-261a1626ebe2): 1 (Operation not permitted) at galera/src/replicator_str.cpp:prepare_for_IST():463. IST will be unavailable.

“This is absolutely normal when you start a new node. ” [7]

2015-06-12 12:32:06 90736 [Note] WSREP: (43ad6807, 'tcp://0.0.0.0:4567') turning message relay requesting off
xbstream: Can't create/write to file '././backup-my.cnf' (Errcode: 17 - File exists)
xbstream: failed to create file.

Just delete hidden /var/lib/mysql/.sst/ directory on node server.

If you have an infinite loop on same log sequence:

xtrabackup:   innodb_log_file_size = 134217728
>> log scanned up to (1628988)
xtrabackup: Generating a list of tablespaces
>> log scanned up to (1628988)
[...]
>> log scanned up to (1628988)
>> log scanned up to (1628988)
>> log scanned up to (1628988)
>> log scanned up to (1628988)
>> log scanned up to (1628988)
>> log scanned up to (1628988)
>> log scanned up to (1628988)
[...]
>> log scanned up to (1628988)
>> log scanned up to (1628988)
[...]
xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
xb_stream_write_data() failed.
xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[01] xtrabackup: Error: failed to copy datafile.
innobackupex: got a fatal error with the following stacktrace: at /usr//bin/innobackupex line 2711
    main::wait_for_ibbackup_file_create('/tmp/tmp.z1GxRXuQXN/xtrabackup_suspended_2') called at /usr//bin/innobackupex line 2731
    main::wait_for_ibbackup_suspend('/tmp/tmp.z1GxRXuQXN/xtrabackup_suspended_2') called at /usr//bin/innobackupex line 1984
    main::backup() called at /usr//bin/innobackupex line 1609
innobackupex: Error: The xtrabackup child process has died at /usr//bin/innobackupex line 2711.

You probably have a network issue, you are losing some packets.
For me, it was a MTU issue.

For this one:

xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
xb_stream_write_data() failed.
xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[01] xtrabackup: Error: failed to copy datafile.
innobackupex: got a fatal error with the following stacktrace: at /usr//bin/innobackupex line 2711
    main::wait_for_ibbackup_file_create('/tmp/tmp.EeCjWcHeNo/xtrabackup_suspended_2') called at /usr//bin/innobackupex line 2731
    main::wait_for_ibbackup_suspend('/tmp/tmp.EeCjWcHeNo/xtrabackup_suspended_2') called at /usr//bin/innobackupex line 1984
    main::backup() called at /usr//bin/innobackupex line 1609
innobackupex: Error: The xtrabackup child process has died at /usr//bin/innobackupex line 2711.

You should have a hidden sst directory on joiner ($datadir/.sst) .
You just have to erase him.
SST should do that, but it does not if the sync fails before completion. even when a new SST clears the datadir, it does not remove the hidden .sst directory then your next sync fails because the files there already exist.

[0] https://www.percona.com/doc/percona-xtradb-cluster/5.6/
[1] https://www.percona.com/doc/percona-xtradb-cluster/5.6/installation/yum_repo.html
[2] https://www.percona.com/doc/percona-xtradb-cluster/5.6/installation.html#installation
[3] https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1278357
[4] https://dl.fedoraproject.org/pub/epel/6/x86_64/
[5] http://rpm.pbone.net/index.php3/stat/4/idpl/25696517/dir/redhat_el_6/com/socat-1.7.2.3-1.el6.x86_64.rpm.html
[6] http://rpm.pbone.net/index.php3/stat/3/srodzaj/1/search/libreadline.so.5%28%29%2864bit%29
[7] https://groups.google.com/forum/#!topic/percona-discussion/9WE0o-PnnYo