Galera集群双节点Rsync故障求助：NFO与Azure VPS节点排查

Hey there, let's work through that Rsync error plaguing your Galera cluster—cross-cloud setups (NFO Servers VPS + Azure) can have some quirky network and permission gotchas, but we'll break this down step by step to get things working.

1. First, Validate Basic Rsync/SSH Connectivity Between Nodes #

Galera uses Rsync for State Snapshot Transfer (SST), so let's rule out fundamental connectivity issues before digging into Galera-specific configs:

• On your secondary node (Azure), run a manual Rsync test to pull a trivial file from the primary (NFO VPS):

  rsync -avz root@<PRIMARY_NODE_IP>:/etc/passwd /tmp/test_passwd
  

  This mimics the core Rsync operation Galera uses. If this fails:
  • Ensure passwordless SSH is set up between nodes: Galera's SST runs non-interactively, so you can't have password prompts. Check that the secondary node's root user has an SSH key added to the primary's ~/.ssh/authorized_keys.
  • Verify SSH access directly: Run ssh root@<PRIMARY_NODE_IP> from the secondary—if this requires a password or fails, fix that first.
  • Check SSH daemon config on the primary: Ensure PermitRootLogin yes is set in /etc/ssh/sshd_config (or adjust to allow the specific user Galera uses for SST) and restart sshd if needed.

2. Verify Galera SST Configuration #

Double-check the [galera] section in /etc/my.cnf.d/server.cnf on both nodes:

• Confirm the SST method is set to rsync:

  wsrep_sst_method = rsync
  

• Check the SST auth setting: If you're using password auth, it should look like wsrep_sst_auth = root:<YOUR_PASSWORD>; if using SSH keys, you can leave the password blank, but ensure the key setup is solid.
• Ensure wsrep_cluster_address on both nodes correctly lists the other node's IP (or resolvable hostname)—ping each node from the other to confirm DNS/IP resolution works.

3. Check SELinux Permissions (CentOS 7 Common Culprit) #

Even with firewalls disabled, SELinux can block Rsync operations on CentOS 7:

• Temporarily disable SELinux to test:

  setenforce 0
  

  Then restart your Galera service (systemctl restart mariadb or systemctl restart mysql, depending on your setup) and see if the Rsync error clears up.
• If disabling SELinux fixes it, create a permanent SELinux rule to allow Galera's Rsync activity:

  grep rsync /var/log/audit/audit.log | audit2allow -a -M galera_rsync
  semodule -i galera_rsync.pp
  

• Also verify permissions on /var/lib/mysql: It should be owned by mysql:mysql (run ls -ld /var/lib/mysql to check)—Rsync needs read/write access here.

4. Cross-Cloud Network Checks (NFO vs Azure) #

Cloud platforms often have hidden network restrictions beyond local firewalls:

• On Azure, check your Network Security Group (NSG) for the Azure node: Ensure inbound rules allow traffic from your NFO VPS's IP on ports 22 (SSH), 3306 (MySQL), 4444 (Galera SST), 4567 (Galera replication), and 4568 (IST).
• On your NFO VPS, confirm there are no outbound firewall rules blocking access to Azure's IP on those same ports.
• Test port connectivity with nc:

  # Test SSH port
  nc -zv <AZURE_NODE_IP> 22
  # Test Galera SST port
  nc -zv <AZURE_NODE_IP> 4444
  

5. Dig Into Detailed Error Logs #

Galera's logs will give you the exact Rsync command that's failing—use that to pinpoint the issue:

• Check your Galera error log (usually /var/log/mariadb/mariadb.log or /var/log/mysqld.log) and search for rsync or SST. You'll likely see a line like:

  2024-05-20 14:30:00 139812345678900 [ERROR] WSREP: Failed to read from: rsync -avz --delete-after --ignore-errors root@10.0.0.5:/var/lib/mysql/ /var/lib/mysql/ --password=REDACTED

• Copy that exact Rsync command (remove the --password part if you're using keys) and run it manually—this will show you the precise error (e.g., permission denied on a specific file, connection timeout, etc.).

Start with the basic connectivity tests first—most cross-cloud Galera Rsync issues boil down to SSH key problems or hidden cloud network restrictions. Once you narrow down the exact error, fixing it should be straightforward.

内容的提问来源于stack exchange，提问作者Cody Ardoin