Redis故障恢复期间读写负载分配失衡问题求助

Hey there, I’ve worked through similar HAProxy + Redis Sentinel setups before, so let’s walk through how to diagnose this issue where your read requests aren’t hitting replicas after a failover.

First, let’s recap your setup to make sure I’m on the same page: #

• .NET Core 3.1 web server using StackExchange.Redis.Extensions 5.5.0
• HAProxy 2.1.7 handling Redis read/write separation
• Redis 5.0.7 with Sentinel for failover
• Issue: When read/write separation is enabled, after a failover all traffic (read + write) goes to the new master. With separation disabled, failover works as expected. You’ve confirmed this via Zabbix’s Instantaneous_ops_per_sec metric.

Here are the key areas to investigate, step by step: #

1. Audit HAProxy’s Backend Configuration & Failover Sync

HAProxy is the gatekeeper for your traffic split, so start here:

• Check health checks for Redis nodes: HAProxy needs to correctly identify master vs replica nodes. Make sure your health check uses INFO replication to parse the node role (master/replica). If your check is too generic (like just PING), HAProxy won’t distinguish roles, and might route all traffic to the only "healthy" node after failover.
• Verify ACL rules for traffic splitting: Double-check your rules that route read commands (like GET, HGET) to replicas. For example, if you’re using tcp-request content accept if { req.payload(0,5) -m str "GET" }, ensure this rule still applies after failover. If the new replica (old master) isn’t being tagged correctly, HAProxy will send reads to the new master instead.
• Check HAProxy stats: If you’ve enabled the stats page, look at the backend status of your replica nodes post-failover. Are they marked as UP? If they’re in DOWN or MAINT state, HAProxy will fallback to sending all traffic to the master.
• Automation script validation: If you’re using a script to sync Sentinel’s master/replica info to HAProxy (common in these setups), run the script manually after failover. Check if it correctly updates HAProxy’s config to mark the new master as the write node and the old master as a read replica. Ensure the script has proper permissions to reload HAProxy after updates.

2. Validate Sentinel & Redis Replica Health Post-Failover

Sentinel’s failover might leave replicas in an inconsistent state that blocks HAProxy from using them:

• Check replica status: Log into your replica node and run INFO replication. Confirm master_link_status is up, slave_repl_offset is close to the master’s offset (no massive sync lag), and role is slave. If the replica is stuck in a sync loop or has a broken link to the new master, HAProxy will mark it as unhealthy.
• Review Sentinel logs: Look for errors in Sentinel’s logs during failover. Did it correctly promote a new master and reconfigure the old master as a replica? Are there any warnings about split-brain or failed replica reconfiguration?
• Test replica read capability: Manually run a few read commands (e.g., GET test-key) on the replica to confirm it can serve requests. If the replica is read-only but functional, the issue is likely with HAProxy routing, not the replica itself.

3. Check StackExchange.Redis.Extensions Behavior in Your App

Your .NET app might be holding onto stale connection info or misconfigured for read splitting:

• Connection multiplexer refresh: StackExchange.Redis uses a connection multiplexer that caches node info. After a failover, does your app refresh this multiplexer? If it’s holding onto the old master/replica mapping, it might send all requests to the new master directly, bypassing HAProxy’s split rules.
• Read command flags: Ensure your code uses CommandFlags.DemandSlave (or the equivalent in StackExchange.Redis.Extensions) for read requests. If all requests are using CommandFlags.None, the library might default to sending everything to the master, even if HAProxy is set up to split traffic.
• Add request logging: Temporarily add logs to your app that record the target Redis node IP for each request. This will confirm whether read requests are even being sent to HAProxy’s read backend, or if the app is routing them straight to the master.

4. Analyze HAProxy Logs for Routing Clues

Enable detailed HAProxy logging (if you haven’t already) to see exactly where requests are going:

• Look for entries that show the backend server used for each read request post-failover. Are they all pointing to the new master?
• Check for health check failures in the logs (e.g., Redis replication check failed for the replica node). This would explain why HAProxy isn’t sending traffic to it.
• Confirm that the traffic-splitting ACLs are matching read requests correctly. If the ACL isn’t triggering, all requests will go to the master backend.

Final Tip #

Start with the simplest checks first: verify HAProxy’s stats page for replica status, then check the replica’s Redis replication info. These will quickly rule out whether the issue is with the replica’s health or HAProxy’s routing logic.

内容的提问来源于stack exchange，提问作者JongHyeun Hahn