基于Redis的RPC机制：如何检测客户端断开及处理服务端崩溃？

Awesome questions—your Redis RPC setup is a solid starting point, but these edge cases are critical to handle for production reliability. Let's tackle each one with practical, actionable solutions:

1. Can we detect client disconnections? #

Redis doesn’t provide a built-in callback for client disconnections, but we can work around this with indirect mechanisms tailored to your RPC flow:

• Client heartbeat keys: When a client submits a job, have it create an expiration-based heartbeat key like SET client:alive:$job_id "active" EX 60 (adjust the TTL to match your expected client lifecycle). The client should periodically refresh this key with EXPIRE client:alive:$job_id 60 while waiting for results. Before processing a job, your server first checks EXISTS client:alive:$job_id—if the key is gone, the client has disconnected, so you can safely discard the job instead of wasting resources processing it.
• Leverage BLPOP’s implicit behavior: If a client disconnects while blocked on BLPOP job:done:$job_id, Redis automatically terminates that blocking connection. However, the server won’t get explicit notification. Combining this with the heartbeat key ensures you don’t push results to a queue that no client is listening to, preventing stale data buildup.
• Optional: Switch to Redis Streams (if architecture changes are feasible): Streams with consumer groups track consumer activity natively. You can use XPENDING to check if a client is still active before processing tasks, but this would require refactoring your list-based queue system.

2. How to avoid infinite client waiting if the server crashes after fetching a task? #

The core here is adding time-bound safeguards on both client and server sides:

• Add a timeout to client-side BLPOP: Instead of using BLPOP job:done:$job_id 0 (infinite block), set a reasonable timeout like BLPOP job:done:$job_id 30 (30 seconds). If the client times out, it can either retry the job submission or return an error to the caller—this prevents the client from hanging indefinitely.
• Track in-progress tasks with TTL keys: When your server fetches a job via BLPOP, immediately create a TTL key to mark it as in progress: SET task:processing:$job_id "<job-details>" EX 60 (use a TTL slightly longer than your client timeout). Run a background cron job or Redis-based scheduler to scan for expired task:processing:* keys—if a key is expired, it means the server crashed mid-processing, so you can RPUSH the job back to job:queue for reprocessing.
• Atomic job fetch + processing marker (Lua script): To eliminate the race condition where the server fetches a job but crashes before marking it as in progress, use a Lua script to atomically pop the job and set the processing key (note: you can’t use BLPOP in Lua, so switch to LPOP for atomicity):

local job = redis.call('LPOP', 'job:queue')
if job then
    -- Assume job is a JSON array with job_id as the first element
    local job_id = cjson.decode(job)[1]
    redis.call('SET', 'task:processing:' .. job_id, job, 'EX', 60)
    return job
end
return nil

• Client-side cleanup on timeout: If a client hits its BLPOP timeout, have it run DEL job:done:$job_id to clean up the now-unused result queue. This prevents leftover data from cluttering Redis and avoids confusion if the server eventually recovers and tries to push a result to a dead queue.

内容的提问来源于stack exchange，提问作者ioquatix