Boost异步TCP服务器accept handler运行一段时间后停止工作求助

Hmm, this sounds like a classic resource exhaustion or event loop block problem—let’s break down the most likely culprits based on your symptoms:

1. Blocked Event Loop from Long-Running Operations #

If your server’s accept handler or request processing logic includes synchronous, long-running tasks (like heavy computations, blocking I/O calls that don’t use async patterns), over time the event loop can get completely stuck. When this happens:

• The kernel still handles TCP three-way handshakes (so telnet/your client can "connect"), but the user-space accept handler never fires because the loop isn’t processing new events.
• Existing connections stop receiving data because the server can’t get around to sending more bytes.

How to debug:

• Check your code for any blocking calls (e.g., sleep(), synchronous file reads, unoptimized loops) in the same thread as your event loop.
• Use tools like strace -p <server-pid> or pstack <server-pid> to see what system calls your server is stuck on.
• If you’re using an async framework (like libevent, epoll, or select), ensure all I/O operations are non-blocking and processing logic doesn’t hog the loop.

2. File Descriptor (FD) Leak #

TCP connections consume file descriptors, and if your server doesn’t properly clean them up, you’ll hit the process’s FD limit after 20-25 minutes of running. Here’s what happens:

• The kernel can still complete TCP handshakes (since that’s handled at the kernel level), but your server’s accept() call will fail silently (or with an EMFILE error) because it can’t open new FDs. This is why clients can "connect" but your handler never triggers.
• Existing connections might stop receiving data if the server can’t manage its existing FDs properly.

How to debug:

• Run lsof -p <server-pid> or ss -tulnp | grep <server-pid> to count the number of open FDs for your server. Compare this to your process’s limit (check with ulimit -n).
• Audit your code to ensure every client connection is closed with close() (or equivalent) in all code paths—including error cases, timeouts, and when the client disconnects unexpectedly.
• Check if your system’s global FD limit is too low (you might need to adjust /etc/security/limits.conf).

3. Send Buffer Blockage or TCP Flow Control #

If your server is sending data faster than the client can receive/process it, the server’s TCP send buffer will fill up. Depending on your socket configuration:

• For blocking sockets: The send() call will hang indefinitely, blocking the event loop and preventing new accept calls or further data sends.
• For non-blocking sockets: If you don’t handle EAGAIN/EWOULDBLOCK errors properly, your server might stop attempting to send data entirely.

How to debug:

• Check if your server uses blocking or non-blocking sockets. If blocking, avoid calling send() in the main event loop thread—use a thread pool or switch to non-blocking I/O with proper buffer management.
• Run ss -t to inspect the TCP state of your connections. Look for high SEND-Q values, which indicate data stuck in the server’s send buffer.
• On the client side, verify that it’s reading data from its receive buffer regularly—if the client’s processing logic is stuck, it won’t acknowledge new data, triggering TCP flow control.

4. Memory Leak Causing Resource Starvation #

A slow memory leak can gradually consume all available RAM, forcing the OS to use swap space. This makes your server process extremely slow, to the point where it can’t handle new accept events or send data in a timely manner.

How to debug:

• Monitor your server’s memory usage over time with top, htop, or ps aux—look for a steady upward trend.
• Use tools like valgrind --leak-check=full ./your-server to detect explicit memory leaks, or pmap <server-pid> to inspect memory allocation patterns.
• Check your code for unclosed dynamic allocations (e.g., forgotten free() or delete calls), especially in connection-specific logic that runs repeatedly.

5. Mishandled Signals #

Signals like SIGPIPE (triggered when sending data to a closed connection) or unhandled SIGCHLD (if using child processes) can disrupt your server’s operation:

• If you don’t ignore SIGPIPE, your server might crash silently (but you said it "looks like it’s still running," so maybe you ignored it— but then send() will return an error that you might not be handling, leading to stopped data sends).
• Unhandled SIGCHLD can leave zombie processes hanging around, consuming system resources over time.

How to debug:

• Add signal handling for SIGPIPE (e.g., signal(SIGPIPE, SIG_IGN);) and ensure you check the return value of send()/write() for errors.
• Run ps aux | grep defunct to check for zombie processes—if you see any, add code to reap them with waitpid().
• Avoid doing heavy work in signal handlers, as they can block the event loop.

Quick Additional Checks #

• Add detailed logging to your server: Track connection counts, FD usage, send/receive success rates, and any error codes returned by system calls.
• Use tcpdump to capture traffic between server and client—this will tell you if the server is actually sending data, if the client is acknowledging it, or if new connection requests are reaching the server.
• Check system logs (/var/log/messages, dmesg) for signs of OOM (Out of Memory) killer triggering, FD limits being hit, or other kernel-level errors.

内容的提问来源于stack exchange，提问作者Shreyash Solanke