排查TCP连接瓶颈：Publisher端socket write方法耗时过长问题

Hey there, let's work through troubleshooting that slow write() call dragging down your TCP publisher's message rate—this is a super common gotcha with raw TCP setups, so let's break it down step by step.

First off, TCP write() blocks when the kernel's send buffer is full. Here's how to dig into that:

• Use tools like ss -ti dst <subscriber-ip>:<port> or netstat -tni to check the send buffer (sndbuf) usage and current congestion window (cwnd). If sndbuf is maxed out, that's a clear sign the publisher can't push data out because the network or subscriber can't keep up.
• Verify your socket's SO_SNDBUF setting. By default, kernels have a limit, but you can check/set it with setsockopt() (make sure you set it before connecting). Keep in mind the kernel might cap it at the system-wide tcp_wmem parameter, so check /proc/sys/net/ipv4/tcp_wmem too.

TCP is a reliable protocol—if the subscriber isn't reading data fast enough, it'll shrink its TCP receive window (rwnd), which tells the publisher to stop sending. This is probably the most common culprit:

• Check if the subscriber's read() calls are backed up. If its processing logic is slow (e.g., heavy computations, blocking I/O), it won't empty its receive buffer, forcing the publisher's write() to block.
• Use tcpdump or wireshark to capture packets between the two. Look at the TCP headers' Window Size field—if it drops to near zero, that's proof the subscriber is overwhelmed.

Sometimes the issue isn't the socket itself, but the host system:

• Monitor CPU usage with top or htop—if the publisher's process is maxing out a core (e.g., from generating messages too fast, or inefficient serialization), that could delay write() calls.
• Check network I/O with iostat or nload—if the network interface is saturated, packets will queue up, increasing write() latency.
• Ensure you haven't hit file descriptor limits. Use ulimit -n to check, and if needed, adjust the soft/hard limits for your process.

Small, frequent write() calls can introduce unnecessary overhead. Try these tweaks:

• Batch messages: Instead of sending each message individually, accumulate a batch of messages and send them in a single write() (or use writev() to send multiple buffers without copying data). This reduces system call overhead and avoids Nagle's algorithm delays.
• Disable Nagle's algorithm: If you're sending low-latency, small messages, set the TCP_NODELAY socket option. Nagle's algorithm waits to combine small packets, which can add latency when you need to send messages at a steady rate.
• Consider non-blocking I/O: Switch to non-blocking sockets with O_NONBLOCK—this way, write() won't block; instead, it'll return EAGAIN/EWOULDBLOCK when the buffer is full, letting you handle backpressure gracefully (e.g., pause message generation temporarily).

Get granular with timing to pinpoint when the slowdown happens:

• Add microsecond-level timing around your write() calls in code. Log how long each call takes, and correlate it with message volume or subscriber behavior.
• Use strace -T -p <publisher-pid> to trace the system calls. The -T flag shows the time spent in each write() call—this will tell you if the delay is happening in the kernel (e.g., waiting for buffer space) or elsewhere.

Start with checking the send buffer and subscriber's read/processing rate—those are the usual suspects. Once you narrow down the root cause, the fix will be much clearer!

内容的提问来源于stack exchange，提问作者user103583