Ubuntu下Python Socket本地进程交互性能异常排查求助

问题现象 #

在Mac（i7）上运行Python Socket实现的本地进程间双向交互，10000次来回耗时不到1秒；但在Ubuntu Server上执行相同代码，耗时超过30秒。仅单向通信（服务器发、客户端收）时，Ubuntu Server上也能在1秒内完成，推测问题出在Send()与Recv()的双向阻塞交互中。

相关代码 #

Server代码 #

if __name__=='__main__':
    server1 = ServerSocket("127.0.0.1", 4321)
    server1._conn_client() 
    print("xxx")

    start_time = time.time()
    for i in range(10000):
        server1.Recv()
        server1.Send("hello") ## delete this row to disable the server to send messages
        print(i)
    end_time = time.time()
    print("Server time:", end_time - start_time)

    server1.Send("hello")
    msg = server1.Recv()
    print("hi:", msg)
    server1.close()

Client代码 #

if __name__ == '__main__':
    client = ClientSocket("127.0.0.1", 4321)
    client.Connect()
    print("xxx")

    start = time.time()
    for i in range(10000):
        client.Send("hello from client1")
        client.Recv() ## delete this to disable the cleint to receive messages
    end = time.time()
    print("Client time:", end - start)

    msg =client.Recv()
    print(msg)
    client.Send("hello from client1")
    client.close()

Socket工具类代码 #

import socket
import struct
import json

class ServerSocket(object):
    def __init__(self, ip="127.0.0.1", port=7778):
        self.ip = ip
        self.port = port
        self._buffer_size = 80000
        self._init_socket()
        self._socket.settimeout(3000)

    def _init_socket(self):
        self._socket = socket.socket()
        ip_port = (self.ip, self.port)
        self._socket.bind(ip_port)

    def _conn_client(self):
        self._socket.listen(100)
        self._client_socket, addr = self._socket.accept()
        self._client_socket.settimeout(3000)

    def Recv(self):
        msg_len_pack = self._client_socket.recv(4)
        msg_len = struct.unpack('i', msg_len_pack)[0]
        if msg_len < self._buffer_size:
            msg_pack = self._client_socket.recv(msg_len)
            msg = json.loads(msg_pack.decode('utf-8'))
        return msg

    def Send(self,data):
        msg_pack = json.dumps(data).encode('utf-8')
        msg_len_pack = struct.pack('i', len(msg_pack))
        self._client_socket.send(msg_len_pack)
        self._client_socket.send(msg_pack) 

    def close(self):
        self._client_socket.close()

class ClientSocket(object):
    def __init__(self, ip="127.0.0.1", port=7778):
        self.ip = ip  
        self.port = port  
        self._buffer_size = 80000 

    def Connect(self):
        self._socket = socket.socket()  
        self._socket.connect((self.ip,self.port)) 

    def Send(self, data):
        msg_pack = json.dumps(data).encode('utf-8')
        msg_len_pack = struct.pack('i', len(msg_pack))
        self._socket.send(msg_len_pack)
        self._socket.send(msg_pack) 

    def Recv(self):
        msg_len_pack = self._socket.recv(4) 
        msg_len = struct.unpack('i', msg_len_pack)[0]
        if msg_len < self._buffer_size:
            msg_pack = self._socket.recv(msg_len)
            msg = json.loads(msg_pack.decode('utf-8'))
        return msg

    def close(self):
        self._socket.close()

问题原因分析 #

1. TCP Nagle算法影响：Ubuntu Server默认开启Nagle算法，该算法会延迟发送小数据包，等待当前数据包的ACK后再发送下一个，避免网络拥塞。但在本地回环的双向交互场景中，每次Send后都要等待Recv的ACK，10000次循环的等待时间被累积，导致总耗时剧增。单方向通信时没有ACK等待的来回逻辑，所以性能不受影响。
2. 频繁的TCP交互开销：代码中每次循环都执行一次Recv+Send，10000次循环对应10000次TCP来回交互，每次交互都涉及TCP的确认机制，放大了延迟。
3. Socket发送未优化：当前Send方法分两次发送长度和数据，可能触发两次TCP发送操作，进一步增加了Nagle算法的延迟触发概率。

修复方案 #

1. 禁用TCP Nagle算法 #

在Socket初始化后添加TCP_NODELAY选项，关闭Nagle算法，让小数据包立即发送，消除ACK等待延迟：

• 修改ServerSocket._init_socket方法：

  def _init_socket(self):
      self._socket = socket.socket()
      self._socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) # 新增
      ip_port = (self.ip, self.port)
      self._socket.bind(ip_port)
  

• 修改ServerSocket._conn_client方法（客户端连接的socket也要设置）：

  def _conn_client(self):
      self._socket.listen(100)
      self._client_socket, addr = self._socket.accept()
      self._client_socket.settimeout(3000)
      self._client_socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) # 新增
  

• 修改ClientSocket.Connect方法：

  def Connect(self):
      self._socket = socket.socket()  
      self._socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) # 新增
      self._socket.connect((self.ip,self.port)) 
  

2. 批量处理消息 #

减少TCP交互次数，比如一次发送多条消息的集合，客户端一次性接收后解析处理：

• Server端修改循环逻辑：

  # 替换原循环
  start_time = time.time()
  # 批量发送10000条消息
  batch_data = [f"hello_{i}" for i in range(10000)]
  server1.Send(batch_data)
  # 批量接收10000条消息（需要对应修改Recv逻辑支持批量）
  batch_msg = server1.Recv()
  end_time = time.time()
  print("Server time:", end_time - start_time)
  

• 对应的Socket工具类的Send/Recv方法需要支持批量数据的序列化与反序列化，本质是减少来回交互次数。

3. 优化Send/Recv可靠性 #

使用sendall()替代send()，确保数据完整发送；Recv时循环接收直到获取足够字节，避免因分片导致的阻塞：

• 修改Send方法：

  def Send(self,data):
      msg_pack = json.dumps(data).encode('utf-8')
      msg_len_pack = struct.pack('i', len(msg_pack))
      # 用sendall确保全部发送
      self._client_socket.sendall(msg_len_pack)
      self._client_socket.sendall(msg_pack) 
  

• 修改Recv方法（处理可能的分片）：

  def Recv(self):
      # 循环接收直到拿到4字节的长度
      msg_len_pack = b''
      while len(msg_len_pack) < 4:
          chunk = self._client_socket.recv(4 - len(msg_len_pack))
          if not chunk:
              raise ConnectionError("Connection closed")
          msg_len_pack += chunk
      msg_len = struct.unpack('i', msg_len_pack)[0]
      # 循环接收直到拿到完整的消息内容
      msg_pack = b''
      while len(msg_pack) < msg_len:
          chunk = self._client_socket.recv(min(msg_len - len(msg_pack), self._buffer_size))
          if not chunk:
              raise ConnectionError("Connection closed")
          msg_pack += chunk
      msg = json.loads(msg_pack.decode('utf-8'))
      return msg
  

内容的提问来源于stack exchange，提问作者XLC