You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

TCP通信速度慢、UDP丢包,如何高效传输JSON格式日志?

需求:高可靠+高效的JSON日志传输方案

我需要将大量JSON格式日志发送至问题排查分析程序,遇到以下问题:

  • 最初尝试UDP通信:收发即时,但丢包率达30%,且会丢弃不完整的JSON数据
  • 改用多线程TCP方案解决了丢包问题,但传输速度极慢,推测原因是发送CSV每行日志时反复创建和关闭socket

服务端多线程代码

import threading
import time
import socket  # 补充原代码遗漏的socket导入

def handle_client(conn, addr):
    print("[thread] starting")

    # recv message
    message = conn.recv(1024)
    message = message.decode()
    print("[thread] client:", addr, 'recv:', message)
    
    # simulate longer work
    time.sleep(5)

    # send answer
    message = "Bye!"
    message = message.encode()
    conn.send(message)
    print("[thread] client:", addr, 'send:', message)
    
    conn.close()

    print("[thread] ending")
   
host = '0.0.0.0'
port = 8080

s = socket.socket()
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)  # 解决"Address already in use"错误,需在bind前调用
s.bind((host, port))
s.listen(1)

all_threads = []

try:
    while True:
        print("Waiting for client")
        conn, addr = s.accept()
    
        print("Client:", addr)
        
        t = threading.Thread(target=handle_client, args=(conn, addr))
        t.start()
    
        all_threads.append(t)
except KeyboardInterrupt:
    print("Stopped by Ctrl+C")
finally:
    if s:
        s.close()
    for t in all_threads:
        t.join()

客户端发送代码

import socket
import struct
import csv
import json

def send_data(conn, data):
    size = len(data)
    size_in_4_bytes = struct.pack('I', size)
    conn.send(size_in_4_bytes)
    conn.send(data)

def recv_data(conn):
    size_in_4_bytes = conn.recv(4)
    size = struct.unpack('I', size_in_4_bytes)
    size = size[0]
    data = conn.recv(size)
    return data

def logLoop(ip, port, logPath):
    with open(logPath, newline='') as csvFile:
        cReader = csv.reader((line.replace('\0', '') for line in csvFile), delimiter=',')
        next(cReader)
        for r in cReader:
            sent = 0
            while not sent:
                try:
                    sock = socket.socket()
                    sock.connect((ip, port))
                    text = json.dumps(r)
                    data = text.encode()
                    send_data(sock, data)
                    sock.close()
                    sent = 1
                except (ConnectionRefusedError, ConnectionResetError, BrokenPipeError) as error:
                    print ("failed")
                    pass

    return

我仅掌握Socket通信基础内容,恳请提供既能保证数据不丢失、又能维持合理传输速度的解决方案。

内容的提问来源于stack exchange,提问作者user4343712

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.07.25 01:57:52