TCP通信速度慢、UDP丢包,如何高效传输JSON格式日志?
需求:高可靠+高效的JSON日志传输方案
我需要将大量JSON格式日志发送至问题排查分析程序,遇到以下问题:
- 最初尝试UDP通信:收发即时,但丢包率达30%,且会丢弃不完整的JSON数据
- 改用多线程TCP方案解决了丢包问题,但传输速度极慢,推测原因是发送CSV每行日志时反复创建和关闭socket
服务端多线程代码
import threading import time import socket # 补充原代码遗漏的socket导入 def handle_client(conn, addr): print("[thread] starting") # recv message message = conn.recv(1024) message = message.decode() print("[thread] client:", addr, 'recv:', message) # simulate longer work time.sleep(5) # send answer message = "Bye!" message = message.encode() conn.send(message) print("[thread] client:", addr, 'send:', message) conn.close() print("[thread] ending") host = '0.0.0.0' port = 8080 s = socket.socket() s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) # 解决"Address already in use"错误,需在bind前调用 s.bind((host, port)) s.listen(1) all_threads = [] try: while True: print("Waiting for client") conn, addr = s.accept() print("Client:", addr) t = threading.Thread(target=handle_client, args=(conn, addr)) t.start() all_threads.append(t) except KeyboardInterrupt: print("Stopped by Ctrl+C") finally: if s: s.close() for t in all_threads: t.join()
客户端发送代码
import socket import struct import csv import json def send_data(conn, data): size = len(data) size_in_4_bytes = struct.pack('I', size) conn.send(size_in_4_bytes) conn.send(data) def recv_data(conn): size_in_4_bytes = conn.recv(4) size = struct.unpack('I', size_in_4_bytes) size = size[0] data = conn.recv(size) return data def logLoop(ip, port, logPath): with open(logPath, newline='') as csvFile: cReader = csv.reader((line.replace('\0', '') for line in csvFile), delimiter=',') next(cReader) for r in cReader: sent = 0 while not sent: try: sock = socket.socket() sock.connect((ip, port)) text = json.dumps(r) data = text.encode() send_data(sock, data) sock.close() sent = 1 except (ConnectionRefusedError, ConnectionResetError, BrokenPipeError) as error: print ("failed") pass return
我仅掌握Socket通信基础内容,恳请提供既能保证数据不丢失、又能维持合理传输速度的解决方案。
内容的提问来源于stack exchange,提问作者user4343712
相关产品推荐
相关产品推荐

