Python Socket多列表数据传输优化方案咨询（Agent-Manager架构）

Hey Quang, let's fix that socket data transfer issue you're dealing with—your current approach has some big inefficiencies (like spawning multiple connections for each resource list) and potential bugs with data splitting (since str(x) can create ambiguous strings, and recv() might not get the full list in one go). Here are a few better, more reliable solutions:

This is the best approach because JSON is human-readable, cross-language compatible, and safe for transmitting structured data. We'll package all your resource lists into a single dictionary, serialize it to a JSON string, and send it over a single socket connection. We'll also add a length prefix to avoid "sticky packet" issues (where multiple sends get merged into one recv).

Agent端代码 #

import socket
import json

# Assume these functions are already implemented
def OS():
    return ["Linux", "Ubuntu", "20.04 LTS"]
def CPU():
    return ["Intel Core i7-10700", "8 cores", "3.8GHz"]
def DISK():
    return ["/dev/sda", "512GB SSD", "150GB used"]
def MEMORY():
    return ["16GB DDR4", "7.2GB used"]
def NETWORK():
    return ["192.168.1.42", "1Gbps Ethernet", "100MB/s current throughput"]
def FIREWALL():
    return ["ufw", "active", "allow 22/tcp, 80/tcp"]

# Package all resource data into a single dictionary
resource_data = {
    "os": OS(),
    "cpu": CPU(),
    "disk": DISK(),
    "memory": MEMORY(),
    "network": NETWORK(),
    "firewall": FIREWALL()
}

# Serialize to JSON and encode to bytes
json_data = json.dumps(resource_data).encode("utf-8")

# Establish socket connection and send data
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
    s.connect(("localhost", 12345))
    # First send the length of the data (4 bytes, big-endian)
    s.sendall(len(json_data).to_bytes(4, byteorder="big"))
    # Then send the actual JSON data
    s.sendall(json_data)
    print("All resource data sent successfully")
except socket.error as e:
    print(f"Error sending data: {e}")
finally:
    s.close()

Manager端代码 #

import socket
import json

# Set up server socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("localhost", 12345))
s.listen(10)
print("Manager listening for Agent connections...")

conn, addr = s.accept()
print(f"Connected to Agent at {addr}")

try:
    # First receive the data length (4 bytes)
    length_bytes = conn.recv(4)
    if not length_bytes:
        raise Exception("No data length received—connection closed early")
    data_length = int.from_bytes(length_bytes, byteorder="big")
    
    # Receive the full JSON data (handle partial recvs)
    received_data = b""
    while len(received_data) < data_length:
        chunk = conn.recv(min(data_length - len(received_data), 1024))
        if not chunk:
            raise Exception("Connection closed before full data was received")
        received_data += chunk
    
    # Parse JSON back to a dictionary
    resource_data = json.loads(received_data.decode("utf-8"))
    
    # Extract individual resource lists
    os_info = resource_data["os"]
    cpu_info = resource_data["cpu"]
    disk_info = resource_data["disk"]
    memory_info = resource_data["memory"]
    network_info = resource_data["network"]
    firewall_info = resource_data["firewall"]
    
    # Example: Print parsed data
    print("\nReceived Resource Data:")
    print(f"OS: {os_info}")
    print(f"CPU: {cpu_info}")
    print(f"Disk: {disk_info}")
    print(f"Memory: {memory_info}")
    print(f"Network: {network_info}")
    print(f"Firewall: {firewall_info}")
    
    # Add your database write logic here
    # e.g., db.insert(os_info, cpu_info, disk_info, ...)

except Exception as e:
    print(f"Error processing data: {e}")
finally:
    conn.close()
    s.close()

If you only need to communicate between Python applications, Pickle can serialize more complex Python objects (like custom classes) that JSON can't. Warning: Never use Pickle with untrusted data—it can execute arbitrary code.

Agent端代码（修改部分） #

import pickle

# Serialize with Pickle
pickle_data = pickle.dumps(resource_data)

# Send length + data (same as JSON example)
s.sendall(len(pickle_data).to_bytes(4, byteorder="big"))
s.sendall(pickle_data)

Manager端代码（修改部分） #

import pickle

# After receiving full data
resource_data = pickle.loads(received_data)

You could separate each list's string representation with a unique delimiter (like |||) that won't appear in your data. But this is error-prone—if any resource data contains your delimiter, parsing will break. Only use this if you can't use serialization.

Agent端代码示例 #

# Convert each list to string and join with delimiter
delimiter = "|||"
data_str = delimiter.join([str(lst) for lst in resource_info])
data = data_str.encode("utf-8")

# Send length + data (same as before)

Manager端代码示例 #

import ast

# After receiving full data
data_str = received_data.decode("utf-8")
delimiter = "|||"
# Use ast.literal_eval() instead of eval() for safety
resource_lists = [ast.literal_eval(lst_str) for lst_str in data_str.split(delimiter)]

Key Improvements Over Your Original Approach #

• Single Connection: No more spawning multiple connections for each resource list—this is far more efficient.
• Structured Data: Serialization preserves the list structure, so you don't have to manually split strings and handle parsing errors.
• Sticky Packet Protection: Sending the data length first ensures we receive the full payload, even if recv() gets partial chunks.

内容的提问来源于stack exchange，提问作者Park Yo Jin