首次使用MQTT传输二进制文件，需在消息中附带元数据

Nice work getting the basic binary file transfer working with Paho MQTT! Sending metadata alongside your file is a common need, and there are a few solid ways to pull this off—let’s walk through the most practical options with code examples tailored to your setup.

1. Package Metadata + Binary Data into a Structured Payload #

One straightforward approach is to wrap both your metadata and encoded binary file into a single JSON object. Since JSON doesn’t natively support binary data, you’ll encode the file bytes to a base64 string (a text-safe format) that the receiver can decode back to bytes.

Here’s how to adjust your code:

import paho.mqtt.client as paho
import json
import base64
import time

# Read your binary file
with open("./file_name.csv.gz", "rb") as f:
    file_bytes = f.read()

# Define your metadata (customize this to your needs)
metadata = {
    "filename": "file_name.csv.gz",
    "timestamp": time.time(),
    "file_size_bytes": len(file_bytes),
    "content_type": "application/gzip"
}

# Encode binary data to base64 and combine with metadata
payload = json.dumps({
    "metadata": metadata,
    "file_data": base64.b64encode(file_bytes).decode("utf-8")
})

# MQTT setup and publish
mqttc = paho.Client()
mqttc.will_set("/event/dropped", "Sorry, I seem to have died.")
mqttc.connect(*connection definition here*)
mqttc.publish("hello/world", payload)

On the receiver side, you’d parse the JSON, extract the base64 string, and decode it back to bytes:

import base64
import json

def on_message(client, userdata, msg):
    payload = json.loads(msg.payload.decode("utf-8"))
    metadata = payload["metadata"]
    file_bytes = base64.b64decode(payload["file_data"])
    
    # Use metadata and file bytes as needed
    print(f"Received file: {metadata['filename']} (size: {metadata['file_size_bytes']} bytes)")
    with open(metadata["filename"], "wb") as f:
        f.write(file_bytes)

Pros: Works with all MQTT versions, self-contained payload (no need to coordinate multiple topics).
Cons: Base64 encoding adds ~33% to your payload size, which might matter for large files.

2. Use MQTT 5.0 Message Properties #

If your MQTT broker supports MQTT 5.0 (most modern brokers like Mosquitto, EMQX do), you can attach metadata directly to the message as user properties—this keeps your binary payload untouched, avoiding base64 overhead.

Adjust your publish code like this:

import paho.mqtt.client as paho
import time

# Read binary file
with open("./file_name.csv.gz", "rb") as f:
    file_bytes = bytearray(f.read())

# Define metadata as user properties
user_properties = [
    ("filename", "file_name.csv.gz"),
    ("timestamp", str(time.time())),
    ("file_size_bytes", str(len(file_bytes))),
    ("content_type", "application/gzip")
]

# MQTT setup (explicitly use MQTTv5 if not default)
mqttc = paho.Client(protocol=paho.MQTTv5)
mqttc.will_set("/event/dropped", "Sorry, I seem to have died.")
mqttc.connect(*connection definition here*)

# Publish with user properties
mqttc.publish(
    "hello/world",
    file_bytes,
    properties=paho.Properties(paho.PROPERTY_USER_PROPERTY, user_properties)
)

On the receiver side, access the properties from the message:

def on_message(client, userdata, msg):
    # Extract metadata from user properties
    metadata = dict(msg.properties.user_property)
    file_bytes = msg.payload
    
    # Convert string values back to appropriate types if needed
    metadata["timestamp"] = float(metadata["timestamp"])
    metadata["file_size_bytes"] = int(metadata["file_size_bytes"])
    
    print(f"Received file: {metadata['filename']}")
    with open(metadata["filename"], "wb") as f:
        f.write(file_bytes)

Pros: No payload bloat, clean separation of metadata and file data, native MQTT feature.
Cons: Requires MQTT 5.0 support from both client and broker.

3. Send Metadata and File Data on Separate Topics #

Another option is to split the data across two MQTT topics: one for metadata (as JSON) and one for the binary file. You can use a shared topic prefix to link them (e.g., hello/world/metadata and hello/world/file).

Sender code:

import paho.mqtt.client as paho
import json
import time

# Read binary file
with open("./file_name.csv.gz", "rb") as f:
    file_bytes = bytearray(f.read())

# Define metadata
metadata = {
    "filename": "file_name.csv.gz",
    "timestamp": time.time(),
    "file_size_bytes": len(file_bytes),
    "correlation_id": "unique-file-id-123"  # Optional: to explicitly link metadata to file
}

# MQTT setup
mqttc = paho.Client()
mqttc.will_set("/event/dropped", "Sorry, I seem to have died.")
mqttc.connect(*connection definition here*)

# Publish metadata first, then file data (order can vary)
mqttc.publish("hello/world/metadata", json.dumps(metadata))
mqttc.publish("hello/world/file", file_bytes)

Receiver code (subscribe to both topics and track metadata):

import json
import paho.mqtt.client as paho

# Store metadata temporarily until matching file is received
pending_metadata = {}

def on_message(client, userdata, msg):
    global pending_metadata
    if msg.topic == "hello/world/metadata":
        metadata = json.loads(msg.payload.decode("utf-8"))
        # Use correlation ID as a key to match with the right file
        pending_metadata[metadata["correlation_id"]] = metadata
    elif msg.topic == "hello/world/file":
        # For this example, we'll assume metadata was received first
        # Alternatively, you could embed the correlation ID in the file topic
        file_bytes = msg.payload
        # Pick the latest metadata (or use a specific correlation ID if needed)
        if pending_metadata:
            correlation_id = next(iter(pending_metadata.keys()))
            metadata = pending_metadata.pop(correlation_id)
            print(f"Received file: {metadata['filename']}")
            with open(metadata["filename"], "wb") as f:
                f.write(file_bytes)

# Subscribe to both topics
mqttc = paho.Client()
mqttc.on_message = on_message
mqttc.connect(*connection definition here*)
mqttc.subscribe("hello/world/metadata")
mqttc.subscribe("hello/world/file")
mqttc.loop_forever()

Pros: No encoding overhead, allows receivers to subscribe to only metadata or only files if needed.
Cons: Requires handling synchronization between the two messages (e.g., using a correlation ID to match metadata to the right file), which adds a bit of complexity.

Which Should You Choose? #

• If your broker supports MQTT 5.0, go with Option 2—it’s the cleanest and most efficient.
• If you need compatibility with older MQTT versions, Option 1 is the safest bet (just be aware of the base64 size increase).
• If you want to decouple metadata and file processing (e.g., some clients only care about metadata), Option 3 makes sense.

内容的提问来源于stack exchange，提问作者user697110