将JSON列表作为单个文件上传Azure Blob调用upload_blob报错如何解决

错误原因 #

虽然upload_blob的类型标注支持Iterable[AnyStr]，但你传入的Python原生list类型会被底层依赖的requests库默认识别为表单键值对参数，requests会尝试遍历列表将每个元素拆为(键, 值)二元组，而你的列表元素是单个JSON字符串，因此触发ValueError: too many values to unpack (expected 2)错误。

解决方案 #

方案1：小数据量场景，直接拼接为完整字符串（最稳妥） #

将生成的JSON字符串列表拼接为单个完整字符串后再上传，可按需在每个JSON之间添加换行符避免解析混乱：

import json
from azure.storage.blob import BlobServiceClient

connect_str = "xxxxx"
container_name = "my-container"
blob_path="xx/yy/zz"
def upload_to_blob_storage(data):
   # 不需要换行可以将'\n'替换为空字符串''
   data_content = '\n'.join([json.dumps(event) for event in data])
   blob_service_client = BlobServiceClient.from_connection_string(connect_str)
   blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_path)
   blob_client.upload_blob(data_content, overwrite=False)

方案2：大数据量场景，使用迭代器/IO对象避免全量内存占用 #

如果数据量很大，不想一次性将全量内容加载到内存，可将数据包装为生成器迭代器或者IO类对象，SDK会将其识别为流数据，不会被requests误判为表单参数：

用生成器实现：

import json
from azure.storage.blob import BlobServiceClient

connect_str = "xxxxx"
container_name = "my-container"
blob_path="xx/yy/zz"
def upload_to_blob_storage(data):
    def data_generator():
        for event in data:
            # 转为字节流传输，按需添加换行
            yield (json.dumps(event) + '\n').encode('utf-8')
    blob_service_client = BlobServiceClient.from_connection_string(connect_str)
    blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_path)
    blob_client.upload_blob(data_generator(), overwrite=False)

用StringIO实现：

import json
from io import StringIO
from azure.storage.blob import BlobServiceClient

connect_str = "xxxxx"
container_name = "my-container"
blob_path="xx/yy/zz"
def upload_to_blob_storage(data):
    sio = StringIO()
    for event in data:
        sio.write(json.dumps(event) + '\n')
    # 将文件指针移到开头位置
    sio.seek(0)
    blob_service_client = BlobServiceClient.from_connection_string(connect_str)
    blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_path)
    blob_client.upload_blob(sio, overwrite=False)

内容的提问来源于stack exchange，提问作者JeyJ