Databricks：无需本地复制，直接从Azure Blob Storage上传文件至其他位置

我在Azure Blob Storage中存有一个文件，希望将其上传至其他位置，且无需复制到Databricks本地存储。当前我的代码需先将文件复制到本地再上传：

# Set up connection to Azure Blob Storage
spark.conf.set("fs.azure.account.key.[some location]", "[account key]")

# Copies the file to Databricks local storage
dbutils.fs.cp("wasbs://[folder location]/some_file.csv", "temp_some_file.csv")

# Setting up for upload data to other system
uploader = client.create_dataset_from_upload('data', 'csv') # This is an external library call

# Read the local copy file and upload it to another system
with open('/dbfs/temp_some_file.csv') as dataset:
    uploader.upload_file(dataset)

请问如何修改open()命令，使其直接指向Azure Blob Storage中的文件？

可以通过两种方式实现直接读取Azure Blob Storage中的文件并上传，无需复制到Databricks本地：

方法一：使用Databricks的dbutils.fs.open #

利用Databricks内置的dbutils.fs.open直接读取Blob存储中的文件，返回的文件流可直接传给上传方法，省去本地复制步骤：

# Set up connection to Azure Blob Storage
spark.conf.set("fs.azure.account.key.[some location]", "[account key]")

# Setting up for upload data to other system
uploader = client.create_dataset_from_upload('data', 'csv') # This is an external library call

# 直接读取Azure Blob Storage中的文件并上传
with dbutils.fs.open("wasbs://[folder location]/some_file.csv", mode="r", encoding="utf-8") as dataset:
    uploader.upload_file(dataset)

• 注意：如果上传方法要求特定编码，可在dbutils.fs.open中指定encoding参数；若需二进制读取，将mode改为rb。

方法二：使用Azure Storage Blob SDK直接读取 #

如果不想依赖Databricks的dbutils，可以使用Azure官方的Storage Blob SDK直接获取文件流：

1. 先安装依赖库（若未安装）：

pip install azure-storage-blob

2. 修改代码如下：

from azure.storage.blob import BlobServiceClient

# Set up connection to Azure Blob Storage
account_url = "https://[some location].blob.core.windows.net"
account_key = "[account key]"
blob_service_client = BlobServiceClient(account_url=account_url, credential=account_key)

# 获取Blob客户端
container_name = "[your container name]"
blob_name = "some_file.csv"
blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_name)

# Setting up for upload data to other system
uploader = client.create_dataset_from_upload('data', 'csv') # This is an external library call

# 直接读取Blob流并上传
with blob_client.download_blob() as dataset_stream:
    uploader.upload_file(dataset_stream)

这种方法更通用，不绑定Databricks环境，但需要额外安装SDK。

内容的提问来源于stack exchange，提问作者rcs