如何用Python无需下载直接读取Azure Data Lake存储的PDF文件？

问题分析 #

你的代码存在两个核心错误：

1. 直接用open()处理字节流：downloaded_bytes是文件的字节数据，open()函数仅接受文件路径字符串，无法直接读取字节流。
2. SAS令牌未拼接成完整URL：generate_file_sas()生成的只是SAS令牌，不是可直接访问的文件URL，且生成时file_name参数误用了未定义的dir_name，应替换为目标PDF文件名。

可行解决方案 #

方法1：用BytesIO包装字节流（内存处理）

将下载的字节数据包装成类文件对象，直接供PDFMiner读取，无需写入本地文件。

from azure.storage.filedatalake import DataLakeServiceClient
import io
from pdfminer.pdfpage import PDFPage

# 配置Azure存储信息
storage_account_name = "mystorageaccount"
storage_account_key = "mystoragekey"
container_name = "mycontainer"
directory_name = 'mydirectory'
pdf_file_name = 'XXX.pdf'

# 连接ADLS并下载文件字节
service_client = DataLakeServiceClient(
    account_url=f"https://{storage_account_name}.dfs.core.windows.net",
    credential=storage_account_key
)
file_system_client = service_client.get_file_system_client(file_system=container_name)
directory_client = file_system_client.get_directory_client(directory_name)
file_client = directory_client.get_file_client(pdf_file_name)
downloaded_bytes = file_client.download_file().readall()

# 用BytesIO模拟文件对象，传递给PDFMiner
with io.BytesIO(downloaded_bytes) as infile:
    pages = PDFPage.get_pages(infile, check_extractable=False)
    # 后续可遍历pages处理，例如提取文本
    for page in pages:
        print(f"读取页面ID: {page.pageid}")

方法2：使用完整SAS URL读取远程文件

先生成有效的SAS URL，通过HTTP请求获取文件流后再处理，适合不想提前下载完整文件的场景。

from azure.storage.filedatalake import generate_file_sas
from azure.storage.filedatalake._shared import FileSasPermissions
import requests
import io
from pdfminer.pdfpage import PDFPage

# 配置Azure存储信息
storage_account_name = "mystorageaccount"
storage_account_key = "mystoragekey"
container_name = "mycontainer"
directory_name = 'mydirectory'
pdf_file_name = 'XXX.pdf'

# 生成带读取权限的SAS令牌，设置合理过期时间
sas_token = generate_file_sas(
    account_name=storage_account_name,
    file_system_name=container_name,
    directory_name=directory_name,
    file_name=pdf_file_name,
    credential=storage_account_key,
    permission=FileSasPermissions(read=True),
    expiry="2024-12-31"  # 根据需求调整过期时间
)

# 拼接完整的可访问URL
sas_url = f"https://{storage_account_name}.dfs.core.windows.net/{container_name}/{directory_name}/{pdf_file_name}?{sas_token}"

# 请求文件流并处理
response = requests.get(sas_url, stream=True)
response.raise_for_status()  # 确保请求成功

with io.BytesIO(response.content) as infile:
    pages = PDFPage.get_pages(infile, check_extractable=False)
    for page in pages:
        print(f"读取页面ID: {page.pageid}")

方法3：大文件临时文件处理

如果PDF文件体积较大，内存处理可能占用过多资源，可先下载到临时文件，读取后再删除。

from azure.storage.filedatalake import DataLakeServiceClient
import tempfile
import os
from pdfminer.pdfpage import PDFPage

# 配置Azure存储信息
storage_account_name = "mystorageaccount"
storage_account_key = "mystoragekey"
container_name = "mycontainer"
directory_name = 'mydirectory'
pdf_file_name = 'XXX.pdf'

# 连接ADLS
service_client = DataLakeServiceClient(
    account_url=f"https://{storage_account_name}.dfs.core.windows.net",
    credential=storage_account_key
)
file_system_client = service_client.get_file_system_client(file_system=container_name)
directory_client = file_system_client.get_directory_client(directory_name)
file_client = directory_client.get_file_client(pdf_file_name)

# 创建临时文件并下载
with tempfile.NamedTemporaryFile(delete=False, suffix='.pdf') as tmp_file:
    file_client.download_file().readinto(tmp_file)

# 读取临时文件
with open(tmp_file.name, 'rb') as infile:
    pages = PDFPage.get_pages(infile, check_extractable=False)
    for page in pages:
        print(f"读取页面ID: {page.pageid}")

# 删除临时文件
os.unlink(tmp_file.name)

内容的提问来源于stack exchange，提问作者JR92