VS Code中Python脚本读取ADLS Gen1 CSV文件及权限疑问

Hey there! Since you've already got the ADL extension set up in VS Code and connected your Azure account, let's break down exactly how to read that CSV file from ADLS Gen1, plus clear up your questions about extra configs.

Do You Need to Create Scopes/Keys? #

Great news—you don't need to manually create scopes, access keys, or service principals right now. Because you're already logged into Azure via VS Code, we can leverage Azure's built-in identity authentication to automatically use your existing session. This saves you from managing extra credentials manually.

Step-by-Step Code to Read the CSV #

First, make sure you have the required packages installed. Open your terminal and run:

pip install azure-datalake-store pandas azure-identity

Then use this Python script to read your ADLS Gen1 CSV:

from azure.datalake.store import core, lib
from azure.identity import DefaultAzureCredential
import pandas as pd
from io import StringIO

# 1. Replace with your ADLS Gen1 account name (e.g., "mydatalakeaccount")
adls_account_name = "<your-adls-gen1-account-name>"

# 2. Authenticate using your VS Code Azure session (no manual keys needed!)
credential = DefaultAzureCredential()
adl_auth = lib.auth(tenant_id=None, credential=credential, resource="https://datalake.azure.net/")

# 3. Create a client to interact with your ADLS Gen1 account
adls_client = core.AzureDLFileSystem(adl_auth, store_name=adls_account_name)

# 4. Replace with your actual file path in ADLS Gen1 (e.g., "/raw-data/breslow.csv")
adls_file_path = "/path/to/breslow.csv"

# 5. Read the file content and load into pandas
with adls_client.open(adls_file_path, 'r') as file:
    csv_content = file.read()

# Use StringIO to mimic a local file for pandas
df = pd.read_csv(StringIO(csv_content))
print(df)

Key Notes #

• Permissions Check: Double-check that your Azure account has at least the Data Lake Store Reader role assigned to the ADLS Gen1 account (or a higher role like Contributor). Without this, you'll get access denied errors.
• Alternative for Non-Interactive Environments: If you ever need to run this script outside VS Code (like on a server), you'd need to set up a service principal with appropriate permissions and use its client ID/secret for authentication. But for your current VS Code workflow, the above method works perfectly.

内容的提问来源于stack exchange，提问作者Lav Mehta