如何通过Python库或Azure CLI提交带代码后置的U-SQL作业？

Absolutely, you can submit U-SQL jobs with code-behind files (either C# or Python) using Azure CLI or the Azure Python SDK—no Visual Studio required, and both tools work seamlessly cross-platform on Linux and Windows. Here's a detailed breakdown of how to implement this:

Using Azure CLI #

The Azure CLI's az u-sql job submit command supports attaching code-behind files via the --additional-files parameter. This accepts one or more local files, which are uploaded and referenced by your U-SQL script during execution.

Example Commands #

# Submit a U-SQL job with a C# code-behind file
az u-sql job submit \
    --account <your-dla-account-name> \
    --job-name "U-SQL-Job-CSharp-CodeBehind" \
    --script @path/to/your/script.usql \
    --additional-files @path/to/your/code-behind.cs

# Submit a U-SQL job with a Python code-behind file
az u-sql job submit \
    --account <your-dla-account-name> \
    --job-name "U-SQL-Job-Python-CodeBehind" \
    --script @path/to/your/script.usql \
    --additional-files @path/to/your/python-code.py

# Submit with multiple code-behind files (mix C# and Python)
az u-sql job submit \
    --account <your-dla-account-name> \
    --job-name "U-SQL-Job-Multiple-CodeBehind" \
    --script @path/to/your/script.usql \
    --additional-files @path/to/file1.cs @path/to/file2.py

Notes #

• Use the @ prefix to reference local files (this tells the CLI to read the file content instead of treating it as a string).
• Ensure you're logged into Azure CLI first with az login (or use service principal authentication for automated workflows).

Using Azure Python SDK #

You can automate job submissions with the azure-mgmt-datalake-analytics library, which supports attaching code-behind files as additional resources in the job request.

Step 1: Install Required Libraries #

pip install azure-mgmt-datalake-analytics azure-identity

Step 2: Python Code Example #

from azure.identity import DefaultAzureCredential
from azure.mgmt.datalake.analytics.job import DataLakeAnalyticsJobManagementClient
from azure.mgmt.datalake.analytics.job.models import (
    USqlJobProperties,
    JobInformation,
    JobState
)
import time

# Configuration
SUBSCRIPTION_ID = "<your-subscription-id>"
RESOURCE_GROUP = "<your-resource-group-name>"
DLA_ACCOUNT_NAME = "<your-dla-account-name>"
JOB_NAME = "U-SQL-Job-With-CodeBehind-PythonSDK"
SCRIPT_PATH = "path/to/your/script.usql"
CODE_BEHIND_PATHS = [
    "path/to/your/csharp-code.cs",
    "path/to/your/python-code.py"
]

# Initialize authentication and client
credential = DefaultAzureCredential()
job_client = DataLakeAnalyticsJobManagementClient(
    credential,
    account_name=DLA_ACCOUNT_NAME
)

# Read script content
with open(SCRIPT_PATH, "r") as script_file:
    script_content = script_file.read()

# Prepare additional resources (code-behind files)
additional_resources = []
for file_path in CODE_BEHIND_PATHS:
    with open(file_path, "r") as code_file:
        additional_resources.append({
            "name": file_path.split("/")[-1],  # Use filename only for reference in U-SQL
            "content": code_file.read()
        })

# Define job properties
job_properties = USqlJobProperties(
    script=script_content,
    additional_resources=additional_resources
)

# Submit the job
job_info = job_client.job.create(
    account_name=DLA_ACCOUNT_NAME,
    job_name=JOB_NAME,
    parameters=JobInformation(
        properties=job_properties,
        type="USql"
    )
)

# Optional: Wait for job completion
print(f"Job submitted with ID: {job_info.job_id}")
while job_info.state not in [JobState.succeeded, JobState.failed, JobState.cancelled]:
    time.sleep(10)
    job_info = job_client.job.get(DLA_ACCOUNT_NAME, job_info.job_id)
    print(f"Current job state: {job_info.state}")

print(f"Job finished with state: {job_info.state}")

Notes #

• DefaultAzureCredential handles authentication across environments (supports Azure CLI, environment variables, Managed Identity, etc.), making it ideal for cross-platform automation.
• The additional_resources parameter expects a list of dictionaries, each with a name (the filename you'll reference in your U-SQL script) and content (the raw text of the code-behind file).

内容的提问来源于stack exchange，提问作者aaronsteers