如何通过REST API/HTTP请求创建Dataproc集群？Python新手代码排障求助

Hey there! Let's break down why your code isn't working and walk through the correct way to create a Dataproc cluster via REST API.

1. Core Issues in Your Current Code #

Let's start with the key problems that are preventing your cluster from being created:

• Incorrect request body format: You're using data=data in the requests.post call, which sends the payload as form-urlencoded data. Dataproc's REST API requires JSON-formatted bodies, so you need to use json=data instead.
• Missing authentication: Google Cloud APIs require valid credentials to authorize requests. Without authentication, you'll get 401/403 errors.
• Wrong endpoint URL: The regions part of the URL should be a region (like us-central1), not a specific zone (like us-central1-b). The zone is specified in the request body instead.
• Optional parameter conflict: Leaving configBucket empty can cause validation issues. If you don't need a custom config bucket, just omit this field entirely.

2. Fixed Code Example #

First, install the required dependencies for authentication and HTTP requests:

pip install requests google-auth

Here's the corrected code with proper authentication and request formatting:

import requests
from google.oauth2 import service_account
from google.auth.transport.requests import Request

# Replace these with your own values
SERVICE_ACCOUNT_KEY_PATH = "/path/to/your/service-account-key.json"
PROJECT_ID = "your-project-id"
REGION = "us-central1"
ZONE = "us-central1-b"

# 1. Get authentication token using a service account
credentials = service_account.Credentials.from_service_account_file(
    SERVICE_ACCOUNT_KEY_PATH,
    scopes=["https://www.googleapis.com/auth/cloud-platform"]
)
credentials.refresh(Request())
auth_headers = {"Authorization": f"Bearer {credentials.token}"}

# 2. Correct endpoint URL (uses region, not zone)
endpoint_url = f"https://dataproc.googleapis.com/v1/projects/{PROJECT_ID}/regions/{REGION}/clusters"

# 3. Cleaned-up cluster configuration
cluster_config = {
    "clusterName": "cluster-1",
    "config": {
        "gceClusterConfig": {
            # Use full resource URI for zone
            "zoneUri": f"projects/{PROJECT_ID}/zones/{ZONE}"
            # Uncomment below if using a custom subnet (use full URI)
            # "subnetworkUri": f"projects/{PROJECT_ID}/regions/{REGION}/subnetworks/default"
        },
        "masterConfig": {
            "numInstances": 1,
            "machineTypeUri": "n1-standard-1",
            "diskConfig": {
                "bootDiskSizeGb": 500,
                "numLocalSsds": 0
            }
        },
        "workerConfig": {
            "numInstances": 2,
            "machineTypeUri": "n1-standard-1",
            "diskConfig": {
                "bootDiskSizeGb": 100,
                "numLocalSsds": 0
            }
        }
    }
}

# 4. Send the POST request with JSON body and auth headers
response = requests.post(url=endpoint_url, json=cluster_config, headers=auth_headers)

# 5. Handle the response
if response.status_code == 200:
    operation_data = response.json()
    print(f"Cluster creation initiated! Operation name: {operation_data['name']}")
    print("You can check the operation status via this URL:")
    print(f"https://dataproc.googleapis.com/v1/{operation_data['name']}")
else:
    print(f"Error creating cluster (status code: {response.status_code})")
    print("Detailed error:")
    print(response.json())

3. Step-by-Step Guide to Creating a Cluster via REST API #

Step 1: Set Up Authentication #

• Use a service account (recommended for server-side code):
  1. Create a service account in your Google Cloud project.
  2. Assign it the roles/dataproc.editor role (or higher, like roles/editor for testing).
  3. Download the service account key JSON file.
• For client-side use, you can also use user OAuth2 tokens, but service accounts are more reliable for automation.

Step 2: Construct the Request #

• HTTP Method: POST
• URL: https://dataproc.googleapis.com/v1/projects/{PROJECT_ID}/regions/{REGION}/clusters
• Headers:
  • Authorization: Bearer {YOUR_AUTH_TOKEN}
  • Content-Type: application/json
• Request Body:
  • Required fields: clusterName, config.gceClusterConfig.zoneUri, masterConfig, workerConfig
  • All resource URIs (like zone, subnet, machine type) should be full Google Cloud resource paths (e.g., projects/my-project/zones/us-central1-b instead of just us-central1-b).

Step 3: Monitor the Operation #

When you send the request, you'll get a 200 OK response with an operation name. You can poll this operation to check if the cluster is created successfully:

# Example: Check operation status
op_url = f"https://dataproc.googleapis.com/v1/{operation_data['name']}"
status_response = requests.get(op_url, headers=auth_headers)
print(status_response.json())

4. Common Troubleshooting Tips #

• Permission errors: Ensure your service account has the necessary permissions (Dataproc Editor, Compute Instance Admin, etc.).
• Invalid resource URIs: Double-check that all URIs (zone, subnet) are correctly formatted with your project ID.
• Quota limits: Verify you haven't exceeded GCE instance or disk quotas in your region.

内容的提问来源于stack exchange，提问作者Dead pool