如何通过API管理YARN容量队列？附已实现的Hadoop安全策略

Got it, since you already have experience using curl to configure Ranger policies for Hadoop, extending that to manage YARN capacity queues via API should feel right at home. Let's walk through common queue management tasks (view, create/update, delete) with curl, and even wrap it into a reusable script like your existing hadoop_policy function to handle your 5 queues efficiently.

Prerequisites #

• Ensure you have access to the YARN ResourceManager's REST API (default endpoint: http://${yarn_resourcemanager_host}:8088/ws/v1/cluster/capacity)
• If your cluster is secured, use the appropriate authentication method (basic auth with -u like your Ranger script, or Kerberos with kinit and --negotiate flag)

1. View All Existing Capacity Queues #

First, let's confirm your current queue setup with a GET request:

curl -u "${yarn_admin}:${yarn_password}" \
http://${yarn_resourcemanager_host}:8088/ws/v1/cluster/capacity

This returns a JSON structure with all queues, their capacities, states, and permissions.

2. Create/Update a Capacity Queue #

YARN uses PUT requests for both creating new queues and updating existing ones. Here's a reusable function modeled after your hadoop_policy script:

manage_yarn_queue() {
    local queue_name=$1
    local queue_capacity=$2
    local queue_max_capacity=$3
    local yarn_host=${yarn_resourcemanager_host}
    local yarn_auth="${yarn_admin}:${yarn_password}"

    curl -H "Content-Type: application/json" -u "${yarn_auth}" \
    -X PUT \
    -d "{
        \"queue\": {
            \"queueName\": \"${queue_name}\",
            \"capacity\": ${queue_capacity},
            \"maximumCapacity\": ${queue_max_capacity},
            \"state\": \"RUNNING\",
            \"aclSubmitApps\": \"${user},*\",
            \"aclAdministerApps\": \"${yarn_admin},*\"
        }
    }" \
    http://${yarn_host}:8088/ws/v1/cluster/capacity/${queue_name}

    if [ $? != 0 ]; then
        echo "Error creating/updating queue ${queue_name}"
        exit 1
    fi
}

Use the Function for Your 5 Queues #

Just call the function with your queue names and capacity values (adjust numbers to fit your total capacity needs, ensuring sum of capacities doesn't exceed 100% for parent queues):

# Example configuration for 5 queues with equal base capacity
manage_yarn_queue "analytics_queue" 20 40
manage_yarn_queue "ml_queue" 20 40
manage_yarn_queue "etl_queue" 20 40
manage_yarn_queue "adhoc_queue" 20 40
manage_yarn_queue "default_queue" 20 40

3. Delete a Capacity Queue #

To delete a queue, it must be empty (no running/pending applications) and in a STOPPED state. Use a DELETE request:

curl -u "${yarn_admin}:${yarn_password}" \
-X DELETE \
http://${yarn_resourcemanager_host}:8088/ws/v1/cluster/capacity/${queue_name}

Key Notes #

• Capacity Validation: The sum of child queue capacities under a parent must not exceed 100%. Adjust queue_capacity values accordingly.
• Security Adjustments: If using Kerberos instead of basic auth, replace the -u parameter with --negotiate -u : and ensure you've run kinit with a privileged user account first.
• Permissions: Tweak aclSubmitApps and aclAdministerApps to control which users/groups can submit jobs to or manage each queue.

Since you're already familiar with shell scripting from your Ranger policy setup, this approach should integrate smoothly into your existing workflow.

内容的提问来源于stack exchange，提问作者vero