能否以yarn-client模式启动Dask Yarn集群？

Great question! I’ve worked with both Dask YARN and Spark YARN deployments, so let’s break this down clearly for you:

Yes, Dask YARN does support an equivalent to Spark’s yarn-client mode—it’s called client mode in Dask’s terminology, and it works almost exactly how you’d expect.

How it works in Dask #

In client mode:

• Your driver program (the Python script/notebook where you define your Dask tasks) runs locally on the master node (the machine you’re submitting from).
• The Dask YARN Application Master (AM) runs as a separate process in the YARN cluster, but its only job is to manage and coordinate the worker nodes—it doesn’t run your driver logic (unlike Dask’s cluster mode, where the driver runs inside the AM).

This means all core node resources can be dedicated to worker nodes, just like in Spark’s yarn-client mode.

Adjusting your configuration to match your needs #

Your current code is actually already using client mode (since you’re running the script on your master node to create the cluster). To ensure the AM doesn’t consume core node resources (freeing them entirely for workers), you can tweak a few parameters:

1. Explicitly set AM resources: You can control how much memory/vCPU the AM uses (it defaults to 1 vCore and 512MB memory, which is minimal).
2. Target the AM to run on non-core nodes: If your YARN cluster uses node labels (e.g., edge nodes labeled edge), you can force the AM to schedule there instead of core nodes.

Here’s an updated version of your code with these adjustments:

import dask_yarn
from dask.distributed import Client

# Initialize the cluster with AM-specific configurations
cluster = dask_yarn.DaskYARNCluster(
    env='/home/hadoop/reqs/dvss.zip',
    lang='en_US.UTF-8',
    am_cpus=1,  # Keep AM's vCore usage minimal
    am_memory=512,  # Minimal memory for AM
    node_label='edge'  # If you have non-core edge nodes with this label
)

# Start workers using core node resources
cluster.start(n_workers=4, memory=5120, cpus=3)

# Connect your local driver to the cluster
client = Client(cluster)

Key notes #

• If your YARN cluster doesn’t use node labels, you can still rely on YARN’s scheduler to place the AM on a less critical node (though this depends on your cluster’s scheduling policies).
• Unlike Spark’s yarn-cluster mode, Dask’s cluster mode runs the driver inside the AM—so sticking to client mode is exactly what you need to mirror Spark’s yarn-client behavior.

内容的提问来源于stack exchange，提问作者j-bennet