部署Dask Cloudprovider EC2集群：私网通信与Dashboard访问的最佳方案

Option 1: Use SSH Tunneling (Simplest Approach) #

This method keeps internal worker-scheduler communication on private IPs (compatible with your security group) while letting you access the scheduler from your local machine via an SSH tunnel.

• Update config to use private IPs: Set use_private_ip: True in your YAML config. This ensures workers connect to the scheduler using its private IP, which is allowed by your security group's self-referential rule.
• Start the cluster: Once the scheduler instance is running, retrieve its public IP (via AWS Console/CLI) and private IP (via instance metadata or AWS tools).
• Establish SSH tunnel: Run this command on your local machine, replacing placeholders with your key path, scheduler public IP, and private IP:

  ssh -i /path/to/your/aws-key.pem -L 8786:<scheduler-private-ip>:8786 -L 8787:<scheduler-private-ip>:8787 ec2-user@<scheduler-public-ip>
  

  This forwards your local ports 8786 (scheduler) and 8787 (dashboard) to the scheduler's private IP ports.
• Connect from local: Point your Dask client to localhost:8786 and access the dashboard at http://localhost:8787. Workers will automatically use the private IP for internal communication.

Option 2: Configure Scheduler with Split Internal/External Addresses #

This approach configures the scheduler to advertise its private IP to workers and public IP to your local client, eliminating the need for an SSH tunnel.

• Adjust YAML config: Keep use_private_ip: False, but add scheduler and worker options to specify separate addresses using AWS instance metadata:

  dask_cloudprovider:
    region: eu-central-1
    bootstrap: True
    instance_type: t3.small
    n_workers: 1
    docker_image: daskdev/dask:2022.10.0-py3.10
    security: False
    debug: True
    use_private_ip: False
    scheduler_options:
      internal-address: "tcp://$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4):8786"
      external-address: "tcp://$(curl -s http://169.254.169.254/latest/meta-data/public-ipv4):8786"
    worker_options:
      scheduler-address: "tcp://$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4):8786"
  

  • The scheduler uses its private IP for internal worker connections (internal-address).
  • It exposes its public IP for your local client (external-address).
  • Workers are configured to connect directly to the scheduler's private IP.
• Update security group: Add an inbound rule allowing your local IP to access port 8786 (scheduler client port), in addition to the existing 8787 rule for the dashboard.
• Start and connect: Launch the cluster as usual. Your local client will connect to the scheduler's public IP (allowed by the new security rule), while workers use the private IP (allowed by the self-referential rule).

内容的提问来源于stack exchange，提问作者filpa