Azure Kubernetes集群部署中如何增加Dask Worker的数量

问题根因 #

Kubernetes中Deployment的replicas字段控制的是Pod整体的副本数量，你当前的配置将Flask、Dask Scheduler、Dask Worker三个业务容器都定义在同一个Deployment的Pod模板中，每新增1个Pod副本就会同时启动3个容器，自然无法单独调整Dask Worker的实例数。

改造方案 #

第一步：拆分独立Deployment #

将原来的1个Deployment拆分为3个独立的Deployment，各自管理对应的容器，同时为Dask Scheduler创建ClusterIP服务，保证Flask和Worker可以正常访问调度器：

• Flask服务Deployment：固定1副本，负责处理接口请求
• Dask Scheduler Deployment：固定1副本，负责任务调度
• Dask Worker Deployment：可自由调整replicas值为你需要的N，负责执行计算任务

示例配置文件

1. Dask Scheduler部署文件

# dask-scheduler.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dask-scheduler
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dask-scheduler
  template:
    metadata:
      labels:
        app: dask-scheduler
    spec:
      containers:
        - name: cont-scheduler
          image: xxx.azurecr.io/myimage:latest
          command: ["dask-scheduler"]
          ports:
            - containerPort: 8786
            - containerPort: 8787
---
# Scheduler服务，供Flask和Worker访问
apiVersion: v1
kind: Service
metadata:
  name: scheduler
  namespace: default
spec:
  selector:
    app: dask-scheduler
  ports:
    - name: tcp-scheduler
      port: 8786
      targetPort: 8786
    - name: http-dashboard
      port: 8787
      targetPort: 8787
  type: ClusterIP

2. Flask服务部署文件

# flask-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: flask-server
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flask-server
  template:
    metadata:
      labels:
        app: flask-server
    spec:
      containers:
        - name: cont-flask
          image: xxx.azurecr.io/myimage:latest
          command: ["gunicorn"]
          args: ["-t", "7200", "-b", "0.0.0.0:5000", "--reload", "app.front.app:app"]
          ports:
            - containerPort: 5000
          volumeMounts:
            - name: volcode
              mountPath: /code
      volumes:
        - name: volcode
          azureFile:
            shareName: py-code
            readOnly: false
---
# 可选：需要对外暴露Flask服务可以加Service，修改type为LoadBalancer即可自动分配Azure公网IP
apiVersion: v1
kind: Service
metadata:
  name: flask-service
  namespace: default
spec:
  selector:
    app: flask-server
  ports:
    - port: 5000
      targetPort: 5000
  type: ClusterIP

3. Dask Worker部署文件

# dask-worker.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dask-worker
  namespace: default
spec:
  # 此处直接填写你需要的Worker实例数N即可
  replicas: 3
  selector:
    matchLabels:
      app: dask-worker
  template:
    metadata:
      labels:
        app: dask-worker
    spec:
      containers:
        - name: cont-worker
          image: xxx.azurecr.io/myimage:latest
          workingDir: /code
          command: ["dask-worker"]
          args: ["scheduler:8786"]
          volumeMounts:
            - name: volcode
              mountPath: /code
      volumes:
        - name: volcode
          azureFile:
            shareName: py-code
            readOnly: false

第二步：部署&调整Worker数量 #

1. 删除原有旧Deployment：
   kubectl delete deployment img-python
2. 依次部署三个新资源：
   kubectl apply -f dask-scheduler.yaml
kubectl apply -f flask-deployment.yaml
kubectl apply -f dask-worker.yaml
3. 后续调整Worker数量直接操作对应的Deployment即可：
   kubectl scale deployment dask-worker --replicas=N
   将N替换为你需要的实例数即可完成单独扩容。

内容的提问来源于stack exchange，提问作者ps0604