使用JupyterHub认证时Dask Gateway连接一段时间后中断求助

我搭建了多用户DaskHub与JupyterHub环境，采用Dask Gateway服务器，但运行一段时间后，与Dask Gateway的连接会出现失败情况。以下是相关错误日志、连接代码及Helm Chart配置信息，恳请协助排查连接中断问题。

错误日志 #

[I 2023-04-24 09:51:14.477 DaskGateway] Creating cluster jhub.6b1db15befa34f6cb9e3abaa38099f1b for user test
[I 2023-04-24 09:51:14.493 DaskGateway] 201 POST /api/v1/clusters/ 30.877ms
[I 2023-04-24 09:51:34.513 DaskGateway] 200 GET /api/v1/clusters/jhub.6b1db15befa34f6cb9e3abaa38099f1b?wait 20001.169ms
[I 2023-04-24 09:51:55.021 DaskGateway] 200 GET /api/v1/clusters/jhub.6b1db15befa34f6cb9e3abaa38099f1b?wait 20001.478ms
[W 2023-04-24 09:52:05.562 DaskGateway] 404 GET / 0.587ms
[I 2023-04-24 09:52:15.530 DaskGateway] 200 GET /api/v1/clusters/jhub.6b1db15befa34f6cb9e3abaa38099f1b?wait 20001.511ms
[I 2023-04-24 09:52:36.040 DaskGateway] 200 GET /api/v1/clusters/jhub.6b1db15befa34f6cb9e3abaa38099f1b?wait 20003.526ms
[I 2023-04-24 09:52:46.935 DaskGateway] 200 GET /api/v1/clusters/jhub.6b1db15befa34f6cb9e3abaa38099f1b?wait 10387.750ms
[W 2023-04-24 09:53:05.562 DaskGateway] 404 GET / 0.659ms
[I 2023-04-24 09:54:03.370 DaskGateway] Cluster jhub.88d0cbe73d7c4a3da1c1b9926a445a5f heartbeat [count: 0, n_active: 0, n_closing: 0, n_closed: 0]
[I 2023-04-24 09:54:03.387 DaskGateway] 200 POST /api/v1/clusters/jhub.88d0cbe73d7c4a3da1c1b9926a445a5f/heartbeat 17.781ms
[W 2023-04-24 09:54:05.562 DaskGateway] 404 GET / 0.505ms
[W 2023-04-24 09:55:05.561 DaskGateway] 404 GET / 0.515ms
[W 2023-04-24 09:56:05.562 DaskGateway] 404 GET / 0.732ms
[W 2023-04-24 09:57:05.561 DaskGateway] 404 GET / 0.526ms

使用的连接代码 #

from dask_gateway import Gateway
gateway = Gateway(
    address="http://traefik-dask-gateway/services/dask-gateway/",
    public_address="https://jupyter-hub01.devops.h2.theagilehub.net/services/dask-gateway/",
    auth="jupyterhub")
gateway.list_clusters()

Helm Chart配置值 #

jupyterhub:
  proxy:
    secretToken: "<token1>"
  hub:
    services:
      dask-gateway:
        apiToken: "<token2>"
    networkPolicy:
      enabled: false
  ingress:
    enabled: true
    ingressClassName: pomerium
    tls:
      - secretName: jupyter-hub-tls
        hosts:
          - jupyter-hub01.devops.h2.theagilehub.net
    hosts:
      - jupyter-hub01.devops.h2.theagilehub.net
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      ingress.pomerium.io/allow_websockets: "true"
      ingress.pomerium.io/pass_identity_headers: 'true'
      ingress.pomerium.io/preserve_host_header: 'true'
      ingress.pomerium.io/policy: |
        [{
          "allow": {
            "or": [
              {"claim/cognito:groups": "eu-west-1_ryHLiyRp9_shell-dev"}
            ]
          }
          }]
dask-gateway:
  gateway:
    auth:
      type: jupyterhub
      jupyterhub:
        apiToken: "<token2>"

排查建议 #

• 处理无效404请求：日志中频繁出现404 GET /，这是访问Dask Gateway根路径的无效请求，需检查是否有监控探针、Ingress路由或外部服务错误配置了根路径访问，避免日志干扰同时排除潜在路由问题。
• 验证心跳机制：日志显示集群心跳count: 0, n_active: 0，说明客户端未持续发送心跳。检查连接代码是否有保持心跳的逻辑，或调整Dask Gateway Helm配置中的gateway.cluster.heartbeatTimeout等参数延长超时时间。
• 测试内部地址连通性：在Jupyter Pod内执行curl http://traefik-dask-gateway/services/dask-gateway/api/v1/clusters，验证内部地址是否能正常访问Dask Gateway API，排查Traefik路由是否正确。
• 检查Pomerium超时配置：Dask Gateway依赖WebSocket通信，确认Pomerium的超时参数（如ingress.pomerium.io/timeout、ingress.pomerium.io/idle_timeout）是否足够长，避免因超时断开连接。
• 确认Token一致性：验证JupyterHub与Dask Gateway配置中的<token2>是否一致且有效，可在JupyterHub Hub Pod内用该Token调用Dask Gateway API测试权限。
• 查看Pod资源状态：检查Dask Worker、Scheduler Pod的日志和资源使用情况，确认是否因内存/CPU耗尽导致Pod被Kubernetes终止，引发连接中断。
• 检查网络策略：确认Dask Gateway的NetworkPolicy是否允许Jupyter Pod与Gateway服务通信，即使JupyterHub的NetworkPolicy已关闭。

内容的提问来源于stack exchange，提问作者Jyoti Singh