Selenium 4.6.0部署K8s集群遇HTTP 504及连接拒绝问题排查
问题背景
本地公司VPN环境下Selenium 4.6.0运行正常,但部署到启用isolateComponents: true的公司Kubernetes集群后,测试用例出现HTTP 504错误。Chrome节点日志无明显业务异常,但Router节点日志显示连接拒绝。已通过自定义Dockerfile为Chrome节点和Router镜像添加SSL证书以实现安全连接,目前无法定位连接拒绝原因,询问是否需要添加代理配置及配置位置。
Chrome节点日志
Starting ChromeDriver 107.0.5304.62 (1eec40d3a5764881c92085aaee66d25075c159aa-refs/branch-heads/5304@{#942}) on port 16517 Only local connections are allowed. Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe. ChromeDriver was started successfully. [1670230314.770][SEVERE]: bind() failed: Cannot assign requested address (99) 08:51:54.972 INFO [LocalNode.newSession] - Session created by the Node. Id: 28c938ed7495ed7987bd6f37987e6f96, Caps: Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 107.0.5304.87, chrome: {chromedriverVersion: 107.0.5304.62 (1eec40d3a576..., userDataDir: /tmp/.com.google.Chrome.QNhMmk}, goog:chromeOptions: {debuggerAddress: localhost:43699}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: LINUX, proxy: Proxy(), se:cdp: http://localhost:43699, se:cdpVersion: 107.0.5304.87, se:vncEnabled: true, se:vncLocalAddress: ws://192.168.161.159:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:virtualAuthenticators: true} 08:52:02.203 INFO [ProxyNodeWebsockets.createWsEndPoint] - Establishing connection to ws://192.168.161.159:7900
Router节点日志
❯ kubectl logs selenium-router-7458b5ccbd-nb48c 2022-12-05 08:45:30,097 INFO Included extra file "/etc/supervisor/conf.d/selenium-grid-router.conf" during parsing 2022-12-05 08:45:30,100 INFO RPC interface 'supervisor' initialized 2022-12-05 08:45:30,100 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2022-12-05 08:45:30,100 INFO supervisord started with pid 7 2022-12-05 08:45:31,103 INFO spawned: 'selenium-grid-router' with pid 9 Starting Selenium Grid Router... 2022-12-05 08:45:31,106 INFO success: selenium-grid-router entered RUNNING state, process has stayed up for > than 0 seconds (startsecs) Tracing is disabled 08:45:31.432 INFO [LoggingOptions.configureLogEncoding] - Using the system default encoding 08:45:31.437 INFO [OpenTelemetryTracer.createTracer] - Using OpenTelemetry for tracing 08:45:32.323 INFO [RouterServer.execute] - Started Selenium Router 4.6.0 (revision 79f1c02ae20): http://192.168.175.198:4444 08:49:55.051 WARN [SimpleDataFetcherExceptionHandler.logException] - Exception while fetching data (/sessionsInfo) : java.net.ConnectException: Connection refused java.io.UncheckedIOException: java.net.ConnectException: Connection refused at org.openqa.selenium.remote.http.jdk.JdkHttpClient.execute(JdkHttpClient.java:284) at org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55)
自定义Dockerfile
Chrome节点镜像
FROM selenium/node-chrome:107.0-chromedriver-107.0 USER root ADD ssl.crt /usr/local/share/ca-certificates/ssl.crt RUN update-ca-certificates USER seluser
Router节点镜像
FROM selenium/router:4.6.0 USER root ADD ssl.crt /usr/local/share/ca-certificates/ssl.crt RUN update-ca-certificates USER seluser
分析与解答
是否需要代理配置?
需要。公司Kubernetes集群通常处于受限网络环境,结合isolateComponents: true的网络隔离设置,无论是Selenium Grid组件间的内部通信,还是测试用例通过Chrome访问目标网站,都可能需要代理才能正常连通。
代理配置位置与方式
1. Selenium Grid组件间通信的JVM代理
Router与Node之间的通信如果需要走公司内部代理,需为组件添加JVM级代理参数:
- 在Dockerfile中添加环境变量:
# 适用于Chrome Node和Router镜像 ENV http_proxy=http://your-proxy-host:port ENV https_proxy=http://your-proxy-host:port ENV no_proxy=localhost,127.0.0.1,cluster.local # 排除集群内部地址 - 或在K8s部署文件的
command字段中追加JVM参数:command: ["java"] args: [ "-Dhttp.proxyHost=your-proxy-host", "-Dhttp.proxyPort=port", "-Dhttps.proxyHost=your-proxy-host", "-Dhttps.proxyPort=port", "-DnoProxy=localhost,127.0.0.1,cluster.local", "-jar", "/opt/selenium/selenium-server.jar", "node" # Router则改为"router" ]
2. Chrome浏览器的代理配置
测试用例访问目标网站时,Chrome需使用公司代理,可通过两种方式配置:
- 测试代码中配置ChromeOptions:
ChromeOptions options = new ChromeOptions(); options.addArguments("--proxy-server=http://your-proxy-host:port"); options.addArguments("--no-proxy-server=localhost,127.0.0.1"); // 按需添加 WebDriver driver = new RemoteWebDriver(new URL("http://router-url:4444"), options); - Node镜像默认配置:在Chrome Node的Dockerfile或K8s部署中设置环境变量,为所有启动的Chrome实例添加代理:
ENV SE_NODE_CHROME_OPTIONS="--proxy-server=http://your-proxy-host:port"
3. Docker镜像构建时的代理(可选)
如果构建镜像过程中需要访问外部资源(如update-ca-certificates依赖外部源),需在Dockerfile中添加构建参数:
ARG http_proxy=http://your-proxy-host:port ARG https_proxy=http://your-proxy-host:port
额外排查点
- 检查K8s网络策略:
isolateComponents: true可能限制了Router与Node之间的端口通信,需确保Router能访问Node的服务端口(默认5555)及ChromeDriver动态端口。 - 修复Chrome Node的bind错误:日志中
bind() failed: Cannot assign requested address (99)说明节点尝试绑定的地址不可用,需在Node的K8s部署中设置SE_NODE_HOST=0.0.0.0或Pod的IP,允许绑定到所有可用地址。
内容的提问来源于stack exchange,提问作者Susanta Adhikary
相关产品推荐
相关产品推荐

