跨K8s集群Jenkins代理连接问题求助及方案探讨

Hey there, let’s break down your cross-cluster Jenkins agent issue—this kind of network mismatch between HTTP-exposed services and TCP-dependent agents is super common when dealing with multi-cluster setups, so I’ve got some actionable guidance for both your proposed solutions.

Quick Recap of Your Setup #

• Jenkins master lives in the TC cluster; you can spin up agents in the FS cluster, but they can’t establish a working JNLP connection back to the master
• TC’s external JNLP service uses HTTP (jenkins-jnlp.tc.com:80), which throws an error about serial vs binary data when FS agents try to connect
• Internal TC traffic to jenkins-jnlp.svc.cluster.local:50000 works perfectly, so the core JNLP functionality is solid

Troubleshooting the WebSocket Approach #

First, let’s tackle the native Jenkins WebSocket option—since it’s a newer feature, the docs are sparse, but there are a few easy-to-miss checks that might fix your "agent shows connected but master marks it offline" issue:

• Double-check your agent image: Make sure you’re using an jenkins/inbound-agent image tagged with a version that supports WebSockets (any image built after Feb 2020 should, but some older custom images might lack the necessary dependencies). Pull the latest official image and test if that resolves the disconnect.
• Master-side config tweaks:
  • Head to Manage Jenkins > Configure Global Security > Agents—ensure "Enable WebSocket" is checked, and that the TCP port for inbound agents is set to either "Random" or a fixed port (even if you’re using WebSockets, this setting needs to be enabled).
  • Verify your Jenkins URL (in Manage Jenkins > Configure System) is fully accessible from the FS cluster. WebSocket handshakes rely on the agent being able to reach this URL, so if there’s a DNS or network policy block here, the connection will silently fail after initial handshake.
• Dig into agent logs: Even if the agent pod says it’s connected, check the logs for WebSocket upgrade errors. Common culprits include TLS certificate mismatches (if your Jenkins uses HTTPS) or missing CORS headers that block the upgrade request.
• Network policies: Make sure both clusters allow traffic on Jenkins’ HTTP/HTTPS port—WebSocket uses an HTTP upgrade request, so if network policies block that port, the connection won’t switch to WebSocket properly.

Finishing Your HTTP-to-TCP Relay Sidecar #

Since you’re already halfway done building the relay container, here’s how to wrap up the TCP-to-HTTP upstream conversion:

• Bidirectional stream handling: Your relay needs to manage two persistent streams:
  1. Downstream: Accept TCP connections from the FS agent, wrap the binary JNLP data in HTTP payloads (use chunked encoding to handle continuous data), and forward to TC’s external JNLP URL.
  2. Upstream: Listen for HTTP responses from TC’s JNLP service, unwrap the binary data, and send it back over the TCP connection to the FS agent.
• Simplify the endpoint: Create a single TCP endpoint on your relay (e.g., port 50000) that the FS agent connects to. The relay handles all the HTTP wrapping/unwrapping behind the scenes, so the agent just thinks it’s talking to a regular JNLP TCP service.
• Sidecar deployment tips:
  • When deploying the relay as a sidecar, configure the Jenkins agent to connect to localhost:50000 (your relay’s port) instead of the TC external URL.
  • Add liveness probes to the relay container (e.g., a simple TCP check on your relay port) to ensure it’s ready before the agent starts trying to connect.
  • Test with a standalone TCP client first (like telnet localhost 50000) to verify the bidirectional flow works before integrating with Jenkins.

Quick Prototype Workaround #

If you want to validate the relay approach without finishing your custom code immediately, you can use existing tools to test:

• Use socat to create a basic TCP-to-HTTP bridge (it’s not perfect for persistent JNLP streams, but it can help you confirm the network path is viable).
• Check out haproxy with the stream module—you can configure it to handle HTTP wrapping for TCP connections, which might save you some development time.

内容的提问来源于stack exchange，提问作者JumpingJack