Kubernetes集群中两个服务间SSL连接异常问题排查

问题背景 #

在Kubernetes同一namespace下尝试建立两个服务间的SSL连接，所有Pod已注入istio-proxy sidecar容器，且已创建PeerAuthentication和DestinationRule资源，但HTTPS调用失败。

现有配置 #

DestinationRule配置

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
spec:
  host: brand-tool-ui-clone
  trafficPolicy:
    tls:
      mode: SIMPLE
      privateKey: /etc/istio/private/mykey.key
      serverCertificate: /etc/istio/certs/mycert.crt

PeerAuthentication配置

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  creationTimestamp: "2023-01-27T12:48:19Z"
  generation: 2
  name: default
  namespace: https-poc
  resourceVersion: "128521847"
  uid: 9035144f-4ae5-4b2e-89af-c14fc081b96a
spec:
  mtls:
    mode: PERMISSIVE

初始报错信息 #

执行命令 curl -k -v https://<hostname>/ecv-status 后出现如下错误：

$ curl -k  -v https://<hostname>/ecv-status
* Expire in 0 ms for 6 (transfer 0x5597ba492680)
* Expire in 1 ms for 1 (transfer 0x5597ba492680)
* Expire in 0 ms for 1 (transfer 0x5597ba492680)
* Expire in 1 ms for 1 (transfer 0x5597ba492680)
* Expire in 0 ms for 1 (transfer 0x5597ba492680)
* Expire in 0 ms for 1 (transfer 0x5597ba492680)
* Expire in 1 ms for 1 (transfer 0x5597ba492680)
* Expire in 0 ms for 1 (transfer 0x5597ba492680)
* Expire in 0 ms for 1 (transfer 0x5597ba492680)
* Expire in 0 ms for 1 (transfer 0x5597ba492680)
*   Trying 1.199.124.123...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x5597ba492680)
* Connected to <host> (1.199.124.123) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

更新配置后的新报错 #

调整配置后，出现"证书要求"告警，curl日志如下：

*   Trying 1.199.124.123:443...
* Connected to <host> (1.199.124.123) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: [NONE]
*  start date: Jan 30 11:30:37 2023 GMT
*  expire date: Jan 31 11:32:37 2023 GMT
*  issuer: O=cluster.local
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55f415ab6b80)
> GET /ecv-status HTTP/2
> Host: <host>
> user-agent: curl/7.79.1
> accept: */*
>
* TLSv1.3 (IN), TLS alert, unknown (628):
* OpenSSL SSL_read: error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required, errno 0
* Failed receiving HTTP2 data
* OpenSSL SSL_write: SSL_ERROR_ZERO_RETURN, errno 0
* Failed sending HTTP2 data
* Connection #0 to host  left intact
curl: (56) OpenSSL SSL_read: error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required, errno 0

问题分析与解决 #

1. 初始报错原因 #

wrong version number错误说明客户端向端口发送了HTTPS请求，但该端口实际提供的是HTTP服务。在Istio场景下，核心问题是DestinationRule的TLS配置模式错误：使用SIMPLE模式并手动指定证书，不符合Istio服务间mTLS的规范。

2. 更新后报错原因 #

certificate required错误是因为服务端开启了双向TLS认证，而curl请求未携带客户端证书。结合PeerAuthentication的PERMISSIVE模式，实际是DestinationRule的配置触发了服务端强制要求客户端证书验证。

具体解决步骤 #

步骤1：修正DestinationRule配置

Istio服务间的mTLS无需手动指定证书，应使用Istio自动生成和管理的证书，将TLS模式改为ISTIO_MUTUAL：

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: brand-tool-ui-clone
  namespace: https-poc
spec:
  host: brand-tool-ui-clone.https-poc.svc.cluster.local
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

步骤2：调整PeerAuthentication（可选）

如果需要强制所有服务间通信使用mTLS，可将模式从PERMISSIVE改为STRICT；保持PERMISSIVE则兼容HTTP和HTTPS请求，服务间通信会自动加密：

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: https-poc
spec:
  mtls:
    mode: STRICT

步骤3：正确调用服务的方式

• Pod内调用：注入sidecar的Pod内直接使用服务名+HTTP即可，Istio会自动加密通信：

  curl http://brand-tool-ui-clone/ecv-status
  

• 外部手动测试HTTPS：需要携带Istio的客户端证书，先从sidecar容器导出证书：

  kubectl exec -it <目标Pod名称> -c istio-proxy -- cat /etc/certs/cert-chain.pem > client-cert.pem
  kubectl exec -it <目标Pod名称> -c istio-proxy -- cat /etc/certs/key.pem > client-key.pem
  kubectl exec -it <目标Pod名称> -c istio-proxy -- cat /etc/certs/root-cert.pem > root-cert.pem
  

  再用curl携带证书调用：

  curl -v --cert client-cert.pem --key client-key.pem --cacert root-cert.pem https://<hostname>/ecv-status
  

步骤4：验证配置生效

检查Istio配置是否存在错误：

istioctl analyze

查看Pod的sidecar destination配置，确认TLS策略已应用：

istioctl pc destination <Pod名称> -c istio-proxy

内容的提问来源于stack exchange，提问作者Rahul Gogyani