AKS集群中Neo4j证书续期后仍无效的自动配置问题

我在AKS集群中通过Helm Charts部署了MongoDB和Neo4j，搭配Node.js服务使用。服务和Neo4j的Bolt连接共用Let's Encrypt颁发的证书，集群2月部署后运行正常，但现在写入Neo4j时抛出Failed to connect to server错误，提示Socket responded with: CERT_HAS_EXPIRED，同时Neo4j Browser也无法连接数据库。

错误日志 #

Neo4jError: Failed to connect to server. Please ensure that your database is listening on the correct host and port and that you have compatible encryption settings both on Neo4j server and driver. Note that the default encryption setting has changed in Neo4j 4.0. Caused by: Server certificate is not trusted. If you trust the database you are connecting to, use TRUST_CUSTOM_CA_SIGNED_CERTIFICATES and add the signing certificate, or the server certificate, to the list of certificates trusted by this driver using `neo4j.driver(.., { trustedCertificates:['path/to/certificate.crt']}). This  is a security measure to protect against man-in-the-middle attacks. If you are just trying  Neo4j out and are not concerned about encryption, simply disable it using `encrypted="ENCRYPTION_OFF"` in the driver options. Socket responded with: CERT_HAS_EXPIRED
0|server  |     at new Neo4jError (/usr/app/node_modules/neo4j-driver-core/lib/error.js:77:16)
0|server  |     at newError (/usr/app/node_modules/neo4j-driver-core/lib/error.js:113:12)
0|server  |     at NodeChannel._handleConnectionError (/usr/app/node_modules/neo4j-driver-bolt-connection/lib/channel/node/node-channel.js:227:56)
0|server  |     at TLSSocket.<anonymous> (/usr/app/node_modules/neo4j-driver-bolt-connection/lib/channel/node/node-channel.js:69:17)
0|server  |     at Object.onceWrapper (node:events:641:28)
0|server  |     at TLSSocket.emit (node:events:527:28)
0|server  |     at TLSSocket.onConnectSecure (node:_tls_wrap:1564:10)
0|server  |     at TLSSocket.emit (node:events:527:28)
0|server  |     at TLSSocket._finishInit (node:_tls_wrap:945:8)
0|server  |     at ssl.onhandshakedone (node:_tls_wrap:726:12) {
0|server  |   constructor: [Function: Neo4jError] { isRetriable: [Function (anonymous)] },
0|server  |   code: 'ServiceUnavailable',
0|server  |   retriable: true
0|server  | }

证书状态 #

证书已在4月通过cert-manager续期，以下是证书资源信息：

Name:         tls-certificate
Namespace:    default
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: cluster
              meta.helm.sh/release-namespace: default
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2023-02-15T15:25:52Z
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:meta.helm.sh/release-name:
          f:meta.helm.sh/release-namespace:
        f:labels:
          .:
          f:app.kubernetes.io/managed-by:
      f:spec:
        .:
        f:dnsNames:
        f:issuerRef:
          .:
          f:kind:
          f:name:
        f:secretName:
    Manager:      helm
    Operation:    Update
    Time:         2023-02-15T15:25:52Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:revision:
    Manager:      cert-manager-certificates-issuing
    Operation:    Update
    Subresource:  status
    Time:         2023-04-16T14:27:13Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
          .:
          k:{"type":"Ready"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:observedGeneration:
            f:reason:
            f:status:
            f:type:
        f:notAfter:
        f:notBefore:
        f:renewalTime:
    Manager:         cert-manager-certificates-readiness
    Operation:       Update
    Subresource:     status
    Time:            2023-04-16T14:27:13Z
  Resource Version:  20023818
  UID:               9edc761c-9382-4597-8048-ec5e85d0871d
Spec:
  Dns Names:
    xxx.westeurope.cloudapp.azure.com
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       letsencrypt-issuer
  Secret Name:  tls-secret
Status:
  Conditions:
    Last Transition Time:  2023-02-15T15:26:48Z
    Message:               Certificate is up to date and has not expired
    Observed Generation:   1
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2023-07-15T13:27:11Z
  Not Before:              2023-04-16T13:27:12Z
  Renewal Time:            2023-06-15T13:27:11Z
  Revision:                2
Events:                    <none>

当前配置与临时解决方法 #

我发现未将证书加入受信任证书列表，仅引用了Certificate生成的Secret。Neo4j Chart的SSL配置如下：

ssl:
    # setting per "connector" matching neo4j config
    bolt:
      privateKey:
        secretName: tls-secret
        subPath: tls.key
      publicCertificate:
        secretName: tls-secret
        subPath: tls.crt
      trustedCerts:
        sources: []
      revokedCerts:
        sources: []

将Neo4j集群缩容至0副本再扩容至1副本后恢复正常，但希望实现证书更新后的自动生效，有以下疑问：

• 是否存在Neo4j续期后仍使用旧证书的可能？
• 在Neo4j Chart的Values中未找到TRUST_CUSTOM_CA_SIGNED_CERTIFICATES参数，该如何配置？
• path/to/certificate.crt具体指什么？
• 是否只需将证书名称tls-certificate添加到trustedCerts.sources数组中？

1. Neo4j续期后仍用旧证书的可能性 #

是的，这完全可能。Kubernetes中，Pod启动时会将Secret挂载到容器内，Secret更新后，容器内的文件不会自动刷新，除非Pod重启。你手动缩容再扩容的操作本质是重启了Pod，让它重新加载了新的Secret内容，所以恢复正常。

2. TRUST_CUSTOM_CA_SIGNED_CERTIFICATES相关配置 #

这个参数是Neo4j驱动的配置项，不是Helm Chart的Values参数：

• 对于你的Node.js服务，需要在初始化Neo4j Driver时指定该配置，示例代码如下：

const neo4j = require('neo4j-driver');
const driver = neo4j.driver(
  'bolt+s://your-neo4j-host:7687',
  neo4j.auth.basic('username', 'password'),
  {
    encrypted: 'ENCRYPTION_ON',
    trust: 'TRUST_CUSTOM_CA_SIGNED_CERTIFICATES',
    trustedCertificates: ['/path/to/ca-cert.crt'] // 这里是容器内CA证书的路径
  }
);

• 如果是Neo4j服务器端需要信任CA证书，才需要在Helm Chart的trustedCerts中配置。

3. path/to/certificate.crt的含义 #

这里指的是证书文件在运行Node.js服务的容器内的挂载路径。你需要将Let's Encrypt的CA证书（或者你的服务器证书）挂载到Node.js Pod中，然后填写对应的路径。

对于AKS + cert-manager场景，Let's Encrypt的根CA证书通常可以通过以下方式获取：

• 直接在Node.js容器内安装系统信任的根CA（比如Debian/Ubuntu下安装ca-certificates包，Let's Encrypt的CA默认已包含）
• 或者将cert-manager生成的Secret中的tls.crt（服务器证书）挂载到Node.js容器，然后指定该路径。

4. 实现证书自动更新的方案 #

方案一：利用Pod的滚动更新机制

修改Neo4j的Helm Chart配置，添加Secret的哈希值作为Pod的注解，当Secret更新时，Pod会自动重启：

# 在Neo4j Chart的values.yaml中添加
statefulset:
  annotations:
    checksum/secret: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }}

注意：Neo4j是StatefulSet，滚动重启需要确保数据一致性，该方式会逐个重启Pod。

方案二：使用cert-manager的Reloader

部署cert-manager/reloader工具，它会监控Secret的变化，自动触发Pod的重启：

kubectl apply -f https://github.com/cert-manager/reloader/releases/latest/download/reloader.yaml

然后给Neo4j的StatefulSet添加注解，让Reloader监控目标Secret：

apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    secret.reloader.stakater.com/reload: "tls-secret"

方案三：配置Neo4j服务器的证书自动重载

Neo4j本身支持在运行时重载证书，你可以通过Helm Chart开启该功能：

ssl:
  bolt:
    # 原有配置...
    refreshInterval: 60s # 每60秒检查证书是否更新

这样Neo4j会定期检查证书文件的变化，自动加载新证书，无需重启Pod。

5. trustedCerts.sources的配置 #

如果需要让Neo4j服务器信任其他CA证书，才需要配置trustedCerts.sources，它的取值是Secret的名称，而不是Certificate资源的名称。比如你要添加Let's Encrypt的CA证书，需要先创建包含CA证书的Secret，然后配置：

trustedCerts:
  sources:
    - secretName: letsencrypt-ca-secret
      subPath: ca.crt

但在你的场景中，Neo4j使用的是自己的服务器证书，驱动端需要配置信任，服务器端不需要额外添加trustedCerts，除非有其他需要信任的证书。

内容的提问来源于stack exchange，提问作者Vincenzo