Kubernetes中MongoDB企业版Operator副本集SSL/TLS证书读取失败求助

环境与配置 #

使用MongoDB企业版Operator，已部署MongoDB OpsManager，通过CRD创建3节点副本集，CRD配置如下：

apiVersion: mongodb.com/v1
kind: MongoDB
metadata:
  name: mongo-dev
  namespace: mongodb-workspace
spec:
  credentials: my-org-apikey
  members: 3
  opsManager:
    configMapRef:
      name: my-org-id
  type: ReplicaSet
   podSpec: 
     cpu: '4'
     memory: '6G'
  version: 5.0.18-ent
  security:
    tls:
      enabled: true
      ca: custom-ca
    authentication:
      enabled: true
      modes: ["SCRAM"]

自签名证书创建步骤 #

1. 生成CA私钥：

   openssl genrsa -out ca.key 2048
   

2. 生成CA证书：

   openssl req -x509 -new -nodes -key ca.key -subj "/CN=*.svc.cluster.local" -days 3650 -reqexts v3_req -extensions v3_ca -out ca.crt
   

3. 创建CA密钥对Secret：

   oc create secret tls ca-key-pair --cert=ca.crt --key=ca.key
   

4. 部署cert-manager：

   oc apply -f cert-manager.yaml
   

5. 部署证书颁发者：

   oc apply -f issuers.yaml
   

6. 部署证书资源：

   oc apply -f certificates.yaml
   

7-9. 导出各节点证书密钥对并合并为PEM文件：

oc get secret mongo-dev-0 -o jsonpath='{.data.tls.crt}{.data.tls.key}' | base64 --decode > mongo-dev-0-pem
oc get secret mongo-dev-1 -o jsonpath='{.data.tls.crt}{.data.tls.key}' | base64 --decode > mongo-dev-1-pem
oc get secret mongo-dev-2 -o jsonpath='{.data.tls.crt}{.data.tls.key}' | base64 --decode > mongo-dev-2-pem

10. 创建包含节点证书的通用Secret：

    oc create secret generic mongo-dev-cert --from-file=mongo-dev-0-pem --from-file=mongo-dev-1-pem --from-file=mongo-dev-2-pem
    

11. 准备CA PEM文件：

    cat ca.crt > ca-pem
    

12. 创建CA配置映射：

    oc create configmap custom-ca --from-file=ca-pem
    

13. 应用MongoDB副本集CRD：

    oc apply -f mongodb-rs.yaml
    

错误日志 #

Agent日志 #

Failed to apply action. Result = :
[15:07:48.267] Error starting mongod : [15:07:48.267]
Error running start command.
cmd=[Args=[/var/lib/mongodb-mms-automation/mongodb-linux-x86_64-5.0.18-ent/bin/mongod
-f /data/automation-mongod.conf]], stip=[args={"net":{"bindIp":"0.0.0.0","port":27017,"tls":{"CAFile":"/mongodb-automation/tls/ca/ca-pem","allowConnectionsWithoutCertificates":true,"certificateKeyFile":"/mongodb-automation/tls/","mode":"preferTLS"}},"replication":{"replSetName":"mongodb-dev"},"storage":{"dbPath":"/data"},"systemLog":{"destination":"file","path":"/var/log/mongodb-mms-automation/mongodb.log"}}[],confPath=/data/automation-mongod.conf,version=5.0.18-ent-796abe56bfdbca6968ff570311bf72d93632825b(enterprise),isKmipRotateMasterKey=false,useOldConfFile=false]

Pod日志 #

sh-4.4$ cat /var/log/mongodb-mms-automation/mongodb.log
{"t":{"$date":"2023-09-06T14:27:24.939+00:00"},"s":"E",
"c":"NETWORK", "id":23248, "ctx":"-","msg":"Cannot read certificate
file","attr":{"keyFile":"/mongodb-automation/tls/","error":"error:0909006C:PEM
routines:get_name:no start line"}}
{"t":{"$date":"2023-09-06T14:27:24.941+00:00"},"s":"F",
"c":"CONTROL", "id":20574, "ctx":"-","msg":"Error during global
initialization","attr":{"error":{"code":140,"codeName":"InvalidSSLConfiguration","errmsg":"Can
not set up PEM key file."}}}

问题分析 #

从日志可见，mongod配置中certificateKeyFile被设置为目录/mongodb-automation/tls/，而非具体的PEM证书文件，导致MongoDB无法读取证书内容，触发PEM解析错误。核心原因是MongoDB CRD中未指定节点证书的Secret引用，Operator无法正确挂载证书文件并配置mongod参数。

解决方案 #

1. 修改MongoDB CRD的TLS配置，添加certificateSecretRef字段指向之前创建的节点证书Secret：

   apiVersion: mongodb.com/v1
   kind: MongoDB
   metadata:
     name: mongo-dev
     namespace: mongodb-workspace
   spec:
     # ... 保留原有其他配置 ...
     security:
       tls:
         enabled: true
         ca: custom-ca
         certificateSecretRef:
           name: mongo-dev-cert # 指向包含节点证书的Secret
       authentication:
         enabled: true
         modes: ["SCRAM"]
   

2. 重新应用CRD配置：

   oc apply -f mongodb-rs.yaml
   

3. 验证Pod状态：

   oc get pods -n mongodb-workspace
   

4. 查看修复后的日志：

   oc logs -f mongo-dev-0 -c mongod -n mongodb-workspace
   

内容的提问来源于stack exchange，提问作者Mugi Coder