启用FIPS模式时Cassandra 4启动崩溃（BouncyCastle相关）

问题背景 #

• 应用支持FIPS/非FIPS双模式：非FIPS模式使用默认Java安全提供者+JKS格式密钥库/信任库；FIPS模式采用BCFKS格式密钥库/信任库，已完成全系统配置变更（包括java.security提供者、cassandra.yaml等）
• 从Cassandra 3升级至Cassandra 4.0.4后，非FIPS模式可正常运行，但FIPS模式启动崩溃，核心错误为：BCFKS KeyStore corrupted: MAC calculation failed
• 根因推测：Cassandra 4依赖Netty库生成SSL上下文，调用过程中密钥库/信任库密码未正确传递，导致MAC验证失败

崩溃日志 #

ERROR [main] 2022-08-19T14:15:58,533 CassandraDaemon.java:911 - Exception encountered during startup
org.apache.cassandra.exceptions.ConfigurationException: Failed to initialize SSL
        at org.apache.cassandra.config.DatabaseDescriptor.applySslContext(DatabaseDescriptor.java:1014) ~[apache-cassandra-4.0.4.jar:4.0.4]
        at org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:364) ~[apache-cassandra-4.0.4.jar:4.0.4]
        at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:178) ~[apache-cassandra-4.0.4.jar:4.0.4]
        at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:162) ~[apache-cassandra-4.0.4.jar:4.0.4]
        at org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:818) [apache-cassandra-4.0.4.jar:4.0.4]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:754) [apache-cassandra-4.0.4.jar:4.0.4]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889) [apache-cassandra-4.0.4.jar:4.0.4]
Caused by: java.io.IOException: Failed to create SSL context using Native transport
        at org.apache.cassandra.security.SSLFactory.validateSslContext(SSLFactory.java:546) ~[apache-cassandra-4.0.4.jar:4.0.4]
        at org.apache.cassandra.config.DatabaseDescriptor.applySslContext(DatabaseDescriptor.java:1009) ~[apache-cassandra-4.0.4.jar:4.0.4]
        ... 6 more
Caused by: javax.net.ssl.SSLException: unable to setup trustmanager
        at io.netty.handler.ssl.ReferenceCountedOpenSslServerContext.newSessionContext(ReferenceCountedOpenSslServerContext.java:183) ~[netty-all-4.1.58.Final.jar:4.1.58.Final]
        at io.netty.handler.ssl.OpenSslServerContext.<init>(OpenSslServerContext.java:353) ~[netty-all-4.1.58.Final.jar:4.1.58.Final]
        at io.netty.handler.ssl.OpenSslServerContext.<init>(OpenSslServerContext.java:334) ~[netty-all-4.1.58.Final.jar:4.1.58.Final]
        at io.netty.handler.ssl.SslContext.newServerContextInternal(SslContext.java:471) ~[netty-all-4.1.58.Final.jar:4.1.58.Final]
        at io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:571) ~[netty-all-4.1.58.Final.jar:4.1.58.Final]
        at org.apache.cassandra.security.SSLFactory.createNettySslContext(SSLFactory.java:339) ~[apache-cassandra-4.0.4.jar:4.0.4]
        at org.apache.cassandra.security.SSLFactory.validateSslContext(SSLFactory.java:496) ~[apache-cassandra-4.0.4.jar:4.0.4]
        at org.apache.cassandra.config.DatabaseDescriptor.applySslContext(DatabaseDescriptor.java:1009) ~[apache-cassandra-4.0.4.jar:4.0.4]
        ... 6 more
Caused by: java.security.KeyStoreException: Failed to load default?trust store
        at org.bouncycastle.jsse.provider.ProvTrustManagerFactorySpi.engineInit(ProvTrustManagerFactorySpi.java:172) ~[bctls-fips-debug-1.0.11.jar:1.0.11]
        at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:250) ~[?:1.8.0_321]
        at io.netty.handler.ssl.ReferenceCountedOpenSslServerContext.newSessionContext(ReferenceCountedOpenSslServerContext.java:147) ~[netty-all-4.1.58.Final.jar:4.1.58.Final]
        at io.netty.handler.ssl.OpenSslServerContext.<init>(OpenSslServerContext.java:353) ~[netty-all-4.1.58.Final.jar:4.1.58.Final]
        at io.netty.handler.ssl.OpenSslServerContext.<init>(OpenSslServerContext.java:334) ~[netty-all-4.1.58.Final.jar:4.1.58.Final]
        at io.netty.handler.ssl.SslContext.newServerContextInternal(SslContext.java:471) ~[netty-all-4.1.58.Final.jar:4.1.58.Final]
        at io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:571) ~[netty-all-4.1.58.Final.jar:4.1.58.Final]
        at org.apache.cassandra.security.SSLFactory.createNettySslContext(SSLFactory.java:339) ~[apache-cassandra-4.0.4.jar:4.0.4]
        at org.apache.cassandra.security.SSLFactory.validateSslContext(SSLFactory.java:496) ~[apache-cassandra-4.0.4.jar:4.0.4]
        at org.apache.cassandra.config.DatabaseDescriptor.applySslContext(DatabaseDescriptor.java:1009) ~[apache-cassandra-4.0.4.jar:4.0.4]
        ... 6 more
Caused by: java.io.IOException: BCFKS KeyStore corrupted: MAC calculation failed.
        at org.bouncycastle.jcajce.provider.ProvBCFKS$BCFIPSKeyStoreSpi.verifyMac(ProvBCFKS.java:945) ~[bc-fips-1.0.2.1.jar:1.0.2.1]
        at org.bouncycastle.jcajce.provider.ProvBCFKS$BCFIPSKeyStoreSpi.engineLoad(ProvBCFKS.java:1236) ~[bc-fips-1.0.2.1.jar:1.0.2.1]
        at java.security.KeyStore.load(KeyStore.java:1445) ~[?:1.8.0_321]
        at org.bouncycastle.jsse.provider.ProvTrustManagerFactorySpi.getDefaultTrustStore(ProvTrustManagerFactorySpi.java:109) ~[bctls-fips-debug-1.0.11.jar:1.0.11]
        at org.bouncycastle.jsse.provider.ProvTrustManagerFactorySpi.engineInit(ProvTrustManagerFactorySpi.java:152) ~[bctls-fips-debug-1.0.11.jar:1.0.11]
        at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:250) ~[?:1.8.0_321]
        at io.netty.handler.ssl.ReferenceCountedOpenSslServerContext.newSessionContext(ReferenceCountedOpenSslServerContext.java:147) ~[netty-all-4.1.58.Final.jar:4.1.58.Final]
        at io.netty.handler.ssl.OpenSslServerContext.<init>(OpenSslServerContext.java:353) ~[netty-all-4.1.58.Final.jar:4.1.58.Final]
        at io.netty.handler.ssl.OpenSslServerContext.<init>(OpenSslServerContext.java:334) ~[netty-all-4.1.58.Final.jar:4.1.58.Final]
        at io.netty.handler.ssl.SslContext.newServerContextInternal(SslContext.java:471) ~[netty-all-4.1.58.Final.jar:4.1.58.Final]
        at io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:571) ~[netty-all-4.1.58.Final.jar:4.1.58.Final]
        at org.apache.cassandra.security.SSLFactory.createNettySslContext(SSLFactory.java:339) ~[apache-cassandra-4.0.4.jar:4.0.4]
        at org.apache.cassandra.security.SSLFactory.validateSslContext(SSLFactory.java:496) ~[apache-cassandra-4.0.4.jar:4.0.4]
        at org.apache.cassandra.config.DatabaseDescriptor.applySslContext(DatabaseDescriptor.java:1009) ~[apache-cassandra-4.0.4.jar:4.0.4]
        ... 6 more

当前相关配置 #

server_encryption_options:
  keystore_password: PASSWORD
  enable_legacy_ssl_storage_port: true
  require_client_auth: false
  internode_encryption: none
  truststore_password: PASSWORD
  keystore: KEYSTORE_PATH
  truststore: TRUSTSTORE_PATH
  store_type: BCFKS
  cipher_suites: [TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]

解决方案 #

1. 验证密钥库完整性与密码正确性 #

使用BouncyCastle工具校验BCFKS密钥库，确认密码无误：

keytool -list -keystore KEYSTORE_PATH -storetype BCFKS -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath bc-fips-1.0.2.1.jar

若校验失败，重新生成符合FIPS标准的BCFKS密钥库/信任库。

2. 强制使用JDK SSL引擎而非Netty原生 #

修改cassandra.yaml的server_encryption_options，添加ssl_engine配置：

server_encryption_options:
  # 保留原有配置
  ssl_engine: JDK

该配置可绕开Netty原生SSL的密码传递问题，使用JDK默认SSL实现加载BCFKS密钥库。

3. 补充JVM启动参数，明确安全属性 #

在Cassandra启动脚本中添加以下JVM参数，强制指定BouncyCastle FIPS提供者及密钥库属性：

-Djavax.net.ssl.trustStorePassword=PASSWORD
-Djavax.net.ssl.keyStorePassword=PASSWORD
-Djavax.net.ssl.keyStoreType=BCFKS
-Djavax.net.ssl.trustStoreType=BCFKS
-Djava.security.provider=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider
-Dorg.bouncycastle.jsse.provider.enableFIPS=true

4. 修复Cassandra SSLFactory密码传递逻辑 #

若需继续使用Netty原生SSL，需修改SSLFactory.createNettySslContext()方法：

• 初始化TrustManagerFactory和KeyManagerFactory时，明确传递密码数组
• 指定信任管理器工厂算法为PKIX，确保与BouncyCastle FIPS兼容

内容的提问来源于stack exchange，提问作者mardanyan