验证RSASSA-PSS算法自签名证书时抛出InvalidKeyException异常

问题核心 #

你的代码在验证采用**RSASSA-PSS（OID 1.2.840.113549.1.1.10）**算法的根证书时，抛出InvalidKeyException: Supplied key is not a RSAPublicKey instance异常，但其他证书验证正常。本质原因是JDK默认的X509Certificate实现与BouncyCastle的RSASSA-PSS签名处理器之间存在密钥类型不兼容的问题：从证书获取的公钥是BouncyCastle自定义的RSAPublicKey实现类，而非JDK标准的java.security.interfaces.RSAPublicKey，导致签名初始化失败。

解决方案 #

方案1：注册BouncyCastle Provider并转换密钥类型 #

先将BouncyCastle注册为全局安全提供者，再将BouncyCastle格式的公钥转换为JDK标准类型，确保签名验证兼容：

import java.io.IOException;
import java.security.*;
import java.security.cert.CertificateException;
import java.security.cert.CertificateFactory;
import java.security.cert.X509Certificate;
import java.security.spec.RSAPublicKeySpec;
import java.net.URL;
import org.bouncycastle.jce.provider.BouncyCastleProvider;
import org.bouncycastle.jce.interfaces.RSAPublicKey;

public class SelfSignedChecker {
    // 全局注册BouncyCastle Provider，避免每次验证创建新实例
    static {
        Security.addProvider(new BouncyCastleProvider());
    }

    public static void main(String[] args) throws CertificateException, IOException, GeneralSecurityException {
        InputStream is = new URL("http://www.d-trust.net/cgi-bin/D-TRUST_Root_CA_1_2017.crt").openStream();
        CertificateFactory certFactory = CertificateFactory.getInstance("X.509");
        X509Certificate cert = (X509Certificate) certFactory.generateCertificate(is);
        System.out.println(cert);
        System.out.println("Self signed? " + isSelfSigned(cert));
    }

    public static boolean isSelfSigned(X509Certificate cert) throws GeneralSecurityException {
        // 前置快速检查：自签名证书的Subject和Issuer必须完全一致
        if (!cert.getSubjectDN().equals(cert.getIssuerDN())) {
            return false;
        }

        try {
            PublicKey publicKey = cert.getPublicKey();
            // 将BouncyCastle格式的RSAPublicKey转换为JDK标准类型
            if (publicKey instanceof RSAPublicKey) {
                RSAPublicKey bcRsaKey = (RSAPublicKey) publicKey;
                RSAPublicKeySpec spec = new RSAPublicKeySpec(bcRsaKey.getModulus(), bcRsaKey.getPublicExponent());
                KeyFactory keyFactory = KeyFactory.getInstance("RSA", "BC");
                publicKey = keyFactory.generatePublic(spec);
            }
            cert.verify(publicKey, "BC");
            return true;
        } catch (SignatureException | InvalidKeyException ex) {
            ex.printStackTrace();
            return false;
        }
    }
}

方案2：直接使用BouncyCastle原生API验证签名 #

绕过JDK的X509Certificate验证逻辑，直接用BouncyCastle的X509CertificateHolder处理签名验证，兼容性更好：

import java.io.IOException;
import java.security.*;
import java.security.cert.CertificateException;
import java.security.cert.CertificateFactory;
import java.security.cert.X509Certificate;
import java.net.URL;
import org.bouncycastle.jce.provider.BouncyCastleProvider;
import org.bouncycastle.cert.X509CertificateHolder;
import org.bouncycastle.cert.jcajce.JcaContentVerifierProviderBuilder;
import org.bouncycastle.operator.ContentVerifierProvider;

public class SelfSignedChecker {
    static {
        Security.addProvider(new BouncyCastleProvider());
    }

    public static void main(String[] args) throws CertificateException, IOException, GeneralSecurityException {
        InputStream is = new URL("http://www.d-trust.net/cgi-bin/D-TRUST_Root_CA_1_2017.crt").openStream();
        CertificateFactory certFactory = CertificateFactory.getInstance("X.509");
        X509Certificate cert = (X509Certificate) certFactory.generateCertificate(is);
        System.out.println(cert);
        System.out.println("Self signed? " + isSelfSigned(cert));
    }

    public static boolean isSelfSigned(X509Certificate cert) throws GeneralSecurityException, IOException {
        if (!cert.getSubjectDN().equals(cert.getIssuerDN())) {
            return false;
        }

        X509CertificateHolder certHolder = new X509CertificateHolder(cert.getEncoded());
        ContentVerifierProvider verifierProvider = new JcaContentVerifierProviderBuilder()
                .setProvider("BC")
                .build(cert.getPublicKey());
        return certHolder.isSignatureValid(verifierProvider);
    }
}

关键说明 #

1. 前置检查的必要性：先对比证书的SubjectDN和IssuerDN是高效的筛选手段——自签名证书的这两个字段必须完全一致，可直接排除非自签名的情况，避免不必要的签名计算。
2. Provider注册方式：全局注册BouncyCastle Provider比每次验证时传入新实例更稳定，能避免类加载和上下文不一致的问题。

内容的提问来源于stack exchange，提问作者Tilman Hausherr