Spring Boot配置TLS时，如何返回unrecognized_name(112)错误？

我使用Spring Boot 3.3.4搭配Undertow和UndertowServletWebServerFactory启用了TLS，并添加了自定义KeyManager。在chooseEngineServerAlias(String keyType, Principal[] issuers, SSLEngine engine)方法中，当请求的服务器名不在我方提供的范围内时，我会返回null。此时应用返回的SSL握手日志如下：

CONNECTED(00000003)
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 309 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

请问能否修改此行为，让服务器返回unrecognized_name(112)错误？如果可以，具体该如何操作？

可以实现该需求，核心是在检测到未识别的服务器名时，主动触发带有指定错误码的TLS握手异常，而非简单返回null。具体操作步骤如下：

1. 自定义X509ExtendedKeyManager，抛出指定错误的异常 #

替换现有自定义KeyManager，在chooseEngineServerAlias方法中，当服务器名不在允许列表时，抛出包含unrecognized_name错误的SSLHandshakeException：

import javax.net.ssl.*;
import java.net.Socket;
import java.security.Principal;
import java.security.cert.X509Certificate;

public class CustomSniKeyManager extends X509ExtendedKeyManager {
    // 替换为你的允许服务器名列表
    private final String[] allowedServerNames = {"your-domain.com", "another-domain.com"};
    private final X509ExtendedKeyManager delegate;

    public CustomSniKeyManager(X509ExtendedKeyManager delegate) {
        this.delegate = delegate;
    }

    @Override
    public String chooseEngineServerAlias(String keyType, Principal[] issuers, SSLEngine engine) {
        // 提取SNI中的服务器名
        SSLParameters params = engine.getSSLParameters();
        SNIServerName sniName = params.getServerNames() != null && !params.getServerNames().isEmpty()
                ? params.getServerNames().get(0)
                : null;

        if (sniName instanceof SNIHostName) {
            String hostName = ((SNIHostName) sniName).getAsciiName();
            boolean isAllowed = false;
            for (String allowedName : allowedServerNames) {
                if (allowedName.equalsIgnoreCase(hostName)) {
                    isAllowed = true;
                    break;
                }
            }

            if (!isAllowed) {
                // 抛出携带unrecognized_name错误码的握手异常
                SSLHandshakeException exception = new SSLHandshakeException("Unrecognized server name");
                exception.initCause(new SSLException("unrecognized_name"));
                throw exception;
            }
        }

        // 合法请求委托给默认KeyManager处理
        return delegate.chooseEngineServerAlias(keyType, issuers, engine);
    }

    // 以下接口方法全部委托给默认KeyManager实现
    @Override
    public String chooseServerAlias(String keyType, Principal[] issuers, Socket socket) {
        return delegate.chooseServerAlias(keyType, issuers, socket);
    }

    @Override
    public String[] getServerAliases(String keyType, Principal[] issuers) {
        return delegate.getServerAliases(keyType, issuers);
    }

    @Override
    public String chooseClientAlias(String[] keyTypes, Principal[] issuers, Socket socket) {
        return delegate.chooseClientAlias(keyTypes, issuers, socket);
    }

    @Override
    public String[] getClientAliases(String keyType, Principal[] issuers) {
        return delegate.getClientAliases(keyType, issuers);
    }
}

2. 配置Spring Boot Undertow使用自定义KeyManager #

通过自定义UndertowServletWebServerFactory，替换默认SSLContext并注入自定义KeyManager：

import org.springframework.boot.web.embedded.undertow.UndertowServletWebServerFactory;
import org.springframework.boot.web.server.Ssl;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import javax.net.ssl.*;
import java.io.FileInputStream;
import java.security.KeyStore;

@Configuration
public class UndertowSslConfig {

    @Bean
    public UndertowServletWebServerFactory undertowServletWebServerFactory() throws Exception {
        UndertowServletWebServerFactory factory = new UndertowServletWebServerFactory();

        // 加载密钥库，替换为你的密钥库路径和密码
        KeyStore keyStore = KeyStore.getInstance("PKCS12");
        try (FileInputStream fis = new FileInputStream("path/to/your/keystore.p12")) {
            keyStore.load(fis, "your-keystore-password".toCharArray());
        }

        // 初始化默认KeyManagerFactory
        KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
        kmf.init(keyStore, "your-key-password".toCharArray());

        // 包装默认KeyManager为自定义实现
        KeyManager[] keyManagers = kmf.getKeyManagers();
        for (int i = 0; i < keyManagers.length; i++) {
            if (keyManagers[i] instanceof X509ExtendedKeyManager) {
                keyManagers[i] = new CustomSniKeyManager((X509ExtendedKeyManager) keyManagers[i]);
                break;
            }
        }

        // 创建并设置自定义SSLContext
        SSLContext sslContext = SSLContext.getInstance("TLS");
        sslContext.init(keyManagers, null, null);
        factory.setSslContext(sslContext);

        // 启用SSL并配置基础参数
        Ssl ssl = new Ssl();
        ssl.setEnabled(true);
        ssl.setProtocol("TLSv1.2");
        factory.setSsl(ssl);

        return factory;
    }
}

3. 验证效果 #

当客户端请求的服务器名不在允许列表中时，服务器会发送unrecognized_name(112)警告并终止握手，客户端SSL日志会显示类似如下内容：

CONNECTED(00000003)
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 325 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 112 (unrecognized_name)
---

内容的提问来源于stack exchange，提问作者Harry Developer