Windows正常Mac异常：DTLS客户端未处理Server Finished问题排查

基于OpenJDK的DTLSOverDatagram实现了DTLS客户端/服务端，Windows虚拟机上的握手单元测试可正常运行，但Mac本地运行时出现异常：

• 服务器已完成握手流程
• 客户端却持续重复发送握手包
• javax.net.ssl日志显示客户端提示“未收到ServerHelloDone消息”，但实际上客户端代码已经处理了该消息

相关SSL日志 #

# 客户端日志片段
...
DEBUG: TLS handshake received: ServerHelloDone
DEBUG: Processing ServerHelloDone message
...
DEBUG: TLS handshake failed: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
DEBUG: Fatal alert received: handshake_failure
...
# 服务器日志片段
...
DEBUG: TLS handshake finished: SSL negotiation completed successfully
...

关键代码片段 #

握手处理逻辑 #

// 客户端握手处理部分
private void processHandshakeMessage(DatagramPacket packet) throws IOException {
    SSLHandshake handshake = sslEngine.getHandshakeStatus();
    if (handshake == SSLHandshakeStatus.NEED_UNWRAP) {
        ByteBuffer input = ByteBuffer.wrap(packet.getData(), 0, packet.getLength());
        sslEngine.unwrap(input, applicationBuffer);
        SSLHandshakeStatus newStatus = sslEngine.getHandshakeStatus();
        if (newStatus == SSLHandshakeStatus.FINISHED) {
            // 标记握手完成
            handshakeCompleted = true;
            System.out.println("Handshake completed locally");
        } else if (newStatus == SSLHandshakeStatus.NEED_WRAP) {
            generateHandshakePacket();
        }
    }
}

生成握手包逻辑 #

private void generateHandshakePacket() throws IOException {
    ByteBuffer output = ByteBuffer.allocate(sslEngine.getSession().getPacketBufferSize());
    SSLEngineResult result = sslEngine.wrap(ByteBuffer.allocate(0), output);
    output.flip();
    byte[] data = new byte[output.remaining()];
    output.get(data);
    DatagramPacket packet = new DatagramPacket(data, data.length, serverAddress, serverPort);
    datagramSocket.send(packet);
    System.out.println("Sent handshake packet, length: " + data.length);
}

内容的提问来源于stack exchange，提问作者omri