多线程SSH客户端/服务端场景下libssh进程core dump原因问询

场景概述 #

单个C++进程内同时运行多个SSH服务器与SSH客户端实例，每个服务器/客户端都在独立线程中执行，每个线程持有独立的ssh_session对象，严格保证会话不在线程间共享。当前使用libssh版本为0.10.5。

崩溃回溯信息 #

#0 0x00007f42de961205 in std::string::assign(char const*, unsigned long) () from /lib64/libstdc++.so.6
#1 0x00000000004baa5c in CServerHandle::run_auth_gssapi_mic(ssh_session_struct*, char const*, char const*) ()
#2 0x00000000004bfc91 in CServerHandle::cb_auth_gssapi_mic(ssh_session_struct*, char const*, char const*, void*) ()
#3 0x0000000000512156 in ssh_client_connection_callback ()
#4 0x0000000000515d36 in ssh_packet_kexinit ()
#5 0x0000000000527efa in ssh_packet_process ()
#6 0x00000000005278e1 in ssh_packet_socket_callback ()
#7 0x00000000005279be in ssh_packet_socket_callback ()
#8 0x00000000005279be in ssh_packet_socket_callback ()
#9 0x00000000005279be in ssh_packet_socket_callback ()
#10 0x00000000005279be in ssh_packet_socket_callback ()
#11 0x00000000005279be in ssh_packet_socket_callback ()
#12 0x00000000005279be in ssh_packet_socket_callback ()
#13 0x00000000005279be in ssh_packet_socket_callback ()
#14 0x00000000005279be in ssh_packet_socket_callback ()
#15 0x00000000005279be in ssh_packet_socket_callback ()
#16 0x00000000005279be in ssh_packet_socket_callback ()
#17 0x00000000005279be in ssh_packet_socket_callback ()
#18 0x0000000000534d99 in ssh_socket_pollcallback ()
#19 0x0000000000532030 in ssh_poll_ctx_dopoll ()
#20 0x00000000005324f2 in ssh_event_dopoll ()
#21 0x000000000048bdde in CSession::tryEventPoll() ()
#22 0x00000000004aad78 in CHandle::handleSession(std::set<int, std::less<int>, std::allocator<int> >&) ()
#23 0x00000000004a5a2a in CClientHandle::service() ()
#24 0x00000000004e0a52 in CThread::callService(void*) ()
#25 0x00007f42defcfea5 in start_thread () from /lib64/libpthread.so.0
#26 0x00007f42de0bb8dd in clone () from /lib64/libc.so.6

崩溃触发函数 #

int CServerHandle::run_auth_gssapi_mic(ssh_session session, const char *user, const char *principal) {
    LOG("[AUTH > GSSAPI-MIC] %s. (user)%s, (principal)%s",
            m_info.getSummary().c_str(),
            user, principal);

    m_account = (NULL != user)? user : ""; // std::string m_account

    int ret = SSH_AUTH_ERROR;

    LOG("[AUTH > GSSAPI-MIC] %s. [RET]%d",
            m_info.getSummary().c_str(),
            ret);

    return ret;
}

相关类与回调注册代码 #

class CServerHandle : public CHandle {
public:
    CServerHandle() : CHandle() {
        m_session.setIsServer(true);

        ::memset(&m_serverCallbacks, 0x00, sizeof(m_serverCallbacks));
        m_serverCallbacks.userdata = this;
        m_serverCallbacks.service_request_function = cb_service_request;
        m_serverCallbacks.channel_open_request_session_function = cb_channel_open;
        m_serverCallbacks.auth_none_function = cb_auth_none;
        m_serverCallbacks.auth_password_function = cb_auth_password;
        m_serverCallbacks.auth_pubkey_function = cb_auth_public;
        m_serverCallbacks.auth_gssapi_mic_function = cb_auth_gssapi_mic;
        m_serverCallbacks.gssapi_select_oid_function = cb_gssapi_select_oid;
        m_serverCallbacks.gssapi_accept_sec_ctx_function = cb_gssapi_accept_sec_ctx;
        m_serverCallbacks.gssapi_verify_mic_function = cb_gssapi_verify_mic;
        ssh_callbacks_init(&m_serverCallbacks);
    }

    static inline int cb_auth_gssapi_mic(ssh_session session, const char *user, const char *principal, void *userdata) {
        return ((CServerHandle *)userdata)->run_auth_gssapi_mic(session, user, principal);
    }

    ...

private:
    struct ssh_server_callbacks_struct m_serverCallbacks;
}

class CHandle {
    ...

private:
    CSession m_session;
    std::string m_account;                              
}

CSession类内部包含一个ssh_session对象作为成员。在CServerHandle类处理客户端连接时，会将ssh_server_callbacks_struct作为参数传入，并通过ssh_set_server_callbacks()为每个ssh_session对象设置回调。

怀疑libssh存在问题的依据 #

• 回溯信息显示崩溃发生在std::string::assign，但崩溃函数本身无明显触发core dump的逻辑。当前SSH服务器不支持GSSAPI认证，此处无特殊处理，若崩溃是在user赋值给m_account时触发，可能是libssh向回调传递了非法内存地址。
• run_auth_gssapi_mic()是注册给服务端的回调函数，未在客户端注册，本不应被调用，但回溯显示它从SSH客户端处理逻辑(CClientHandle::service())发起，经libssh的poll流程最终被执行。
• SSH客户端已调用ssh_connect()成功连接服务器，此时通过libssh poll流程调用GSSAPI相关函数不符合预期。

目前无法稳定复现该问题。

待确认的问题 #

• 是否存在libssh使用方式错误？
• 是否有其他开发者遇到过类似问题？
• 这是否属于libssh的bug？
• 是否可能是操作系统与库交互导致的内存相关bug？

已尝试的排查操作 #

分析崩溃回溯及相关代码，怀疑问题与libssh的内存管理或回调处理逻辑有关，但尚未找到明确根因。

预期结果 #

SSH客户端与服务端应在各自线程中独立运行，不会触发core dump。

内容的提问来源于stack exchange，提问作者seungmin