Linux下select报告套接字就绪后recvmsg返回EAGAIN问题排查

在Linux应用中，UDP套接字的收发操作在两个独立线程中执行，发送功能正常，但接收函数持续返回EAGAIN（Resource temporarily unavailable）。尽管已经调用select检查是否有可读数据，select仍会远快于设置的超时时间返回，导致控制台频繁刷出recvmsg failed (11). Resource temporarily unavailable错误。

原始接收代码 #

// Passed values as comments
MySocket::ReceiveRet MySocket::Receive(
  void *data /* Pointer to std::array on stack*/,
  size_t max_len /* = 170 */,
  suseconds_t usTimeout /* = 1000*/
)
{
    static uint64_t rcvId = 0;
    MySocket::ReceiveRet ret;
    if (m_socket == -1)
    {
        ret.errorCode = -1001;
        return ret;
    }
    struct timeval delta;
    memset(&delta, 0, sizeof(delta));
    delta.tv_usec = usTimeout;

    fd_set readfs, errorfs;
    FD_ZERO(&readfs);
    FD_ZERO(&errorfs);
    FD_SET(m_socket, &readfs);
    //FD_SET(m_socket, &errorfs);

    // Check for new packages
    int retSelct = select(m_socket + 1, &readfs, 0, &errorfs, &delta);
    if (retSelct <= 0)
    {
        // Nothing received
        ret.errorCode = retSelct;
        return ret;
    }

    if(!FD_ISSET(m_socket, &readfs))
    {
        std::cerr << rcvId << " Not ready for read" << std::endl;
        ret.errorCode = -1002;
        return ret;
    }

    struct msghdr msg;
    struct iovec entry;
    struct sockaddr_in from_addr;
    union
    {
        char control[512];
        struct cmsghdr cm;
    } control;

    memset(&msg, 0, sizeof(msg));
    msg.msg_iov = &entry;
    msg.msg_iovlen = 1;
    entry.iov_base = data;
    entry.iov_len = max_len;
    msg.msg_name = (caddr_t)&from_addr;
    msg.msg_namelen = sizeof(from_addr);
    msg.msg_control = &control;
    msg.msg_controllen = sizeof(control);
    // Read the message and metadata
    int res = recvmsg(m_socket, &msg, MSG_DONTWAIT);
    if (res < 0)
    {
        std::cerr << rcvId << "recvmsg failed (" << errno << "). " << strerror(errno) << " FD_ISSET(m_socket, &readfs): " << FD_ISSET(m_socket, &readfs) << std::endl;
        ret.errorCode = res;
        return ret;
    }

[...]

排查与解决 #

最初怀疑select返回假就绪，但实际是持续异常。进一步分析发现：套接字存在多个队列（数据队列、错误队列等），select会在任一可读队列有数据时返回，且不会区分具体是哪个队列。原代码只监听了数据队列，但错误队列中存在未处理的数据，导致select反复返回就绪，而读取普通数据时自然会得到EAGAIN。

修改后的接收逻辑如下：先尝试读取普通数据，若返回EAGAIN则尝试读取错误队列，之后根据msg_flags判断接收的是错误信息还是正常数据，分别处理：

int res = recvmsg(m_socket, &msg, MSG_DONTWAIT);
if(res < 0 && errno == EAGAIN)
{
    res = recvmsg(m_socket, &msg, MSG_ERRQUEUE| MSG_DONTWAIT);
}

if (res < 0)
{
    std::cerr << "ProcessReceive recvmsg failed (" << errno << "). " << strerror(errno)
    << std::endl;
    return;
}

if((msg.msg_flags & MSG_ERRQUEUE) == MSG_ERRQUEUE)
{
    // Error was received
    [...]
} else {
    // Normal data was received
    [...]
}

感谢pmacfarlane的提示，明确了select不区分套接字不同队列的特性。

内容的提问来源于stack exchange，提问作者arch