Mellanox ConnectX-5网卡DPDK多RX队列场景下IP分片包RSS哈希一致性计算技术问询

Got it, let's break down how to solve this IP fragment RSS misdistribution issue with your ConnectX-5 and DPDK setup. Here's a step-by-step guide to implementing the same hash logic as your NIC in C++:

1. Understand the Root Cause #

Your problem happens because:

• For full TCP/UDP packets, the ConnectX-5 uses a tuple of src IP + dst IP + src port + dst port + protocol to compute the RSS hash.
• For IP fragments (non-first fragments), the NIC doesn't have access to L4 ports, so it falls back to a smaller tuple (usually src IP + dst IP + protocol + fragment ID), leading to a different hash value and thus a different queue/process.

To fix this, you need to reassemble IP fragments first, then compute the hash using the full tuple (matching what the NIC does for non-fragmented packets), so the reassembled packet gets sent to the same process as the original full packet.

2. Get the NIC's RSS Configuration #

First, you need to fetch the exact RSS hash algorithm and key your ConnectX-5 is using—this ensures your code's hash matches the NIC's exactly. Use DPDK's API to retrieve this:

#include <rte_ethdev.h>
#include <cassert>

// Replace with your DPDK port ID
const uint16_t port_id = 0;
struct rte_eth_rss_conf rss_conf;

// Fetch RSS config from the NIC
int ret = rte_eth_dev_rss_hash_conf_get(port_id, &rss_conf);
if (ret != 0) {
    // Handle error (e.g., log and exit)
    fprintf(stderr, "Failed to get RSS config: %d\n", ret);
    return -1;
}

// ConnectX-5 uses Toeplitz hash by default—verify this
assert(rss_conf.hash_func == RTE_ETH_HASH_FUNCTION_TOEPLITZ && 
       "NIC is using an unexpected hash algorithm");

3. Implement IP Fragment Reassembly #

You'll need to build a fragment reassembly engine (or use a lightweight library) to reconstruct full IP packets from fragments. Key things to track for reassembly:

• src IP + dst IP + fragment ID + protocol (unique fragment group identifier)
• Track the offset of each fragment to reconstruct the full payload
• Time out stale fragment groups to avoid memory leaks

4. Compute the Matching RSS Hash #

Once you have the reassembled full packet, extract the necessary tuple fields (all in network byte order, since the NIC uses network byte order for hash calculation), then use DPDK's built-in hash function to compute the same hash as the NIC:

#include <rte_hash.h>
#include <netinet/ip.h>
#include <netinet/tcp.h>
#include <netinet/udp.h>

uint32_t compute_nic_rss_hash(const struct ip* ip_hdr, const rte_eth_rss_conf& rss_conf) {
    struct rte_eth_rss_hdr rss_tuple;
    memset(&rss_tuple, 0, sizeof(rss_tuple));

    // Fill Ethernet type (IP)
    rss_tuple.eth_type = htons(ETH_P_IP);
    // Fill IP addresses (already in network byte order)
    rss_tuple.ip_src = ip_hdr->ip_src.s_addr;
    rss_tuple.ip_dst = ip_hdr->ip_dst.s_addr;
    rss_tuple.proto = ip_hdr->ip_p;

    // Fill L4 ports based on protocol
    if (ip_hdr->ip_p == IPPROTO_TCP) {
        const struct tcphdr* tcp_hdr = (struct tcphdr*)((uint8_t*)ip_hdr + (ip_hdr->ip_hl << 2));
        rss_tuple.tcp_src = tcp_hdr->th_sport;
        rss_tuple.tcp_dst = tcp_hdr->th_dport;
    } else if (ip_hdr->ip_p == IPPROTO_UDP) {
        const struct udphdr* udp_hdr = (struct udphdr*)((uint8_t*)ip_hdr + (ip_hdr->ip_hl << 2));
        rss_tuple.udp_src = udp_hdr->uh_sport;
        rss_tuple.udp_dst = udp_hdr->uh_dport;
    }

    // Compute the hash using the NIC's key and algorithm
    uint32_t hash_val;
    rte_eth_rss_hash(&rss_tuple, sizeof(rss_tuple), rss_conf.key, rss_conf.hash_func, &hash_val);
    return hash_val;
}

5. Map Hash to Queue/Process #

Finally, map the computed hash to the same queue ID the NIC would use. This is done by taking the hash value modulo the number of RX queues:

// Replace with your number of RX queues
const uint16_t num_rx_queues = 8;
uint16_t target_queue = compute_nic_rss_hash(reassembled_ip_hdr, rss_conf) % num_rx_queues;

// Send the reassembled packet to the corresponding process/queue
// (e.g., use DPDK's rte_eth_tx_burst or inter-process communication)

Key Notes #

• Byte Order Matters: All tuple fields must be in network byte order—don't convert them to host byte order, as the NIC uses network byte order for hash calculation.
• Reassembly Timeouts: Make sure to add timeouts for fragment groups to prevent memory leaks from incomplete fragments.
• Use DPDK's Built-in Functions: Avoid rolling your own Toeplitz hash—DPDK's rte_eth_rss_hash is already aligned with Mellanox's NIC logic, so it's the most reliable way to get a matching hash.

内容的提问来源于stack exchange，提问作者yaron