recvfrom重复接收UDP广播但tcpdump仅收一次的技术问题排查

Let’s tackle your two core problems one by one, with practical fixes and explanations based on how Linux network stacks and SmartNIC offload work.

Problem 1: SO_BINDTODEVICE Fails to Restrict Reception to enp1s0np1 #

Why This Happens #

The SO_BINDTODEVICE option has a few gotchas, especially with SmartNICs and raw sockets:

1. Permission Requirements: You need CAP_NET_RAW and CAP_NET_ADMIN capabilities to use this option effectively—running your receiver as root is a quick test, but using setcap is better for production.
2. AF_INET vs AF_PACKET Limitations: If your receiver uses an AF_INET socket (for UDP), SO_BINDTODEVICE might not work as expected on interfaces without an IP address (like your SmartNIC ports). Raw AF_PACKET sockets require a different approach to bind to specific interfaces.
3. SmartNIC Driver/Offload Logic: Netronome’s drivers might bypass standard Linux socket filtering for offloaded traffic, making SO_BINDTODEVICE less effective.

Fixes to Try #

1. Use bind() with AF_PACKET Sockets (More Reliable)
   If you’re using a raw socket, bind directly to the interface’s index instead of relying on SO_BINDTODEVICE:

   #include <net/if.h>
   #include <linux/if_packet.h>
   
   struct sockaddr_ll sll = {0};
   sll.sll_family = AF_PACKET;
   sll.sll_protocol = htons(ETH_P_IP); // Adjust to ETH_P_ALL if capturing all frames
   sll.sll_ifindex = if_nametoindex("enp1s0np1");
   
   if (bind(sock_fd, (struct sockaddr *)&sll, sizeof(sll)) == -1) {
       perror("Failed to bind to enp1s0np1");
       exit(EXIT_FAILURE);
   }
   

2. Add Required Capabilities
   Run this command to grant your receiver the necessary permissions without running as root:

   setcap cap_net_raw,cap_net_admin+ep ./your_receiver_binary
   

3. Verify PACKET_ADD_MEMBERSHIP Setup
   If you’re mimicking tcpdump’s approach, ensure you’re targeting the correct interface index and using the right membership type (for broadcast traffic, add PACKET_MR_BROADCAST):

   struct packet_mreq mreq = {0};
   mreq.mr_ifindex = if_nametoindex("enp1s0np1");
   mreq.mr_type = PACKET_MR_BROADCAST;
   mreq.mr_alen = ETH_ALEN;
   
   if (setsockopt(sock_fd, SOL_PACKET, PACKET_ADD_MEMBERSHIP, &mreq, sizeof(mreq)) == -1) {
       perror("Failed to add broadcast membership");
   }
   

Problem 2: Receiving Both Original and XDP-Modified Packets (But tcpdump Only Sees Modified Ones) #

Why This Happens #

This usually stems from a mismatch between your XDP program’s return action and how the Linux network stack processes the packet:

1. XDP Return Value Misconfiguration: If your XDP program returns XDP_PASS after modifying the packet, the modified frame continues through the host’s network stack. Meanwhile, the SmartNIC might be forwarding the original (or modified) frame directly to enp1s0np1 via hardware offload, creating two copies.
2. Loopback/Broadcast Leakage: Your sender’s UDP broadcast might be leaking to the host’s loopback interface, which your receiver is picking up as the "original" packet.

Fixes to Try #

1. Adjust XDP Program Return Action
   Modify your XDP program to return XDP_TX (to send the modified frame out enp1s0np0 directly) or XDP_REDIRECT (to send it straight to enp1s0np1 via SmartNIC offload) instead of XDP_PASS. This prevents the modified packet from entering the host stack, eliminating duplicate copies.
   Example XDP snippet:

   // After modifying the packet's 28-35 byte range to "!......!"
   return XDP_TX; // Sends the modified frame out the incoming interface (enp1s0np0)
   

2. Identify Packet Source Interface
   Add code to your receiver to print which interface each packet comes from—this will confirm if the "original" packet is coming from the loopback (lo) or another unintended interface:
   For AF_INET UDP sockets, use IP_PKTINFO with recvmsg:

   #include <linux/ip.h>
   #include <sys/socket.h>
   
   // Enable IP_PKTINFO to get interface info
   int opt = 1;
   setsockopt(sock_fd, IPPROTO_IP, IP_PKTINFO, &opt, sizeof(opt));
   
   struct msghdr msg = {0};
   struct iovec iov = {.iov_base = buf, .iov_len = sizeof(buf)};
   msg.msg_iov = &iov;
   msg.msg_iovlen = 1;
   
   char cmsg_buf[CMSG_SPACE(sizeof(struct in_pktinfo))];
   msg.msg_control = cmsg_buf;
   msg.msg_controllen = sizeof(cmsg_buf);
   
   ssize_t recv_len = recvmsg(sock_fd, &msg, 0);
   if (recv_len > 0) {
       struct cmsghdr *cmsg = CMSG_FIRSTHDR(&msg);
       if (cmsg && cmsg->cmsg_level == IPPROTO_IP && cmsg->cmsg_type == IP_PKTINFO) {
           struct in_pktinfo *pkt_info = (struct in_pktinfo *)CMSG_DATA(cmsg);
           char if_name[IFNAMSIZ];
           if_indextoname(pkt_info->ipi_ifindex, if_name);
           printf("Packet received from interface: %s\n", if_name);
       }
   }
   

   If the original packet is coming from lo, you can filter it out by checking the interface index in your receiver code.

3. Disable Unnecessary Broadcast Forwarding
   Ensure the host isn’t forwarding broadcast packets between interfaces by checking the net.ipv4.ip_forward sysctl:

   sysctl net.ipv4.ip_forward
   

   If it’s set to 1, disable it with:

   sysctl -w net.ipv4.ip_forward=0
   

Final Verification Steps #

1. Run your receiver with the fixes above and confirm it only receives packets from enp1s0np1.
2. Check that you only get the XDP-modified packet (not the original) by inspecting the 28-35 byte range.
3. Compare tcpdump output on enp1s0np1 with your receiver’s output to ensure they match.

内容的提问来源于stack exchange，提问作者iBug