Ubuntu 22.04 LTS下启用GSO实现TUN设备大TCP包内核分段时连接无法建立的问题求助

I've successfully implemented this exact scenario—sending jumbo TCP packets via a MTU 65535 TUN interface and letting the kernel handle segmentation via GSO/TSO before sending out eth0 (MTU 1500). Let's break down the issues in your code and fix them, plus share a working minimal example.

Key Issues in Your Current Code #

1. Incorrect GSOSize Value: You're setting GSOSize to the MTU (1500), but this field expects the TCP MSS (Maximum Segment Size)—the size of the TCP payload per segment. For IPv4, MSS = MTU - IPv4 header length (20) - TCP header length (20) = 1460. Using the full MTU here will cause segmented packets to exceed the eth0 MTU, leading to drops or failed connections.
2. Unnecessary Offload Flags for Control Packets: SYN/SYN-ACK/FIN/RST packets are small control frames and don't need GSO or checksum offloading. Forcing these flags on such packets can confuse the kernel's TCP stack.

Fixed Implementation #

1. TUN Initialization (with post-setup steps)

Your TUN setup code is mostly correct, but you need to configure the TUN interface with an IP, jumbo MTU, and route after creation:

# After creating the TUN interface (e.g., tun0)
ip addr add 10.0.0.1/24 dev tun0
ip link set tun0 mtu 65535
ip link set tun0 up
ip route add 0.0.0.0/0 dev tun0  # Or specific target routes as needed

2. Corrected PrependVnetHeader Function

This version fixes the MSS calculation and skips offload for TCP control packets:

package main

import (
	"encoding/binary"
	"errors"
)

const (
	VIRTIO_NET_HDR_F_NEEDS_CSUM = 1
	VIRTIO_NET_HDR_GSO_NONE     = 0
	VIRTIO_NET_HDR_GSO_TCPV4    = 1
	VIRTIO_NET_HDR_GSO_TCPV6    = 4
	TCP_CHECKSUM_OFFSET         = 16
)

type VirtioNetHdr struct {
	Flags       uint8
	GSOType     uint8
	HdrLen      uint16
	GSOSize     uint16
	CSumStart   uint16
	CSumOffset  uint16
}

func PrependVnetHeader(pkt []byte, mtu int) ([]byte, error) {
	if len(pkt) < 1 {
		return nil, errors.New("empty packet")
	}

	ipVersion := (pkt[0] & 0xF0) >> 4
	var hdr VirtioNetHdr
	isTCPCtrl := false

	switch ipVersion {
	case 4: // IPv4
		if len(pkt) < 20 {
			return nil, errors.New("invalid IPv4 packet")
		}
		if pkt[9] != 6 { // Only handle TCP packets
			break
		}
		ipHeaderLen := int(pkt[0]&0x0F) * 4
		tcpHeaderLen := int(pkt[ipHeaderLen+12]&0xF0) >> 4 * 4
		tcpPayloadLen := len(pkt) - ipHeaderLen - tcpHeaderLen

		// Detect TCP control packets (SYN/FIN/RST)
		tcpFlags := pkt[ipHeaderLen+13]
		if (tcpFlags & 0x02) != 0 || (tcpFlags & 0x01) != 0 || (tcpFlags & 0x04) != 0 {
			isTCPCtrl = true
		}

		if !isTCPCtrl {
			hdr.Flags = VIRTIO_NET_HDR_F_NEEDS_CSUM
			hdr.HdrLen = uint16(ipHeaderLen + tcpHeaderLen)
			hdr.CSumStart = uint16(ipHeaderLen)
			hdr.CSumOffset = TCP_CHECKSUM_OFFSET

			mss := mtu - ipHeaderLen - tcpHeaderLen
			if tcpPayloadLen > mss {
				hdr.GSOType = VIRTIO_NET_HDR_GSO_TCPV4
				hdr.GSOSize = uint16(mss)
			} else {
				hdr.GSOType = VIRTIO_NET_HDR_GSO_NONE
			}
		}
	case 6: // IPv6
		if len(pkt) < 40 {
			return nil, errors.New("invalid IPv6 packet")
		}
		if pkt[6] != 6 { // Only handle TCP packets
			break
		}
		ipHeaderLen := 40
		tcpHeaderLen := int(pkt[ipHeaderLen+12]&0xF0) >> 4 * 4
		tcpPayloadLen := len(pkt) - ipHeaderLen - tcpHeaderLen

		// Detect TCP control packets (SYN/FIN/RST)
		tcpFlags := pkt[ipHeaderLen+13]
		if (tcpFlags & 0x02) != 0 || (tcpFlags & 0x01) != 0 || (tcpFlags & 0x04) != 0 {
			isTCPCtrl = true
		}

		if !isTCPCtrl {
			hdr.Flags = VIRTIO_NET_HDR_F_NEEDS_CSUM
			hdr.HdrLen = uint16(ipHeaderLen + tcpHeaderLen)
			hdr.CSumStart = uint16(ipHeaderLen)
			hdr.CSumOffset = TCP_CHECKSUM_OFFSET

			mss := mtu - ipHeaderLen - tcpHeaderLen
			if tcpPayloadLen > mss {
				hdr.GSOType = VIRTIO_NET_HDR_GSO_TCPV6
				hdr.GSOSize = uint16(mss)
			} else {
				hdr.GSOType = VIRTIO_NET_HDR_GSO_NONE
			}
		}
	}

	// Serialize virtio header + original packet
	buf := make([]byte, 12+len(pkt))
	buf[0] = hdr.Flags
	buf[1] = hdr.GSOType
	binary.LittleEndian.PutUint16(buf[2:4], hdr.HdrLen)
	binary.LittleEndian.PutUint16(buf[4:6], hdr.GSOSize)
	binary.LittleEndian.PutUint16(buf[6:8], hdr.CSumStart)
	binary.LittleEndian.PutUint16(buf[8:10], hdr.CSumOffset)
	// Reserved bytes (10-12) remain 0
	copy(buf[12:], pkt)

	return buf, nil
}

Critical Additional Checks #

1. Verify Kernel Support: Ensure your kernel has these configs enabled:
   • CONFIG_TUN=y
   • CONFIG_NET_GSO=y
   • CONFIG_TCP_MD5SIG=n (MD5 signatures conflict with TSO/GSO)
2. Enable eth0 Offload Capabilities: Confirm your physical interface supports TSO/GSO, and enable them if needed:

   # Check current status
   ethtool -k eth0 | grep -E 'tcp-segmentation-offload|generic-segmentation-offload'
   # Enable if off
   ethtool -K eth0 tso on gso on
   

Testing the Setup #

1. Write a jumbo TCP packet (e.g., 10KB) to the TUN device after prepending the corrected vnet header.
2. Capture packets on eth0 with tcpdump -i eth0 -nn -vv—you should see multiple 1500-byte segments instead of one jumbo packet.
3. Verify the TCP connection completes successfully (SYN → SYN-ACK → ACK, followed by data segments).

Troubleshooting Tips #

• If connections still fail, check dmesg for kernel warnings about invalid GSO packets.
• For TCP control packets (SYN/FIN/RST), you must compute valid IP/TCP checksums in user space (since we skip offload for these).
• Double-check the HdrLen field: it must match the total length of IP + TCP headers (adjust if using IP/TCP options).

内容的提问来源于stack exchange，提问作者Alex Barysevich