各位好，我最近在做一个基于eBPF的项目，用libbpf-rs加载包含数组映射（ARRAY_OF_MAPS）的BPF程序时卡壳了——每次调用MapHandle::create创建LRU_HASH子映射都会报错Invalid argument (os error 22)，试了好几种调整参数的方式都没解决，想请大家帮忙看看问题出在哪。

先把相关代码和配置贴出来：

我的BPF程序（src/bpf/test.bpf.c） #

这个程序定义了一个LRU_HASH类型的子映射flows_t，然后外层是一个数组映射per_cpu_flows用来按CPU存储这些子映射：

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#define MAX_CPUS 32

// Size: 8 bytes
struct flow_key_t {
    __u32 src_ip_be;
    __u16 src_port_be;
    __u16 dst_port_be;
};

// Size: 16 bytes
struct flow_entry_t {
    __u64 timestamp_ns;
    __u32 dst_ip_be;
    __u16 dst_port_be;
    __u16 src_port_be;
};

// Clang generates FWD types for the above without these...
struct flow_key_t key __attribute__((unused));
struct flow_entry_t entry __attribute__((unused));

struct flows_t {
    __uint(type, BPF_MAP_TYPE_LRU_HASH);
    __uint(max_entries, 1024);
    __type(key, struct flow_key_t);
    __type(value, struct flow_entry_t);
};

struct {
    __uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
    __uint(max_entries, MAX_CPUS + 1);
    __type(key, __u32);
    __array(values, struct flows_t);
} per_cpu_flows SEC(".maps");

SEC("xdp") int test(struct xdp_md *ctx) {
    return XDP_PASS;
}

char LICENSE[] SEC("license") = "GPL";

Rust主程序（src/main.rs） #

我尝试手动创建每个CPU对应的LRU_HASH子映射，然后更新到外层的数组映射里：

use anyhow::Result;
use libbpf_rs::libbpf_sys::bpf_map_create_opts;
use libbpf_rs::skel::OpenSkel;
use libbpf_rs::skel::SkelBuilder;
use libbpf_rs::MapCore;
use libbpf_rs::MapFlags;
use libbpf_rs::MapHandle;
use libbpf_rs::MapType;
use std::ffi::OsStr;
use std::mem::MaybeUninit;
use std::os::fd::AsFd;
use std::os::fd::AsRawFd;

mod test {
    include!(concat!(env!("CARGO_MANIFEST_DIR"), "/src/bpf/test.skel.rs"));
}

#[allow(clippy::wildcard_imports)]
use test::*;

fn main() -> Result<()> {
    let skel_builder = TestSkelBuilder::default();
    let mut open_object = MaybeUninit::uninit();
    let open_skel = skel_builder.open(&mut open_object)?;
    let skel = open_skel.load()?;

    let cpus = num_cpus::get() as u32;
    let per_cpu_flows = &skel.maps.per_cpu_flows;

    for i in 0..cpus + 1 {
        let key = &i.to_ne_bytes();
        let name = format!("cpu-{}-flows", i);
        let name: &OsStr = OsStr::new(&name);

        let mut opts: bpf_map_create_opts = unsafe { std::mem::zeroed() };
        opts.sz = size_of::<bpf_map_create_opts>() as u64;

        let value = MapHandle::create(
            MapType::LruHash,
            name.into(),
            8,
            16,
            1024,
            &opts
        ).expect("Error creating map");

        per_cpu_flows
            .update(
                key,
                &value.as_fd().as_raw_fd().to_ne_bytes(),
                MapFlags::ANY
            )
            .expect("Unable to initialize map for CPU");
    }

    let link = skel
        .progs
        .test
        .attach_xdp(1)
        .expect("Failed to attach XDP program");

    Ok(())
}

build.rs 构建脚本 #

用来生成BPF程序的skeleton：

use std::env;
use std::ffi::OsStr;
use std::path::PathBuf;
use libbpf_cargo::SkeletonBuilder;

const SRC: &str = "src/bpf/test.bpf.c";

fn main() {
    let out = PathBuf::from(
        env::var_os("CARGO_MANIFEST_DIR").expect("CARGO_MANIFEST_DIR must be set in build script"),
    )
    .join("src")
    .join("bpf")
    .join("test.skel.rs");

    let arch = env::var("CARGO_CFG_TARGET_ARCH")
        .expect("CARGO_CFG_TARGET_ARCH must be set in build script");

    SkeletonBuilder::new()
        .source(SRC)
        .clang_args([
            OsStr::new("-I"),
            vmlinux::include_path_root().join(arch).as_os_str(),
        ])
        .build_and_generate(&out)
        .unwrap();

    println!("cargo:rerun-if-changed={SRC}");
}

Cargo.toml 依赖配置 #

[package]
name = "libbpf-test"
version = "0.1.0"
edition = "2021"

[dependencies]
anyhow = "1.0.98"
libbpf-rs = { version = "0.25.0", features = [ "libbpf-sys" ] }
num_cpus = "1.17.0"

[build-dependencies]
libbpf-cargo = "0.25.0"
vmlinux = { git = "https://github.com/libbpf/vmlinux.h.git", rev = "83a228cf37fc65f2d14e4896a04922b5ee531a94" }

遇到的错误 #

运行程序后直接panic，错误信息是：

thread 'main' panicked at src/main.rs:40:14:
Error creating map: Error: Invalid argument (os error 22)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

补充：用ebpf-go实现的成功版本 #

之前我用ebpf-go实现过完全相同的逻辑是可以正常工作的，代码大概是这样：

for i := uint32(0); i <= numCPUs+1; i++ {
    flowMapSpec := spec.Maps["flows"]
    innerMap, err := ebpf.NewMap(flowMapSpec)
    if err != nil {
        log.Fatalf("creating inner map %d: %v", i, err)
    }
    if err := objs.UdpRouterMaps.PerCpuFlows.Put(i, innerMap); err != nil {
        log.Fatalf("inserting into per_cpu_flows[%d]: %v", i, err)
    }
    defer innerMap.Close()
}

这个Go版本里，我在C程序里给struct flows_t加上了SEC(".maps")定义，直接通过读取spec来创建子映射。用strace跟踪Go程序时看到的bpf调用片段是：

bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_LRU_HASH, key_size=8, value_size=16, max_entries=512000, map_flags=0, inner_map_fd=0, map_name="per_cpu_flows_i", map_...

我感觉问题应该出在bpf_map_create_opts的参数设置上，或者是libbpf-rs创建映射时的某些细节没处理对，但就是找不到具体原因。有没有熟悉libbpf-rs的朋友能帮我排查一下？

内容来源于stack exchange