Linux内核shm.c修改疑问：shmget()/shmat()改动未生效排查

Hey there, let's dig into why your shmget()/shmat() changes aren't taking effect even after compiling your custom kernel. This is a super common gotcha, so let's break down the likely culprits step by step:

Kernel paths for SysV IPC code can shift across versions. For most modern kernels, shm logic lives in ipc/shm.c, but you need to confirm this matches the source tree of the kernel you're actually running.

• Run uname -r to get your current kernel version, then verify your modified source tree's version matches exactly (no mismatched minor versions or leftover patches from distros).
• Some distros apply custom patches to their kernel sources—make sure you're compiling from a clean tree that matches what you modified, not a pre-patched distro source.

Here's a key mistake many people make: the shmget() you call in user space is a glibc wrapper. The actual kernel syscall entry has a different name:

• For older kernels, look for sys_shmget() and sys_shmat() in ipc/shm.c
• For 64-bit modern kernels, it's likely __x64_sys_shmget() and __x64_sys_shmat()
• Run grep -r "shmget" ipc/ in your source tree to find the real kernel function names. If you only modified a helper function named shmget() (not the syscall entry), your changes won't trigger when user code calls the syscall.

If SysV shared memory is built as a module instead of built-in, modifying the main kernel won't affect it.

• Check your config: cat /boot/config-$(uname -r) and look for CONFIG_SYSVIPC=y and CONFIG_SYSVSHM=y (the y means built-in; m means it's a module).
• If it's a module, you'll need to recompile the shm module (make modules), install it (make modules_install), and reload it with modprobe shm (or reboot).

Add a simple debug print to the function you're modifying to test this. For example:

printk(KERN_INFO "Custom shmget called! PID: %d\n", current->pid);

After compiling and rebooting, run dmesg | grep "Custom shmget"—if you don't see this line when you call shmget() from a test program, your changes weren't included in the kernel.

• Pro tip: Always run make clean or make mrproper (backup your .config first!) before recompiling to avoid stale object files.

It's easy to compile a kernel but forget to update your bootloader:

• Check /boot for the kernel image (vmlinuz-<your-version>) and initramfs files—their timestamps should match when you compiled.
• Update your bootloader: run update-grub (Debian/Ubuntu) or grub2-mkconfig -o /boot/grub2/grub.cfg (RHEL/CentOS/Fedora) to ensure your custom kernel is set as default.
• Reboot and confirm with uname -a that the kernel version and build timestamp match your custom compile.

Write a tiny test program to confirm the syscall is being triggered:

#include <sys/ipc.h>
#include <sys/shm.h>
#include <stdio.h>

int main() {
    int shmid = shmget(IPC_PRIVATE, 4096, IPC_CREAT | 0666);
    printf("shmid: %d\n", shmid);
    return 0;
}

Compile it with gcc test.c -o test, then run strace ./test. You should see a line like shmget(IPC_PRIVATE, 4096, IPC_CREAT|0666) = <some-number>—this proves the syscall is hitting the kernel, so the issue is definitely in your kernel code or setup.

内容的提问来源于stack exchange，提问作者Luksh Kumar