Ubuntu 23.04环境下fs.inotify.max_user_watches设置后仍报错的问题咨询

Hey there, let's break down why you're still hitting that "Too many open files" error even after cranking up fs.inotify.max_user_watches to a huge number, and what you can do about it.

First off, that error message is a bit misleading here—it might not actually be about inotify watches at all. Let's unpack the possible causes:

1. You're hitting process-level file descriptor limits, not inotify watch limits #

The "Too many open files" error often refers to the maximum number of file descriptors (FDs) a process can open, which includes regular files, sockets, pipes, and inotify instances. Even if you have tons of watches allowed, if a process (or processes in your containers) hits its FD limit, you'll get this error.

To check this:

• For a running process in a container, get its PID and run cat /proc/<pid>/limits—look for the "Max open files" line (soft and hard limits).
• On the host, check the default limits with ulimit -n (soft) and ulimit -Hn (hard). Systemd-nspawn containers might inherit these limits unless you override them.

2. You're hitting the fs.inotify.max_user_instances limit #

This is a separate inotify setting: it controls how many inotify instances a single user can create (each instance can have multiple watches). If you have 10 containers running multiple services that use inotify (like file watchers, IDEs, or monitoring tools), you might be hitting this default limit (usually 128 for most systems).

Check the current value with:

cat /proc/sys/fs/inotify/max_user_instances

3. Your max_user_watches setting is way too large for your RAM #

Let's do quick math: each inotify watch takes roughly 1KB of kernel memory (some sources say ~540 bytes, but let's use 1KB to be safe). Your 120,000,000 watches would need 120GB of RAM—but you only have 32GB. The kernel can't allocate that much memory, so it's probably ignoring your overly large setting, or failing to allocate watches because it's out of memory for them.

What to do next: #

• Fix the file descriptor limits:
  • For systemd-nspawn containers, you can set FD limits when starting the container with:

    systemd-nspawn --setenv=ULIMIT_NOFILE=65536 -D /path/to/container
    

  • Or persist it by adding LimitNOFILE=65536 to the container's service file (if you're using systemd to manage it).
• Increase max_user_instances:
  • Temporarily set it with:

    sudo sysctl -w fs.inotify.max_user_instances=8192
    

  • Persist it by adding fs.inotify.max_user_instances=8192 to /etc/sysctl.conf and running sudo sysctl -p.
• Set a realistic max_user_watches value:
  • Based on your 32GB RAM, a safe upper bound would be around 20-25 million (leaving plenty of RAM for your system and containers). Temporarily adjust it with:

    sudo sysctl -w fs.inotify.max_user_watches=20000000
    

  • Persist it in /etc/sysctl.conf as well.
• Identify which processes are using inotify:
  • Use this command to see which processes are creating inotify instances:

    for pid in $(pgrep -d ',' -f 'your-process-name'); do ls -l /proc/$pid/fd | grep inotify; done
    

  • Or use inotifywatch (from the inotify-tools package) to monitor usage, though it might be noisy with 10 containers.

Start with checking the file descriptor and max_user_instances limits first—those are the most likely culprits here, not the watch count itself.

备注：内容来源于stack exchange，提问作者CabbageGremlin