Linux中madvise MADV_SEQUENTIAL对页面驱逐的影响及多线程场景疑问

Great question—this is exactly the kind of nuanced tradeoff that makes memory advice flags feel more art than science without a clear mental model. Let’s break down what’s happening here, starting with the core behavior of MADV_SEQUENTIAL as defined in the man page:

MADV_SEQUENTIAL表示“预期页面会被顺序访问。（因此，指定范围内的页面可被激进预读，且可能在访问后立即释放。）”

First, the good news: pre-read benefits still apply #

Since both of your threads are traversing the memory region sequentially, the aggressive pre-read behavior of MADV_SEQUENTIAL will absolutely work in your favor. The kernel will speculatively load pages ahead of each thread’s current position, cutting down on page faults and speeding up both traversals—this part is a safe win, regardless of the thread gap.

Now the tricky part: when does post-access eviction hurt? #

The key worry you have is whether pages get evicted after the first thread accesses them, but before the second thread gets to them. The kernel doesn’t evict pages blindly here—its decision depends on two critical factors:

1. Current system memory pressure #

• Low memory pressure: The kernel has no reason to rush evicting pages, even if they’re marked with MADV_SEQUENTIAL. Pages accessed by the first thread will likely stay in memory until the second thread reaches them. No extra page faults, just the pre-read benefit.
• High memory pressure: The kernel will prioritize evicting "cold" pages (pages that haven’t been accessed recently). This is where the thread gap matters.

2. The gap between your two threads #

• Small gap (threads are close together): When the first thread moves past a page, the second thread is right behind it. The page is still considered "warm" (recently accessed), so the kernel won’t evict it. No problem here—you get pre-read gains without eviction losses.
• Large gap (threads are far apart): If the first thread has already traversed pages that the second thread won’t reach for a long time, those pages become "cold" quickly. Under high memory pressure, the kernel will evict them to free up space. When the second thread finally gets to those pages, it’ll trigger a page fault, which adds overhead that could outweigh the pre-read benefits.

Your mental model for decision-making #

Think of this as a sliding scale:

• Best case for MADV_SEQUENTIAL: Threads stay close together, system memory is mostly available. You get maximum pre-read speedup with zero eviction-related losses.
• Risky case: Threads drift far apart, and the system is often under high memory pressure. Here, you might see net losses from unnecessary page faults.
• Middle ground: If threads sometimes drift but not consistently, or memory pressure varies, test both configurations (with and without MADV_SEQUENTIAL) and measure which performs better for your specific workload.

A quick side note: Most kernels don’t evict pages immediately after they’re accessed with MADV_SEQUENTIAL. Instead, they move those pages to the end of the page reclamation queue—giving your second thread a window to access them before they’re marked for eviction. This buffer helps reduce the risk of unnecessary faults even if there’s a small gap.

内容的提问来源于stack exchange，提问作者Kaz Wesley