关于Criterion基准测试统计假设的疑问：迭代需完全相同代码还是仅统计等价即可？

Great question—this is a super common tradeoff between benchmark efficiency and statistical validity, especially for complex simulation models like yours. Let’s break this down in plain terms:

Core Criterion Statistical Assumptions #

First, let’s clarify what Criterion actually needs for reliable results: it requires that each iteration in a sample produces a measurement from the same underlying performance distribution. This doesn’t mean the code paths have to be exactly identical (same model state, same low-level operations) every single time—just that the performance characteristics we’re measuring are consistent across iterations.

In short: if each iteration’s runtime is drawn from the same mean and variance, Criterion’s statistical analyses (like mean estimation, confidence intervals, and cross-benchmark comparisons) will be valid.

Your Modified Approach Is (Likely) Perfectly Valid #

Your second approach—setting up the model once per sample, running it to equilibrium, then reusing it for multiple iterations of 1000 steps—is almost certainly acceptable given your description of the model’s behavior:

// For each sample:
|b, parameters| {
    // Setup model at the beginning of a sample
    let mut model = Model::new(parameters.clone());
    model.setup().unwrap();
    // Brings the simulation to statistical equilibrium
    model.run(100);
    // For each iteration (sequentially, without resetting the model):
    b.iter(
        || { model.run(black_box(1000)) }
    )
}

Here’s why this works:

• Each sample starts with a fresh, identical model setup, so samples are fully independent (a key requirement for Criterion’s cross-sample statistical tests).
• Each iteration measures the time to run 1000 steps in the model’s equilibrium state. Even though the model state changes between iterations, you’ve confirmed that per-step performance in equilibrium is consistent (with only 20% max variance per step, which is smoothed out by summing 1000 steps). This means each iteration’s runtime will come from the same underlying distribution—exactly what Criterion needs to produce reliable stats.
• This approach cuts out the costly per-iteration model setup, letting you run far more iterations per sample. More iterations mean tighter confidence intervals and more robust results, which is a massive improvement over your original inefficient setup.

Why the Single Timestep Alternative Is Less Ideal #

Benchmarking a single timestep would reduce per-iteration overhead, but it introduces unnecessary variance into your measurements. Since individual steps can vary by up to 20%, the runtime of a single step will have a much wider distribution than the sum of 1000 steps.

When you scale this to estimating runtime for 100 million steps, the relative error of the single-step benchmark will be far higher than the 1000-step benchmark. The 1000-step iteration averages out per-step noise, giving you a more stable estimate of the average per-step runtime—which is exactly what you need for large-scale projections.

Key Checks to Confirm Validity #

To be 100% confident, you should verify two small things:

1. No performance drift across iterations: After each 1000-step run, the model stays in a state where the next 1000 steps have the same performance characteristics. If running 1000 steps pushes the model out of equilibrium (e.g., into a slower/faster phase), your iteration distributions will shift and invalidate the stats. But you mentioned the model stays in equilibrium, so this shouldn’t be an issue.
2. No hidden compiler optimizations: Ensure the compiler isn’t eliding critical parts of model.run(1000) when reusing the model. Your use of black_box on the step count is a great guardrail here, but if your model has complex state, this is unlikely to be a problem anyway.

Final Recommendation #

Stick with your modified approach. It strikes the perfect balance between efficiency and statistical validity: you get more iterations per sample (better stats) while measuring exactly what you care about (1000 steps in equilibrium, which scales reliably to larger numbers of steps).

内容来源于stack exchange