如何编写Criterion性能测试？——基于可变向量与Data.Set的函数性能对比排查求助

Hey there! You're doing great diving into benchmarking Haskell code—let's walk through the issues in your setup and how to fix them to get meaningful performance comparisons.

1. Critical Benchmark Code Bug #

First, the biggest issue in your Bench.hs is that both benchmarks are running the same function (isUniqueChars2):

bench "with Data.Set" $ whnf isUniqueChars2 str
bench "with Mutable Vector" $ whnf isUniqueChars2 str

You meant to use isUniqueChars for the mutable vector test. Fix that first, otherwise you're not comparing anything!

2. Fix Random Input Generation #

Your current random string uses randomIO, which generates arbitrary Unicode Chars. But your vector implementation only allocates 128 slots (for ASCII 0-127). If a Char with ord x > 127 is generated, your code will crash with an index out-of-bounds error.

Change the random generation to only produce ASCII characters:

import Data.Char (chr)
import System.Random (randomRIO)

-- ...

str <- replicateM 1000 (chr <$> randomRIO (0, 127)) :: IO String

This ensures all characters stay within the bounds your vector expects.

3. Fix Function Implementation Issues #

Mutable Vector Version (isUniqueChars) #

• You're missing necessary imports for vector and ST:

  import Control.Monad.ST (runST)
  import Data.Char (ord)
  import qualified Data.Vector as V
  import qualified Data.Vector.Mutable as MV
  

  Without these, your code won't compile.
• The PrimMonad constraint is correct, but you can simplify the go function's type if you want (though it's not strictly necessary).

Data.Set Version (isUniqueChars2) #

• Initializing charset with S.fromList [] is redundant—use S.empty instead (it's the same result, but more idiomatic and slightly faster):

  isUniqueChars2 :: String -> Bool
  isUniqueChars2 str = go str S.empty
    where
      go [] _ = True
      go (x:xs) charset =
        if S.member x charset
          then False
          else go xs (S.insert x charset)
  

• Also, don't forget to import Data.Set (qualified is best practice):

  import qualified Data.Set as S
  

4. Cabal Configuration Fixes #

Your cabal file has a couple of issues that will cause build problems:

• Circular Dependency: The executable happyhaskell has build-depends: ... happyhaskell—this is a loop, since the executable is part of the happyhaskell package. Remove happyhaskell from the executable's build-depends.
• Benchmark Dependency: Similarly, the benchmark bench doesn't need happyhaskell in its build-depends (you're already including Lib.Happy via other-modules).

Here's the corrected cabal file section:

executable happyhaskell
  main-is: Main.hs
  build-depends: base >=4.14 && <4.15
  default-language: Haskell2010
  -- If your Main.hs uses Lib.Happy, add:
  -- other-modules: Lib.Happy

library
  default-language: Haskell2010
  exposed-modules: Lib.Happy
  build-depends: base >=4.14 && <4.15, containers, primitive >= 0.7.1.0, vector

benchmark bench
  default-language: Haskell2010
  type: exitcode-stdio-1.0
  main-is: Bench.hs
  other-modules: Lib.Happy
  build-depends: base >=4.14 && <4.15, criterion, random, containers, primitive >= 0.7.1.0, vector

5. Extra Benchmarking Tips #

• Test Multiple Inputs: Right now you're using one random string. For more reliable results, generate a few different inputs (e.g., all unique characters, some duplicates, all duplicates) and benchmark against each.
• Use nf Instead of whnf: Since your functions return Bool (a small value), whnf is fine—but if you were returning larger structures, nf (normal form) would ensure full evaluation. For Bool, both are equivalent, but it's good to keep in mind.
• Compile with Optimizations: When running benchmarks, make sure to compile with -O2 (Cabal does this by default for benchmarks, but double-check with cabal bench --enable-optimization=2).

Once you fix these issues, your benchmark should properly compare the mutable vector and Data.Set implementations. You should see the vector version perform much better for ASCII strings, since it's O(n) time with constant-time lookups/writes, whereas the Set version is O(n log n).

内容的提问来源于stack exchange，提问作者daydaynatation