Rspec使用--only-failures时重复运行所有测试问题排查

问题现象 #

封装RSpec用于Selenium测试的Ruby gem，尝试实现失败用例重试逻辑，但第二次运行--only-failures时，不仅执行了失败用例，还出现重复执行的情况（输出显示4个示例，实际仅2个）。调试发现spec_helper.rb仅加载一次，且未出现未设置持久化路径的错误。

相关代码 #

spec_helper.rb

RSpec.configure do |config|
  config.example_status_persistence_file_path = "logs/LO_RunResults.txt"
end

my_spec.rb

it "should pass" do
  expect(true).to be true
end

it "should fail" do
  expect(false).to be true
end

rspec_runner.rb

test_results_file = "#{File.expand_path(Dir.pwd)}/logs/RunResults.txt"
File.delete(test_results_file) if File.exist?(test_results_file)

args = %w(-r spec_helper --order defined --format CustomFormatter --format documentation)
RSpec::Core::Runner.run(args, $stderr, $stdout)

# RunResults.txt is generated from previous run

args = %w(-r spec_helper --order defined --format CustomFormatter --format documentation --only-failures)
RSpec::Core::Runner.run(args, $stderr, $stdout)

运行输出 #

#################### Running spec_helper ####################

Test Suite - my_spec
  Positive Tests
    should pass
    should fail (FAILED - 1)

Failures:

  1) Test Suite - my_spec Positive Tests should fail
     Failure/Error: expect(false).to be true
     
       expected true
            got false
     # ./spec/my_spec.rb:59:in `block (3 levels) in <top (required)>'

Finished in 0.06265 seconds (files took 0.71696 seconds to load)
2 examples, 1 failure

Failed examples:

rspec ./spec/my_spec.rb:58 # Test Suite - my_spec Positive Tests should fail

Run options: include {:last_run_status=>"failed"}

Test Suite - my_spec
  Positive Tests
    should pass
    should fail (FAILED - 2)

Failures:

  1) Test Suite - my_spec Positive Tests should fail
     Got 2 failures:

     1.1) Failure/Error: expect(false).to be true
          
            expected true
                 got false
          # ./spec/my_spec.rb:59:in `block (3 levels) in <top (required)>'

     1.2) Failure/Error: expect(false).to be true
          
            expected true
                 got false
          # ./spec/my_spec.rb:59:in `block (3 levels) in <top (required)>'

  2) Test Suite - my_spec Positive Tests should fail
     Got 2 failures:

     2.1) Failure/Error: expect(false).to be true
          
            expected true
                 got false
          # ./spec/my_spec.rb:59:in `block (3 levels) in <top (required)>'

     2.2) Failure/Error: expect(false).to be true
          
            expected true
                 got false
          # ./spec/my_spec.rb:59:in `block (3 levels) in <top (required)>'

Finished in 0.03786 seconds (files took 0.79882 seconds to load)
4 examples, 2 failures

Failed examples:

rspec ./spec/my_spec.rb:58 # Test Suite - my_spec Positive Tests should fail
rspec ./spec/my_spec.rb:58 # Test Suite - my_spec Positive Tests should fail


Process finished with exit code 0

问题根源 #

• 持久化文件路径不匹配：spec_helper.rb中配置的持久化文件是logs/LO_RunResults.txt，但rspec_runner.rb中操作的是logs/RunResults.txt，两者文件名不一致，导致第二次运行时无法读取到正确的失败用例记录，RSpec fallback执行所有用例。
• 同一进程重复运行RSpec的状态污染：RSpec的设计不支持在同一个Ruby进程内多次调用RSpec::Core::Runner.run，因为全局的示例定义、配置等状态不会自动重置，第二次运行时会重复注册之前的示例，导致输出中出现4个示例（实际仅2个）的异常。

修复方案 #

1. 统一持久化文件路径 #

修改spec_helper.rb，将持久化文件路径与runner脚本保持一致：

RSpec.configure do |config|
  config.example_status_persistence_file_path = "logs/RunResults.txt"
end

2. 使用子进程执行RSpec #

修改rspec_runner.rb，用子进程替代同一进程内的重复调用，彻底避免状态污染：

test_results_file = "#{File.expand_path(Dir.pwd)}/logs/RunResults.txt"
File.delete(test_results_file) if File.exist?(test_results_file)

# 第一次运行
system("rspec -r spec_helper --order defined --format CustomFormatter --format documentation")

# 第二次运行失败用例
system("rspec -r spec_helper --order defined --format CustomFormatter --format documentation --only-failures")

如果你的项目使用bundle管理依赖，将命令改为bundle exec rspec ...即可。

验证效果 #

修复后，第二次运行--only-failures只会执行失败的那个示例，输出不会出现重复用例，且仅重试失败测试。

内容的提问来源于stack exchange，提问作者kroe761