Scientist4J疑难：故意将候选路径设为Null仍无法触发Mismatch

我在使用Scientist4J验证新旧代码路径的响应是否存在不匹配（Mismatch），但即使故意将候选路径的响应设为null，也无法触发Mismatch检测，不清楚哪里配置或逻辑有问题。

调用实验的代码示例 #

//调用候选路径时传入startTime、startTime（而非endtime）以观察Mismatch表现
callWithExperiment(() -> getAnalyticsInfos(startTime, startTime), "Running candidate `getAnalyticsInfoByTime`");

private <T> List<T> callWithExperiment(Callable<List<T>> callable, String logMessage) {
        try {
            final DropwizardMetricsProvider dropwizardMetricsProvider = new DropwizardMetricsProvider();
            Experiment<List<T>> experiment = new Experiment<>("analytics-experiment", dropwizardMetricsProvider);
            experiment.run(
                    () -> callable.call(), // control路径
                    () -> {
                        log.info(logMessage);
                        Random random = new Random();
                        Thread.sleep(random.nextInt(5) * 100);
                        return callable.call(); // candidate路径
             });

              ConsoleReporter reporter = ConsoleReporter.forRegistry((MetricRegistry) experiment.getMetricsProvider().getRegistry())
                .convertRatesTo(TimeUnit.SECONDS)
                .convertDurationsTo(TimeUnit.MILLISECONDS)
                .build();
             reporter.start(10, TimeUnit.MINUTES);
             reporter.report();
             reporter.stop();

        } catch (Exception e) {
            e.printStackTrace();
        }
        return null;
    }

故意引入不匹配的代码 #

private List<AnalyticsInfo> getAnalyticsInfos(long startTime, long endTime) {
        return withClient(client -> {
            GetAnalyticsInfoRequest req = new GetAnalyticsInfoRequest().setStartTime(startTime).setEndTime(endTime);
            GetAnalyticsInfoResponse resp = client.geAnalyticsInfo(req);
            List<AnalyticsInfo> results = resp.getInfo();
//当startTime == endTime时故意设置错误响应
            if(startTime == endTime){
                log.info("Updating results to null");
                results = null;
            }
            if(results != null){
                return results.parallelStream()
                        .map(Transformer::toAnalyticsInfo)
                        .collect(Collectors.toList());
            }
            return null;
        });
    }

实验结果 #

3/31/23 9:39:50 PM =============================================================
-- Counters --------------------------------------------------------------------
scientist.analytics-experiment.candidate.exception
count = 0
scientist.analytics-experiment.mismatch
count = 0
scientist.analytics-experiment.total
count = 1
-- Timers ----------------------------------------------------------------------
scientist.analytics-experiment.candidate
count = 1
sum = 24135238
mean rate = 9.77 calls/second
1-minute rate = 0.00 calls/second
5-minute rate = 0.00 calls/second
15-minute rate = 0.00 calls/second
min = 24.14 milliseconds
max = 24.14 milliseconds
mean = 24.14 milliseconds
stddev = 0.00 milliseconds
median = 24.14 milliseconds
75% <= 24.14 milliseconds
95% <= 24.14 milliseconds
98% <= 24.14 milliseconds
99% <= 24.14 milliseconds
99.9% <= 24.14 milliseconds
scientist.analytics-experiment.control
count = 1
sum = 76235890
mean rate = 9.73 calls/second
1-minute rate = 0.00 calls/second
5-minute rate = 0.00 calls/second
15-minute rate = 0.00 calls/second
min = 76.24 milliseconds
max = 76.24 milliseconds
mean = 76.24 milliseconds
stddev = 0.00 milliseconds
median = 76.24 milliseconds
75% <= 76.24 milliseconds
95% <= 76.24 milliseconds
98% <= 76.24 milliseconds
99% <= 76.24 milliseconds
99.9% <= 76.24 milliseconds

问题排查与解决方案 #

1. 核心问题：Control和Candidate路径执行了相同逻辑 #

当前代码中，experiment.run()的control和candidate都调用了同一个callable.call()，也就是都会执行getAnalyticsInfos(startTime, startTime)。当startTime == endTime时，两者都会返回null，结果完全一致，自然不会触发Mismatch。

2. 修正方案：分离Control和Candidate的代码路径 #

你需要让control调用旧的正确逻辑，candidate调用你要测试的（或故意修改的）逻辑。修改callWithExperiment方法，让它接收两个Callable分别对应control和candidate：

private <T> List<T> callWithExperiment(Callable<List<T>> controlCallable, Callable<List<T>> candidateCallable, String logMessage) {
        try {
            final DropwizardMetricsProvider dropwizardMetricsProvider = new DropwizardMetricsProvider();
            Experiment<List<T>> experiment = new Experiment<>("analytics-experiment", dropwizardMetricsProvider);
            experiment.run(
                    controlCallable, // 旧代码路径（正常逻辑）
                    () -> {
                        log.info(logMessage);
                        Random random = new Random();
                        Thread.sleep(random.nextInt(5) * 100);
                        return candidateCallable.call(); // 新代码路径（或修改后的逻辑）
             });

            // 上报代码保持不变...
        } catch (Exception e) {
            e.printStackTrace();
        }
        return null;
    }

调用时传入不同的逻辑：

// Control用正确的endTime，Candidate用startTime==endTime触发null
callWithExperiment(
    () -> getAnalyticsInfos(startTime, endTime), 
    () -> getAnalyticsInfos(startTime, startTime), 
    "Running candidate `getAnalyticsInfoByTime`"
);

3. 额外检查：确认Scientist4J的匹配器配置 #

默认情况下，Scientist4J使用Objects.equals()比较结果，null和null会被判定为相等，只有当control和candidate返回不同结果（比如正常List和null）时，才会触发Mismatch。只要路径分离后结果不同，就能被检测到。

内容的提问来源于stack exchange，提问作者user1751510