TestNG&Allure：首次失败后终止测试执行并标记为失败

Got it, let's tackle this problem step by step! You want your flaky test to stop running as soon as it fails once, and make sure Allure marks it as red (failed) instead of green (flaky). Here's how to do it:

方案1：直接在测试方法中添加线程安全的失败检查（简单直接） #

This approach works great for individual tests without needing extra listener classes. We'll use an atomic boolean to track failures safely in a multi-threaded environment:

import org.testng.Assert;
import org.testng.annotations.Test;
import java.util.concurrent.atomic.AtomicBoolean;

public class FlakyTestExample {
    // AtomicBoolean ensures thread safety with threadPoolSize=3
    private final AtomicBoolean testHasFailed = new AtomicBoolean(false);

    @Test(invocationCount = 20, threadPoolSize = 3)
    public void unstableFlakyTest() {
        // Check if test already failed before running logic
        if (testHasFailed.get()) {
            Assert.fail("Test failed in a previous invocation, aborting this run");
        }

        try {
            // Replace with your actual test logic
            // Simulate intermittent failure (1 in 20 chance)
            if (Math.random() < 0.05) {
                throw new RuntimeException("Intermittent bug triggered!");
            }

            // Other test steps...
            System.out.println("Invocation passed");
        } catch (Throwable t) {
            // Only let the first failing thread set the failure flag
            if (testHasFailed.compareAndSet(false, true)) {
                throw t; // Throw original exception to mark this invocation as failed
            } else {
                // Mark subsequent invocations as failed to avoid Allure's flaky tag
                Assert.fail("Test failed in an earlier invocation", t);
            }
        }
    }
}

Why this works: #

• Thread safety: AtomicBoolean guarantees only the first failing thread sets the failure flag, even with parallel execution.
• Stop on first failure: All subsequent invocations will immediately fail instead of running the full test logic.
• Allure report fix: Since all invocations either pass entirely or fail entirely (after the first failure), Allure won't see a mix of passes and failures. It will mark the test as red failed instead of green flaky.

方案2：自定义TestNG监听器（通用，适合 multiple tests） #

If you have multiple flaky tests that need this behavior, a reusable listener will save you duplicate code:

1. Create the listener class #

import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestResult;
import org.testng.SkipException;
import java.util.concurrent.atomic.AtomicBoolean;

public class StopOnFirstFailureListener implements IInvokedMethodListener {
    // Global flag to track failures across all invocations
    private final AtomicBoolean globalTestFailed = new AtomicBoolean(false);

    @Override
    public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {
        // Skip and mark as failed if test already failed once
        if (globalTestFailed.get()) {
            testResult.setStatus(ITestResult.FAILURE);
            throw new SkipException("Test already failed in a previous invocation");
        }
    }

    @Override
    public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
        // Set global flag if current invocation failed
        if (testResult.getStatus() == ITestResult.FAILURE) {
            globalTestFailed.set(true);
        }
    }
}

2. Register the listener in your test class #

import org.testng.Assert;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(StopOnFirstFailureListener.class)
public class FlakyTestWithListener {
    @Test(invocationCount = 20, threadPoolSize = 3)
    public void unstableTest() {
        // Your test logic here
        if (Math.random() < 0.05) {
            throw new RuntimeException("Intermittent bug triggered!");
        }
        System.out.println("Invocation passed");
    }
}

Pro tip: #

Add a quick failure check at the start of your test method to terminate running threads faster, instead of waiting for the listener to intercept the next invocation.

Key Reason Your Original Setup Failed #

When your test failed in the middle of invocations but passed later, Allure saw a mix of success and failure results, so it labeled the test as "Flaky" (green). By ensuring all invocations fail after the first failure, we force the test's final status to FAILED, which Allure will display as red.

内容的提问来源于stack exchange，提问作者Igor Klimchuk