如何为Jenkins环境中的TestCafe测试配置通过率阈值

Absolutely! You’ve got two solid paths to set up that 98% pass rate threshold for your TestCafe tests in Jenkins—either by rolling custom logic with TestCafe’s reporting, or using Jenkins plugins designed for exactly this kind of quality check. Let’s walk through both with tweaks you can drop right into your existing Jenkinsfile:

Option 1: Custom TestCafe Reporting + Jenkins Pipeline Logic #

This approach uses TestCafe’s built-in JSON reporter to generate test results, then adds a custom Groovy script to calculate the pass rate and enforce your threshold. It’s lightweight and doesn’t require extra plugins.

Step 1: Update your TestCafe command to generate a JSON report #

First, modify the test target in your Makefile to output test results to a JSON file. Add the --reporter json:test-results.json flag to your TestCafe command:

# Example Makefile test target
test:
	testcafe chrome tests/ --reporter json:test-results.json

Step 2: Add a threshold check stage to your Jenkinsfile #

Insert a new stage after your Tests stage to parse the JSON report, calculate the pass rate, and fail the build if it falls below 98%. You’ll need the Pipeline Utility Steps plugin installed for the readJSON function:

#!groovy
pipeline {
    environment {
        CI = 'true'
    }
    options {
        buildDiscarder(logRotator(numToKeepStr: '50'))
        disableResume()
        ansiColor('xterm')
    }
    agent none
    stages {
        stage('setup') {
            steps {
                script {
                    cicd.setupBuild()
                }
            }
        }
        stage('Tests') {
            agent any
            steps {
                script {
                    cicd.withSecret(<keys>) {
                        cicd.runMake("test")
                    }
                }
            }
            post {
                cleanup {
                    archiveArtifacts artifacts: "screenshots/**, test-results.json", allowEmptyArchive: true
                }
            }
        }
        # New stage to enforce pass rate threshold
        stage('Verify Test Pass Rate') {
            agent any
            steps {
                script {
                    def testResults = readJSON file: 'test-results.json'
                    def totalTests = testResults.total
                    def passedTests = testResults.passed
                    def passRate = (passedTests / totalTests) * 100

                    echo "✅ Test Pass Rate: ${String.format("%.2f", passRate)}%"

                    if (passRate < 98) {
                        error "❌ Test pass rate (${String.format("%.2f", passRate)}%) is below required 98% threshold. Marking build as failed."
                    }
                }
            }
        }
    }
    post {
        success {
            script {
                cicd.buildSuccess()
            }
        }
        failure {
            script {
                slackSend channel: "#<test-notifications-channel>", color: 'bad', message: "Regression tests failed or unstable <${env.RUN_DISPLAY_URL}|${env.JOB_NAME}>"
                cicd.buildFailure()
            }
        }
    }
}

Option 2: Use Jenkins Plugins (Quality Gates + JUnit) #

If you prefer a more UI-friendly approach with built-in reporting, use the Quality Gates Plugin alongside TestCafe’s JUnit reporter. This gives you visual test dashboards and configurable thresholds.

Step 1: Update TestCafe to generate JUnit reports #

Modify your Makefile’s test target to output a JUnit XML report (most Jenkins test plugins support this format):

# Example Makefile test target
test:
	testcafe chrome tests/ --reporter junit:test-results.xml

Step 2: Add plugin-based threshold checks to your Jenkinsfile #

First, install the JUnit Plugin (for publishing test results) and Quality Gates Plugin (for enforcing thresholds). Then update your Jenkinsfile to publish results and check the pass rate:

#!groovy
pipeline {
    environment {
        CI = 'true'
    }
    options {
        buildDiscarder(logRotator(numToKeepStr: '50'))
        disableResume()
        ansiColor('xterm')
    }
    agent none
    stages {
        stage('setup') {
            steps {
                script {
                    cicd.setupBuild()
                }
            }
        }
        stage('Tests') {
            agent any
            steps {
                script {
                    cicd.withSecret(<keys>) {
                        cicd.runMake("test")
                    }
                }
            }
            post {
                cleanup {
                    archiveArtifacts artifacts: "screenshots/**, test-results.xml", allowEmptyArchive: true
                }
            }
        }
        # New stage to enforce quality gate
        stage('Enforce Quality Gate') {
            agent any
            steps {
                script {
                    // Publish test results to Jenkins first
                    junit 'test-results.xml'
                    // Check pass rate threshold
                    def gatePassed = qualityGates evaluate: [
                        [threshold: 98, type: 'PERCENTAGE_PASSED', operator: 'GREATER_OR_EQUAL']
                    ]
                    if (!gatePassed) {
                        error "❌ Quality gate failed: Test pass rate is below 98%"
                    }
                }
            }
        }
    }
    post {
        success {
            script {
                cicd.buildSuccess()
            }
        }
        failure {
            script {
                slackSend channel: "#<test-notifications-channel>", color: 'bad', message: "Regression tests failed or unstable <${env.RUN_DISPLAY_URL}|${env.JOB_NAME}>"
                cicd.buildFailure()
            }
        }
    }
}

Quick Notes #

• Custom Script Pros: No extra plugin dependencies, full control over logic (e.g., exclude flaky tests from calculations if needed).
• Plugin Pros: Built-in test result visualization, easier to adjust thresholds via Jenkins UI later.

内容的提问来源于stack exchange，提问作者user2451016