如何自动对比两次Jenkins触发的JMeter测试运行的响应时间？

Hey Abhi, great question! Automating the comparison of two JMeter test runs (Run 1 and Run 2) triggered via Jenkins is totally feasible—here’s a practical, step-by-step approach I’ve used in production setups:

Step 1: Ensure JMeter generates parseable result files #

First, you need structured output from each JMeter run to make comparison easy. Here’s how:

• When triggering JMeter via Jenkins, use the -l flag to save raw results to a CSV/JTL file, and the -e -o flags to generate a dashboard report (which includes a pre-summarized stats.csv file with key metrics like average response time, 95th percentile, error rate). Example command:

  jmeter -n -t your_api_test.jmx -l run_${BUILD_NUMBER}.jtl -e -o report_${BUILD_NUMBER}
  

  Using Jenkins' ${BUILD_NUMBER} variable ensures each run’s results have a unique name, so you don’t overwrite previous data.

Step 2: Fetch Run 1’s results into Run 2’s workspace #

To compare, you need Run 1’s files accessible in Run 2’s Jenkins job. Use the Copy Artifacts Plugin (install it first if you haven’t):

• For freestyle jobs: Add a "Copy artifacts from another project" build step. Specify your JMeter job name, select Run 1’s build (use Specific build and input its number, or Last successful build if Run 1 is the prior run), and copy the relevant files (e.g., report_<run1_build_number>/stats.csv) to a folder like ./previous_run/ in Run 2’s workspace.
• For Pipeline jobs, use the copyArtifacts function:

  copyArtifacts(
      projectName: 'Your_JMeter_Test_Job',
      selector: specific('123'), // Replace with Run 1's build number
      filter: 'report_123/stats.csv',
      target: 'previous_run/'
  )
  

Step 3: Automate the comparison (3 options to choose from) #

Option 1: Write a Groovy script (native to Jenkins) #

Since Jenkins runs on Java, Groovy is a natural fit. Create a script like compare_results.groovy to read both stats.csv files, calculate differences, and flag anomalies:

// Load stats files
def run1Stats = new File('./previous_run/stats.csv').readLines()
def run2Stats = new File("./report_${BUILD_NUMBER}/stats.csv").readLines()

// Skip CSV headers and parse data
def run1Data = run1Stats.drop(1).collect { it.split(',') }
def run2Data = run2Stats.drop(1).collect { it.split(',') }

// Define your performance threshold (e.g., 10% increase in response time)
def threshold = 10

println "=== JMeter Run Comparison Results ==="
run1Data.eachWithIndex { row1, idx ->
    def apiName = row1[0]
    def run1Avg = row1[1].toDouble()
    def run2Avg = run2Data[idx][1].toDouble()
    def diffPercent = ((run2Avg - run1Avg) / run1Avg) * 100

    def status = Math.abs(diffPercent) > threshold ? "*ALERT*" : "OK"
    println "${status} API: ${apiName} | Run1 Avg: ${run1Avg}ms | Run2 Avg: ${run2Avg}ms | Difference: ${diffPercent.round(2)}%"
}

Add a "Execute system Groovy script" build step in Jenkins to run this—output will show up in the job console.

Option 2: Use a Python script (flexible for complex comparisons) #

If you prefer Python, write a script to parse the CSV files, compute differences, and even generate a simple HTML report if needed:

import csv

def load_stats(file_path):
    """Load JMeter stats.csv into a dictionary keyed by API name"""
    stats = {}
    with open(file_path, 'r') as f:
        reader = csv.DictReader(f)
        for row in reader:
            stats[row['Label']] = {
                'avg': float(row['Average']),
                'p95': float(row['95% Line']),
                'error_rate': float(row['Error %'])
            }
    return stats

# Load results from both runs
run1 = load_stats('./previous_run/stats.csv')
run2 = load_stats(f"./report_{BUILD_NUMBER}/stats.csv")

threshold = 10

print("=== JMeter Performance Comparison ===")
for api in run1:
    if api not in run2:
        print(f"⚠️ Warning: API '{api}' missing from Run 2 results")
        continue
    
    avg_diff = ((run2[api]['avg'] - run1[api]['avg']) / run1[api]['avg']) * 100
    p95_diff = ((run2[api]['p95'] - run1[api]['p95']) / run1[api]['p95']) * 100

    print(f"\nAPI: {api}")
    print(f"Avg Response Time: Run1={run1[api]['avg']}ms | Run2={run2[api]['avg']}ms | Diff: {avg_diff:.2f}%")
    print(f"95th Percentile: Run1={run1[api]['p95']}ms | Run2={run2[api]['p95']}ms | Diff: {p95_diff:.2f}%")
    
    if abs(avg_diff) > threshold:
        print(f"❌ Average response time exceeded {threshold}% threshold!")

Add a "Execute shell" step to run python compare_results.py in Jenkins.

Option 3: Use Jenkins Performance Plugin (no code needed) #

If you want a no-code solution, install the Performance Plugin:

1. Add a "Publish Performance test result report" post-build action to your Jenkins job.
2. Configure it to read JMeter’s JTL/CSV files (e.g., run_*.jtl).
3. In the "Comparison" section, set "Compare with baseline build" to Run 1’s build number. Define thresholds (e.g., "Mark build as failed if average response time increases by more than 10%").
4. The plugin will automatically generate a comparison report, highlight performance regressions, and fail the build if thresholds are breached.

Step 4: Integrate into a Jenkins Pipeline (optional) #

If you use Pipeline, you can chain all steps into a single workflow:

pipeline {
    agent any
    stages {
        stage('Run JMeter Test') {
            steps {
                sh 'jmeter -n -t your_api_test.jmx -l run_${BUILD_NUMBER}.jtl -e -o report_${BUILD_NUMBER}'
            }
        }
        stage('Fetch Previous Run Results') {
            steps {
                copyArtifacts(
                    projectName: env.JOB_NAME,
                    selector: specific('123'), // Replace with Run 1's build number
                    filter: 'report_123/stats.csv',
                    target: 'previous_run/'
                )
            }
        }
        stage('Compare Results') {
            steps {
                sh 'python compare_results.py'
            }
        }
        stage('Publish Reports') {
            steps {
                publishHTML(target: [
                    reportDir: "report_${BUILD_NUMBER}",
                    reportFiles: 'index.html',
                    reportName: 'JMeter Run 2 Report'
                ])
            }
        }
    }
}

Final Tips #

• Always store historical results (e.g., using Jenkins’ artifact retention policy) so you can compare against older runs if needed.
• Set clear thresholds based on your application’s SLOs (Service Level Objectives) to avoid false positives.
• If you need visual comparisons, extend the Python/Groovy script to generate a bar chart or use the Performance Plugin’s built-in trend graphs.

内容的提问来源于stack exchange，提问作者Abhi