如何用JMeter的JSR223元件实现100K条CSV数据分批发送JSON请求？

Got it, let’s break down exactly how to pull off this batch CSV processing and JSON request setup in JMeter using JSR223 elements. I’ve tackled similar large-scale batch tasks before, so here’s a practical, step-by-step approach:

Option 1: Load All CSV Data at Once (Simpler for 100k Rows) #

This method reads the entire CSV into memory first, then splits it into 5,000-row batches and sends each as a JSON request. It’s straightforward and works well for 100k rows (just make sure JMeter has enough heap space).

Step 1: Set Up Your Test Plan #

Start with a basic test plan structure:

• Add a Thread Group (use 1 thread, since we’re processing batches sequentially)
• Add a JSR223 Sampler (this will handle reading the CSV and sending batches)
• Add a View Results Tree (for debugging and verifying requests/responses)

Step 2: Groovy Code for Batch Processing #

Paste this into the JSR223 Sampler (use Groovy as the language—it’s far more efficient than other options for large datasets):

// Configure your parameters here
def csvFilePath = "/path/to/your/codes.csv" // Replace with your actual CSV path
def batchSize = 5000
def apiEndpoint = "https://your-target-api.com/endpoint" // Replace with your API URL

// Read all codes from CSV (trim whitespace to avoid invalid entries)
def allCodes = []
new File(csvFilePath).eachLine { line ->
    allCodes.add(line.trim())
}

// Process in batches
for (int i = 0; i < allCodes.size(); i += batchSize) {
    // Calculate end index for the current batch (handle last partial batch)
    def endIdx = Math.min(i + batchSize, allCodes.size())
    def currentBatch = allCodes.subList(i, endIdx)

    // Build the JSON request body
    def jsonBuilder = new groovy.json.JsonBuilder()
    jsonBuilder {
        Codes currentBatch.collect { code ->
            [CodeName: code]
        }
    }
    def jsonBody = jsonBuilder.toString()

    // Send the HTTP POST request
    def httpClient = org.apache.http.impl.client.HttpClients.createDefault()
    def httpPost = new org.apache.http.client.methods.HttpPost(apiEndpoint)
    httpPost.setHeader("Content-Type", "application/json")
    httpPost.setEntity(new org.apache.http.entity.StringEntity(jsonBody))

    // Execute request and log response (adjust logging as needed)
    def response = httpClient.execute(httpPost)
    def responseBody = org.apache.http.util.EntityUtils.toString(response.getEntity())
    log.info("Batch ${(i / batchSize) + 1} sent | Rows: ${currentBatch.size()} | Status: ${response.getStatusLine().getStatusCode()}")

    // Clean up resources
    response.close()
    httpClient.close()

    // Optional: Add a delay between batches to avoid overwhelming the API
    // sleep(1000) // 1-second pause, adjust based on your API's rate limits
}

Key Tips for This Option #

• Heap Space: 100k rows won’t take too much memory, but to be safe, adjust JMeter’s heap size in jmeter.bat/jmeter.sh (e.g., set -Xmx4g to allocate 4GB of RAM).
• Error Handling: Wrap the HTTP request logic in a try-catch block to handle network errors or invalid CSV rows without crashing the test.

Option 2: Stream CSV Rows (Memory-Friendly for Extra Large Datasets) #

If you’re worried about loading all 100k rows into memory at once, use this streaming approach. It reads rows one at a time, accumulates them into batches, and sends requests when the batch hits 5,000 rows.

Step 1: Test Plan Setup #

Add these elements to your Thread Group:

1. CSV Data Set Config:
   • Filename: /path/to/your/codes.csv
   • Variable Names: code
   • Recycle on EOF: False
   • Stop thread on EOF: True
2. Counter:
   • Start: 1
   • Increment: 1
   • Reference Name: rowCounter
3. JSR223 PreProcessor:
   • This will accumulate rows into batches and trigger requests when ready
4. HTTP Request:
   • Method: POST
   • Body Data: ${requestBody}
5. View Results Tree

Step 2: JSR223 PreProcessor Code #

Paste this Groovy code into the preprocessor:

// Initialize batch list on first run
if (!vars.getObject("currentBatch")) {
    vars.putObject("currentBatch", [])
}

def currentBatch = vars.getObject("currentBatch")
def currentRow = vars.get("code")?.trim()

// Add current row to batch if it's not empty
if (currentRow) {
    currentBatch.add([CodeName: currentRow])
}

def batchSize = 5000
def totalRows = 100000 // Replace with your actual total row count
def currentCount = vars.get("rowCounter") as int

// Send batch if we hit the size limit or it's the last row
if (currentBatch.size() == batchSize || currentCount == totalRows) {
    // Build JSON body
    def jsonBuilder = new groovy.json.JsonBuilder()
    jsonBuilder {
        Codes currentBatch
    }
    vars.put("requestBody", jsonBuilder.toString())

    // Reset batch for next iteration
    currentBatch.clear()
} else {
    // Clear request body so HTTP Request doesn't send empty payloads
    vars.remove("requestBody")
}

Step 3: Configure HTTP Request #

In the HTTP Request element, set:

• Method: POST
• Server Name or IP: Your API’s host
• Path: Your API’s endpoint path
• Body Data: ${requestBody}
• Add a HTTP Header Manager with a Content-Type header set to application/json

Final Notes #

• Always test with a small subset of your CSV first to verify the JSON format and API response.
• Use Groovy exclusively for JSR223 elements in JMeter—it’s optimized for performance and avoids the overhead of other languages like JavaScript.
• Check JMeter’s jmeter.log file for any errors in your Groovy code if things don’t work as expected.

内容的提问来源于stack exchange，提问作者Santosh sanwal