Mule 3.9.0 CE多线程负载测试出现ConcurrentModificationException问题咨询

Let's break down what's happening here and how to fix it.

Problem Context #

You're running multi-threaded load tests on Mule 3.9.0 CE flows (all using the default processing strategy, all synchronous steps) and hitting a ConcurrentModificationException. The stack trace references AsyncResponseFlowProcessingPhase—even though your flows are fully synchronous—and this issue didn't occur in Mule 3.4.1.

Here's the full stack trace for reference:

java.util.ConcurrentModificationException: null
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442) ~[?:1.8.0_152]
at java.util.HashMap$EntryIterator.next(HashMap.java:1476) ~[?:1.8.0_152]
at java.util.HashMap$EntryIterator.next(HashMap.java:1474) ~[?:1.8.0_152]
at org.mule.module.http.internal.HttpMapParam.resolveMapObjObj(HttpMapParam.java:68) ~[mule-module-http-3.9.0.jar:3.9.0]
at org.mule.module.http.internal.HttpMapParam.resolve(HttpMapParam.java:48) ~[mule-module-http-3.9.0.jar:3.9.0]
at org.mule.module.http.internal.HttpMessageBuilder.resolveParams(HttpMessageBuilder.java:48) ~[mule-module-http-3.9.0.jar:3.9.0]
at org.mule.module.http.internal.listener.HttpResponseBuilder.build(HttpResponseBuilder.java:116) ~[mule-module-http-3.9.0.jar:3.9.0]
at org.mule.module.http.internal.listener.HttpMessageProcessorTemplate.doBuildResponse(HttpMessageProcessorTemplate.java:122) ~[mule-module-http-3.9.0.jar:3.9.0]
at org.mule.module.http.internal.listener.HttpMessageProcessorTemplate.buildResponse(HttpMessageProcessorTemplate.java:112) ~[mule-module-http-3.9.0.jar:3.9.0]
at org.mule.module.http.internal.listener.HttpMessageProcessorTemplate.sendResponseToClient(HttpMessageProcessorTemplate.java:86) ~[mule-module-http-3.9.0.jar:3.9.0]
at org.mule.execution.AsyncResponseFlowProcessingPhase.runPhase(AsyncResponseFlowProcessingPhase.java:79) ~[mule-core-3.9.0.jar:3.9.0]
at org.mule.execution.AsyncResponseFlowProcessingPhase.runPhase(AsyncResponseFlowProcessingPhase.java:36) ~[mule-core-3.9.0.jar:3.9.0]
at org.mule.execution.PhaseExecutionEngine$InternalPhaseExecutionEngine.process(PhaseExecutionEngine.java:114) ~[mule-core-3.9.0.jar:3.9.0]
at org.mule.execution.PhaseExecutionEngine.process(PhaseExecutionEngine.java:41) ~[mule-core-3.9.0.jar:3.9.0]
at org.mule.execution.MuleMessageProcessingManager.processMessage(MuleMessageProcessingManager.java:32) ~[mule-core-3.9.0.jar:3.9.0]
at org.mule.module.http.internal.listener.DefaultHttpListener$1.handleRequest(DefaultHttpListener.java:135) ~[mule-module-http-3.9.0.jar:3.9.0]
at org.mule.module.http.internal.listener.grizzly.GrizzlyRequestDispatcherFilter.handleRead(GrizzlyRequestDispatcherFilter.java:100) ~[mule-module-http-3.9.0.jar:3.9.0]
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) ~[grizzly-framework-2.3.33.jar:2.3.33]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) ~[grizzly-framework-2.3.33.jar:2.3.33]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) ~[grizzly-framework-2.3.33.jar:2.3.33]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) ~[grizzly-framework-2.3.33.jar:2.3.33]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) ~[grizzly-framework-2.3.33.jar:2.3.33]
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) ~[grizzly-framework-2.3.33.jar:2.3.33]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:539) ~[grizzly-framework-2.3.33.jar:2.3.33]
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) ~[grizzly-framework-2.3.33.jar:2.3.33]
at org.mule.module.http.internal.listener.grizzly.ExecutorPerServerAddressIOStrategy.run0(ExecutorPerServerAddressIOStrategy.java:119) ~[mule-module-http-3.9.0.jar:3.9.0]
at org.mule.module.http.internal.listener.grizzly.ExecutorPerServerAddressIOStrategy.access$100(ExecutorPerServerAddressIOStrategy.java:31) ~[mule-module-http-3.9.0.jar:3.9.0]
at org.mule.module.http.internal.listener.grizzly.ExecutorPerServerAddressIOStrategy$WorkerThreadRunnable.run(ExecutorPerServerAddressIOStrategy.java:142) ~[mule-module-http-3.9.0.jar:3.9.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]

Why AsyncResponseFlowProcessingPhase Shows Up (Even with Sync Flows) #

Don't worry—this doesn't mean your flow logic is running asynchronously. The AsyncResponseFlowProcessingPhase is an internal optimization in Mule 3.9.0's HTTP listener:

• In Mule 3.4.1, the HTTP listener sent responses to clients synchronously in the same thread that ran your flow.
• In 3.9.0, after your synchronous flow finishes processing the request, the listener offloads sending the response to a separate thread pool. This frees up the original request thread faster to handle more incoming traffic.
• This async response handling is part of the HTTP module's internals, not related to your flow's processing strategy. That's why you see it in the stack trace even though your flows are fully synchronous.

Root Cause of the ConcurrentModificationException #

The exception is thrown when iterating over a HashMap in HttpMapParam.resolveMapObjObj. Here's why this happens under load:

• HTTP response headers/parameters are stored in a non-thread-safe HashMap by default.
• During load testing, two threads might interact with this map at the same time: your flow thread (modifying response headers/parameters) and the async response thread (iterating over the map to build the response).
• This concurrent read/write triggers the ConcurrentModificationException—a classic issue with non-thread-safe collections.

Fixes to Resolve the Issue #

Try these actionable solutions to fix the problem:

• Use thread-safe collections for response data: Instead of modifying the default HTTP response map directly, create a thread-safe copy (like ConcurrentHashMap) before making changes. For example:

  // Bad: Modifying the non-thread-safe default map
  message.getOutboundProperty("http.response.headers").put("X-Custom-Header", "value");
  
  // Good: Create a thread-safe copy first
  Map<String, String> safeHeaders = new ConcurrentHashMap<>(message.getOutboundProperty("http.response.headers"));
  safeHeaders.put("X-Custom-Header", "value");
  message.setOutboundProperty("http.response.headers", safeHeaders);
  

• Finalize response changes early: Ensure all modifications to response headers or parameters are completed before your flow finishes processing, so there's no overlap with the async response thread's work.
• Upgrade to a newer 3.9.x patch: MuleSoft released patches for this concurrency bug in later 3.9.x versions. Upgrading to the latest available patch for 3.9.0 might resolve the issue without code changes.

内容的提问来源于stack exchange，提问作者Mahmoud Ezzat