JTA事务超时故障排查：Oracle 12c与JBoss EAP7环境问题

Alright, let's dive into this transaction timeout issue you're facing under high production load—this is a classic scenario tied to Oracle sequence configuration and JBoss transaction management, especially when dealing with heavy concurrent calls.

Environment Setup #

• Oracle 12c Database
• JBoss EAP 7
• Web service deployed on JBoss, responsible for inserting data into the database
• Batch program calls this web service ~130,000 times within 1 hour via multi-threading

Problem Symptom #

Under high production load, you're seeing frequent Arjuna transaction warnings like these:

2018-04-26 18:20:44,675 +0200 [WARN ] [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffffac110923:-4c44ed1d:5ac9329e:6866ea in state RUN
2018-04-26 18:20:44,675 +0200 [WARN ] [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012095: Abort of action id 0:ffffac110923:-4c44ed1d:5ac9329e:6866ea invoked while multiple threads active within it.
2018-04-26 18:20:44,679 +0200 [WARN ] [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012381: Action id 0:ffffac110923:-4c44ed1d:5ac9329e:6866ea completed with multiple threads - thread default task-48 was in progress with xxx.BaseEntity.getNextValue(BaseEntity.java:28)

This only pops up under heavy load—low-traffic periods or identical test environments don't trigger it. The logs clearly point to the timeout happening when fetching the next value from your sequence, which is defined as:

CREATE SEQUENCE "XXX_S" MINVALUE xxx MAXVALUE xxx INCREMENT BY 1 START WITH xxx CACHE 2 NOORDER NOCYCLE NOPARTITION ;

Root Cause Analysis #

Your initial hunch about sequence lock contention is spot-on. Here's why this is blowing up under load:

1. Tiny Sequence Cache: The CACHE 2 setting means Oracle only pre-fetches 2 sequence values at a time. With 130k calls hitting this sequence every hour, Oracle has to constantly refresh the cache by hitting the data dictionary—and each refresh requires a shared lock on the sequence object. Hundreds of concurrent threads block waiting for this lock, and when the wait exceeds JBoss's default 300-second transaction timeout, you get those Arjuna warnings.
2. Transaction Scope Misalignment: The sequence fetch (getNextValue()) is running inside the JBoss-managed transaction. That means every second spent waiting for the sequence lock adds directly to the transaction's runtime, making timeouts inevitable under heavy concurrency.
3. NOORDER Doesn't Fix Contention: While NOORDER is fine for non-order-critical use cases, it doesn't eliminate the need for Oracle to serialize cache refreshes—so the lock bottleneck remains.

Tuning Solutions #

Let's split this into database and application-side fixes to get to the root of the problem:

Oracle Sequence Tuning (Most Impactful Fix)

• Bump the Cache Size: This is the single biggest change you can make. For your traffic volume, try setting the cache to 1000 (or even 5000 if you don't mind a small gap in sequence values if the database restarts). Update the sequence with:

  ALTER SEQUENCE "XXX_S" CACHE 1000;
  

  A larger cache drastically reduces how often Oracle needs to hit the data dictionary, eliminating almost all lock contention.
• Reassess the ORDER Clause: If your app needs sequence values to match commit order, switch to ORDER—but note this adds minor overhead. If order doesn't matter, keep NOORDER.
• Validate Sequence Constraints: Double-check that your MINVALUE/MAXVALUE are set appropriately for your long-term usage to avoid unexpected sequence exhaustion.

JBoss EAP 7 Transaction Tuning

• Adjust Transaction Timeout (Band-Aid): If you can't fix the sequence cache immediately, increase the transaction timeout to give blocked threads more breathing room. Edit your standalone.xml (or domain.xml for domain mode):

  <subsystem xmlns="urn:jboss:domain:transactions:4.0">
      <core-environment>
          <process-id>
              <uuid/>
          </process-id>
      </core-environment>
      <recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>
      <coordinator-environment default-timeout="600"/> <!-- Changed from 300 to 600 seconds -->
  </subsystem>
  

  Note: This is not a permanent fix—addressing the sequence cache is critical.
• Move Sequence Fetch Outside the Transaction: Refactor your getNextValue() method to run before starting the JBoss transaction. This way, any lock wait for the sequence doesn't count against the transaction's timeout. You'll need to adjust your web service logic to separate sequence generation from the data insertion transaction.
• Tune Thread Pools: Make sure JBoss's thread pool is sized for your load. If the pool is exhausted, requests queue up, adding to latency. Check the urn:jboss:domain:threads:3.0 subsystem in your config to adjust core/max thread counts.

Additional Debugging Steps

• Generate Oracle AWR Reports: Run an AWR report during high load to look for wait events like enq: SQ - contention—this will confirm sequence lock contention is the main bottleneck.
• Enable Arjuna Debug Logs: Temporarily turn on debug logging for Arjuna to get more details on the multi-threaded transaction behavior (though sequence contention is the likely culprit):

  <logger category="com.arjuna.ats.arjuna">
      <level name="DEBUG"/>
  </logger>
  

内容的提问来源于stack exchange，提问作者Thomas