Jacinto 6设备CTS测试异常：adb无响应、结果不稳定排查

I'm running CTS tests on the Jacinto 6 Evaluation Module (ti-jacinto6evm) and encountering several confusing failures and issues:

1. Setup Background:

   • Successfully compiled AOSP and CTS, flashed both to the device without issues, completed CTS configuration as guided.
   • Ran CTS over 10 times, but results are inconsistent each run. The device randomly freezes during test execution.

2. Common Freeze Error:
   When the device freezes, this error appears:

   Reason: 'Failed to receive adb shell test output within 600000 ms. Test may have timed out, or adb connection to device became unresponsive'. Check device logcat for details Device 170090035a700002 shell is unresponsive 05-30 04:52:21 W/TestInvocation: Invocation did not complete due to device 170090035a700002 becoming not available. Reason: Could not find device 170090035a700002
   

3. Test Cases That Trigger Freezes:
   These modules consistently cause the device to freeze:

   CtsPreference2TestCases CtsUiHostTestCases CtsServicesHostTestCases CtsTrustedVoiceHostTestCases CtsTransitionTestCases CtsAppTestCases CtsGraphicsTestCases CtsCameraTestCases CtsWebkitTestCases CtsFragmentTestCases CtsViewTestCases
   

4. Post-Exclusion Issues:
   After excluding the above modules with this command:

   run cts --skip-preconditions --exclude-filter CtsPreference2TestCases --exclude-filter CtsServicesHostTestCases --exclude-filter CtsUiHostTestCases --exclude-filter CtsTrustedVoiceHostTestCases --exclude-filter CtsAppTestCases --exclude-filter CtsGraphicsTestCases --exclude-filter CtsTransitionTestCases --exclude-filter CtsCameraTestCases --exclude-filter CtsWebkitTestCases --exclude-filter CtsFragmentTestCases --plan cts
   

   Two new problems arose:

   Problem 1: Inconsistent Pass/Fail Results Between Runs #

   • First Run: 166 modules passed. For example:

     Testcase name Passed Failed Total executed armeabi-v7a CtsWebkitTestCases 201 12 213
     

   • Second Run: Only 91 modules passed. For the same test case:

     Testcase name Passed Failed Total executed armeabi-v7a CtsWebkitTestCases 80 1 81
     

   Problem 2: TimeoutException During Stuck Tests #

   When CTS hangs on certain test cases, this stack trace is thrown:

   com.android.ddmlib.TimeoutException at com.android.ddmlib.AdbHelper.read(AdbHelper.java:767) at com.android.ddmlib.AdbHelper.read(AdbHelper.java:736) at com.android.ddmlib.AdbHelper.readAdbResponse(AdbHelper.java:222) at com.android.ddmlib.AdbHelper.executeRemoteCommand(AdbHelper.java:456) at com.android.ddmlib.AdbHelper.executeRemoteCommand(AdbHelper.java:382) at com.android.ddmlib.Device.executeShellCommand(Device.java:617) at com.android.tradefed.device.NativeDeviceStateMonitor.waitForDeviceShell(NativeDeviceStateMonitor.java:170) at com.android.tradefed.device.WaitDeviceRecovery.recoverDevice(WaitDeviceRecovery.java:142) at com.android.tradefed.device.NativeDevice.recoverDevice(NativeDevice.java:1720) at com.android.tradefed.device.NativeDevice.performDeviceAction(NativeDevice.java:1661) at com.android.tradefed.device.NativeDevice.runInstrumentationTests(NativeDevice.java:615) at com.android.tradefed.device.NativeDevice.runInstrumentationTests(NativeDevice.java:698) at com.android.tradefed.testtype.InstrumentationTest.runWithRerun(InstrumentationTest.java:797) at com.android.tradefed.testtype.InstrumentationTest.doTestRun(InstrumentationTest.java:740) at com.android.tradefed.testtype.InstrumentationTest.run(InstrumentationTest.java:643) at com.android.tradefed.testtype.AndroidJUnitTest.run(AndroidJUnitTest.java:233) at com.android.compatibility.common.tradefed.testtype.ModuleDef.run(ModuleDef.java:250) at com.android.compatibility.common.tradefed.testtype.CompatibilityTest.run(CompatibilityTest.java:506) at com.android.tradefed.invoker.TestInvocation.runTests(TestInvocation.java:761) at com.android.tradefed.invoker.TestInvocation.prepareAndRun(TestInvocation.java:446) at com.android.tradefed.invoker.TestInvocation.performInvocation(TestInvocation.java:300) at com.android.tradefed.invoker.TestInvocation.invoke(TestInvocation.java:886) at com.android.tradefed.command.CommandScheduler$InvocationThread.run(CommandScheduler.java:567)
   

Question: What's causing these failures and freezes?

Let's break down what's likely happening here and how to address each issue:

1. Random Device Freezes & ADB Disconnects #

The core problem here is almost certainly underlying hardware/software stability issues on the Jacinto 6 EVM when running resource-intensive CTS tests. Here's why:

• The test modules that trigger freezes (like CtsGraphicsTestCases, CtsCameraTestCases, CtsUiHostTestCases) are all heavy on system resources—they push the GPU, camera hardware, or system services to their limits.
• The "device not found" error after a freeze suggests the board is either crashing and rebooting (losing ADB connection) or locking up completely, requiring a hard reset.

Fixes to Try: #

• Check for Kernel/Platform Bugs: Jacinto 6 is a relatively older platform, so there might be known stability issues with the AOSP kernel you're using. Verify you're on the latest supported kernel/device tree for this EVM.
• Adjust Test Resource Limits: Some CTS tests might be requesting more resources (like GPU memory) than the EVM can provide. Try adding the --disable-animations flag to your CTS run command to lighten UI-related load, or modify test configurations to reduce resource demands.
• Power Supply Check: Ensure the EVM is receiving a stable, adequate power supply. Underpowering can cause random freezes when the device is under heavy load.
• Capture Freeze Logs: Run adb logcat -v threadtime > freeze_log.txt in a separate terminal while running CTS. Even if the device disconnects, you might capture the crash log right before the freeze to pinpoint the exact failure (e.g., a kernel panic, GPU driver crash).

2. Inconsistent Pass/Fail Results Between Runs #

This is a direct side effect of the device's instability. Here's why it happens:

• When the device freezes mid-test, CTS marks incomplete tests as failed or skips them entirely. Since freezes are random, each run will have a different set of interrupted tests.
• The --skip-preconditions flag bypasses setup steps that ensure the device is in a clean state between runs. Leftover app data, broken system services, or cached resources can cause previously passing tests to fail.

Fixes to Try: #

• Remove --skip-preconditions: Let CTS run its pre-test setup (like clearing app data, resetting system settings, checking device state). This ensures each test run starts from a consistent baseline.
• Reset Device Between Runs: After each CTS run, perform a factory reset or reboot the device to clear any lingering state that might affect subsequent tests.
• Run Tests in Isolation: Instead of running the full CTS plan, run individual modules one at a time. This helps you identify exactly which specific test cases are flaky due to instability, rather than grouping them with interrupted tests.

3. TimeoutException Stack Trace #

This exception is just a symptom of the device becoming unresponsive. The CTS framework waits for a response from the device's ADB shell, but the device has frozen or crashed, so no response comes within the timeout window.

Fix: #

Address the root stability issues mentioned above, and this exception will disappear. While you can adjust the ADB timeout value in the CTS configuration (look for adb_timeout_ms in tradefed config files), this is a band-aid—fixing the underlying freeze is the real solution.

Final Notes #

Jacinto 6 is a specialized industrial platform, not a consumer device, so CTS compatibility might not be fully optimized out of the box. Check the vendor's documentation for any CTS-specific patches or recommended configurations for this EVM. You might also need to report stability issues to the vendor if they're related to platform-specific drivers or kernel code.

内容的提问来源于stack exchange，提问作者shubhamA