Akka持久化恢复超时求助：测试因RecoveryTimedOut失败

Let me walk you through practical, actionable steps to figure out and resolve this recovery timeout issue you're facing:

• Start with Cassandra Connectivity and Performance Checks
  Your logs show you're using Cassandra as the snapshot store, and that's the first place to look. The Netty epoll warning is mostly a performance note, but the real red flag is the 30-second timeout waiting for a snapshot.

  • First, confirm your test Cassandra instance is up and responsive. Use nodetool status to check node health, and run a simple CQL query (like SELECT * FROM your_keyspace.snapshots LIMIT 1;) via cqlsh to see how long it takes to return. Slow responses here directly cause recovery timeouts.
  • Check Cassandra's own logs for signs of trouble—slow queries, connection timeouts, or resource bottlenecks (like low memory or CPU). Test environments often skimp on resources, which can cripple Cassandra's performance.
  • Verify that your Akka Persistence Cassandra plugin is configured with the correct keyspace, contact points, and credentials. A misconfiguration could lead to failed or delayed snapshot reads.

• Tweak Akka Persistence Recovery Timeouts
  The default 30-second recovery window might be too tight for your test setup, especially if Cassandra is running on underpowered hardware. Adjust these settings in your Akka config:

  akka.persistence {
    recovery {
      timeout = 60s # Extend this to give Cassandra more time to respond
    }
  }
  
  akka.persistence.cassandra {
    snapshot {
      read-timeout = 20s # Adjust snapshot-specific read timeout too
    }
  }
  

  Start with a longer timeout to see if the issue goes away—if it does, you know the root cause is slow snapshot retrieval.

• Check if Snapshots Exist (or Should Exist)
  Your logs mention "Last known sequence number [0]" for both actors. That means either:

  • This is the first time these actors are starting, so there's no snapshot to load. But Akka should handle this gracefully without timing out. Double-check that Cassandra's keyspace and snapshot table were created correctly (the plugin usually does this automatically, but misconfigurations can prevent it).
  • Snapshots should exist but weren't saved properly. Look back at earlier logs for errors related to snapshot persistence, or query Cassandra's snapshots table directly to see if there are records for the persistence IDs user-email-indexer and /sharding/userdataCoordinator.

• Validate Cluster Singleton Startup Order
  Since UniquelyIndexingActor is a cluster singleton, its startup depends on a stable cluster. If it tries to start recovery before the cluster is fully initialized or before the Cassandra connection is established, it can time out.

  • In your test code, add a wait for the cluster to reach a stable state (e.g., wait for MemberUp events for all nodes) before initializing the singleton.
  • Check logs for the PersistentShardCoordinator—the error there suggests the sharding coordinator is also hitting recovery timeouts, which points to broader cluster or persistence issues.

• Dig Into Dead Letters and Actor Lifecycle
  The dead letters in your logs are a side effect of the recovery timeout: when an actor fails recovery, it might stop, leaving messages like LoadSnapshotFailed undelivered.

  • Review your actor's supervision strategy. If it's stopping the actor on RecoveryTimedOut, try adjusting it to restart the actor instead—this gives it another chance to complete recovery.
  • Check your Akka dispatcher configuration. If the dispatcher thread pool is exhausted, messages (including recovery-related ones) can pile up and cause timeouts. Ensure the dispatcher assigned to persistence actors has enough threads for your test workload.

内容的提问来源于stack exchange，提问作者Pavel Voronin