使用JanusGraph对接AWS Keyspaces添加顶点遇Lock write retry count exceeded错误

问题现象 #

通过Gremlin控制台向对接AWS Keyspaces的JanusGraph添加顶点时，触发Lock write retry count exceeded错误，具体操作及日志详情如下：

Gremlin控制台操作 #

gremlin> :remote connect tinkerpop.server conf/remote.yaml
==>Configured localhost/127.0.0.1:8182

gremlin> :remote console
==>All scripts will now be sent to Gremlin Server - [localhost/127.0.0.1:8182] - type ':remote console' to return to local mode

gremlin> graph
==>standardjanusgraph[cql:[cassandra.us-east-2.amazonaws.com]]
gremlin> g = graph.traversal()

==>graphtraversalsource[standardjanusgraph[cql:[cassandra.us-east-2.amazonaws.com]], standard]
gremlin> g.addV('student').property('name', 'Jeffery').property('GPA', 'A+');

Lock write retry count exceeded

当前配置 #

janusgraph.storage.cql.ssl.enabled: "true"
janusgraph.index.search.hostname: jce-elastic
janusgraph.storage.cql.local-datacenter: "us-east-2"
janusgraph.storage.cql.partitioner-name: DefaultPartitioner
janusgraph.storage.cql.metadata-schema-enabled: "false"
janusgraph.storage.cql.metadata-token-map-enabled: "false"
janusgraph.storage.cql.only-use-local-consistency-for-system-operations: "true"
janusgraph.log.janusgraph.key-consistent: "true"
janusgraph.log.tx.key-consistent: "true"
janusgraph.storage.cql.read-consistency-level: "LOCAL_QUORUM"
janusgraph.storage.cql.write-consistency-level: "LOCAL_QUORUM"
janusgraph.graph.assign-timestamp: "false"

服务端日志 #

jce-janusgraph   | 232996 [gremlin-server-exec-4] WARN  org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker  - Lock write succeeded but took too long: duration PT0.248S exceeded limit PT0.1S
jce-janusgraph   | 233251 [gremlin-server-exec-4] WARN  org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker  - Lock write succeeded but took too long: duration PT0.255S exceeded limit PT0.1S
jce-janusgraph   | 233499 [gremlin-server-exec-4] WARN  org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker  - Lock write succeeded but took too long: duration PT0.248S exceeded limit PT0.1S
jce-janusgraph   | 233747 [gremlin-server-exec-4] ERROR org.janusgraph.graphdb.database.StandardJanusGraph  - Could not commit transaction [1] due to exception
jce-janusgraph   | org.janusgraph.diskstorage.locking.TemporaryLockingException: Temporary locking failure
jce-janusgraph   |      at org.janusgraph.diskstorage.locking.AbstractLocker.writeLock(AbstractLocker.java:306)
jce-janusgraph   |      at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStore.acquireLock(ExpectedValueCheckingStore.java:102)
jce-janusgraph   |      at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.acquireLock(KCVSProxy.java:52)
jce-janusgraph   |      at org.janusgraph.diskstorage.BackendTransaction.acquireIndexLock(BackendTransaction.java:258)
jce-janusgraph   |      at org.janusgraph.graphdb.database.StandardJanusGraph.prepareCommit(StandardJanusGraph.java:634)
jce-janusgraph   |      at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:755)
jce-janusgraph   |      at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1525)
jce-janusgraph   |      at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.doCommit(JanusGraphBlueprintsGraph.java:322)
jce-janusgraph   |      at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:104)
jce-janusgraph   |      at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.commit(JanusGraphBlueprintsGraph.java:300)

问题原因 #

从日志可见，锁写入操作虽成功，但耗时超过了JanusGraph默认的100ms超时限制，多次重试后触发锁重试次数超限错误。核心原因是JanusGraph默认的锁配置未适配AWS Keyspaces的远程网络延迟特性。

解决方案 #

调整JanusGraph的锁相关配置，放宽超时时间和重试次数，适配远程连接延迟：

• 增加锁写入超时时间：将锁操作超时阈值从默认100ms调高至500ms

  janusgraph.lock.consistent-key.write-lock-time: PT0.5S
  

• 增加锁重试次数：将默认3次重试提高至5次

  janusgraph.lock.consistent-key.max-retries: 5
  

• 调整锁重试等待间隔：延长每次重试的等待间隔，避免频繁重试消耗资源

  janusgraph.lock.consistent-key.retry-wait: PT0.2S
  

将以上配置添加到JanusGraph配置文件中，重启Gremlin Server后重新执行操作即可。

辅助排查要点 #

• 确认AWS Keyspaces区域与JanusGraph服务所在区域一致，减少跨区域网络延迟
• 检查网络连接稳定性，排查是否存在丢包或高延迟情况
• 验证一致性级别配置合理性，LOCAL_QUORUM为AWS Keyspaces默认级别，可根据业务需求调整

内容的提问来源于stack exchange，提问作者Abhiram