GAE Java应用迁移Datastore模式后新部署报认证缺失无效异常

我有一个已稳定运行多年的Google App Engine（Java）应用，其配套Datastore已于2022年6月6日自动迁移至「Cloud Firestore in Datastore mode」。在未做任何相关代码变更的前提下，新部署的版本开始抛出如下异常：

com.google.appengine.api.datastore.DatastoreFailureException: Missing or invalid authentication.
    at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:69)
    at com.google.appengine.api.datastore.DatastoreApiHelper$1.convertException(DatastoreApiHelper.java:127)
    at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:97)
    at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)
    at com.google.appengine.api.datastore.FutureHelper.getInternal(FutureHelper.java:68)
    at com.google.appengine.api.datastore.FutureHelper.quietGet(FutureHelper.java:32)
    at com.google.appengine.api.datastore.BaseQueryResultsSource.getIndexList(BaseQueryResultsSource.java:154)
    at com.google.appengine.api.datastore.BaseQueryResultsSource.loadMoreEntities(BaseQueryResultsSource.java:187)
    at com.google.appengine.api.datastore.BaseQueryResultsSource.loadMoreEntities(BaseQueryResultsSource.java:166)
    at com.google.appengine.api.datastore.QueryResultIteratorImpl.ensureLoaded(QueryResultIteratorImpl.java:146)
    at com.google.appengine.api.datastore.QueryResultIteratorImpl.hasNext(QueryResultIteratorImpl.java:64)
    at com.googlecode.objectify.impl.KeysOnlyIterator.hasNext(KeysOnlyIterator.java:29)
    at com.googlecode.objectify.impl.ChunkIterator.next(ChunkIterator.java:48)
    at com.googlecode.objectify.impl.ChunkIterator.next(ChunkIterator.java:20)
    at com.google.common.collect.MultitransformedIterator.hasNext(MultitransformedIterator.java:52)
    at com.google.common.collect.MultitransformedIterator.hasNext(MultitransformedIterator.java:50)
    at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1105)
    at com.googlecode.objectify.impl.ChunkingIterator.hasNext(ChunkingIterator.java:51)
    at com.google.common.collect.Iterators.addAll(Iterators.java:366)
    at com.google.common.collect.Lists.newArrayList(Lists.java:163)
    at com.googlecode.objectify.util.MakeListResult.translate(MakeListResult.java:22)
    at com.googlecode.objectify.util.MakeListResult.translate(MakeListResult.java:12)
    at com.googlecode.objectify.util.ResultTranslator.nowUncached(ResultTranslator.java:21)
    at com.googlecode.objectify.util.ResultCache.now(ResultCache.java:30)
    at com.googlecode.objectify.util.ResultProxy.invoke(ResultProxy.java:34)
    at com.sun.proxy.$Proxy68.isEmpty(Unknown Source)

异常触发场景 #

业务代码逻辑为加载实体列表后，对返回的列表调用isEmpty()方法时抛出异常，对应代码片段如下：

List<SystemSetting> applicationSettings = ofy().load().type(SystemSetting.class).filter("name", name).list(); 

if (applicationSettings.isEmpty()) { // Exception thrown here

问题特征 #

• 已上线的旧版本Datastore读写操作仍可正常运行，仅新部署版本存在该问题
• 即便使用相同CI服务重新部署旧提交的历史版本也会报错，最初怀疑该问题由Datastore迁移导致
• 曾有一次通过Bitbucket Pipeline CI部署的新版本可正常运行，但在无任何代码、配置改动的前提下重新部署后即失效，前后App Engine版本ID、代码、用户权限配置均完全一致

已尝试的无效排查方案 #

• 重新部署最后已知可正常运行的版本
• 清理memcache缓存
• 从本地开发环境直接部署
• 停止所有运行中实例后重新部署
• 尝试替换使用其他服务账号密钥
• 扩容提升服务账号权限范围
• 清理Bitbucket Pipeline（CI构建端）的缓存
• 剥离Objectify框架，直接调用Datastore API可复现相同问题
• 使用不同用户账号执行重新部署操作，未成功
• 使用全新创建的服务账号执行部署，未成功

排查更新1 #

参考官方文档在本地Java应用中运行如下代码（使用相同服务账号连接线上Datastore），本地可正常执行，但将相同代码部署到对应项目ID的App Engine后即运行失败：

DatastoreOptions datastoreOptions = DatastoreOptions.getDefaultInstance();
Datastore datastore = datastoreOptions.getService();

// A known entity in my live datastore
Key key = datastore.newKeyFactory()
                   .setKind("MyEntityType")
                   .newKey(123123123l);
Entity retrieved = datastore.get(key);

失败时可获取到更详细的报错信息，核心报错为尝试从Compute Engine元数据服务获取默认服务账号安全访问令牌时返回500错误，无法拉取对应令牌接口：

java.io.IOException: Unexpected Error code 500 trying to get security access token from Compute Engine metadata for the default service account: Could not fetch URI /computeMetadata/v1/instance/service-accounts/default/token

完整错误堆栈如下：

com.google.cloud.datastore.DatastoreException: I/O error
    at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:138)
    at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:123)
    at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.lookup(HttpDatastoreRpc.java:173)
    at com.google.cloud.datastore.DatastoreImpl$3.call(DatastoreImpl.java:434)
    at com.google.cloud.datastore.DatastoreImpl$3.call(DatastoreImpl.java:431)
    at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
    at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
    at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
    at com.google.cloud.datastore.DatastoreImpl.lookup(DatastoreImpl.java:430)
...
Caused by: com.google.datastore.v1.client.DatastoreException: I/O error, code=UNAVAILABLE
    at com.google.datastore.v1.client.RemoteRpc.makeException(RemoteRpc.java:171)
    at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:117)
    at com.google.datastore.v1.client.Datastore.lookup(Datastore.java:93)
    at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.lookup(HttpDatastoreRpc.java:171)
    ... 74 more
Caused by: java.io.IOException: Unexpected Error code 500 trying to get security access token from Compute Engine metadata for the default service account: Could not fetch URI /computeMetadata/v1/instance/service-accounts/default/token

    at com.google.auth.oauth2.ComputeEngineCredentials.refreshAccessToken(ComputeEngineCredentials.java:206)
    at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:257)
    at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:254)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
    at com.google.auth.oauth2.OAuth2Credentials$AsyncRefreshResult.executeIfNew(OAuth2Credentials.java:580)
    at com.google.auth.oauth2.OAuth2Credentials.asyncFetch(OAuth2Credentials.java:220)
    at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:170)
    at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
    at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:159)
    at com.google.cloud.http.CensusHttpModule$CensusHttpRequestInitializer.initialize(CensusHttpModule.java:109)
    at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc$1.initialize(HttpDatastoreRpc.java:91)
    at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:95)
    ... 76 more

排查更新2 #

已付费联系Google官方支持，对方已确认该问题与影响部分用户的GAE标准环境Service Agent、访问令牌相关已知问题有关。

内容的提问来源于stack exchange，提问作者matt burns