Temporal Activity触发context deadline exceeded错误排查求助
问题排查:Temporal Activity执行DynamoDB批量写入时触发
context deadline exceeded错误 在Kubernetes Pod中运行多个Temporal Workflow,其中某个Workflow启动的Activity会向DynamoDB发起大量BatchWriteItem调用(单次Activity执行近4万次请求)。该Activity的批量写入逻辑代码片段如下:
av, err := attributevalue.MarshalMap(msg) if err != nil { return err } writeRequests = append(writeRequests, types.WriteRequest{ PutRequest: &types.PutRequest{ Item: av, }, }) if len(writeRequests) == maxBatchSize { input := &dynamodb.BatchWriteItemInput{ RequestItems: map[string][]types.WriteRequest{ repo.tableName: writeRequests, }, } out, err := repo.store.BatchWriteItem(ctx, input) if err != nil { return err } }
执行该Workflow时,Activity正常运行约10分钟后,Pod中突然出现以下错误日志:
2025/03/17 00:02:40 INFO Task processing failed with error Namespace <namespace> TaskQueue <queue-name> WorkerID 1@kubernetes-worker-pod-name@ WorkerType ActivityWorker Error context deadline exceeded
已将Temporal Activity的StartToClose超时设置为2小时,因此该错误并非由此导致。检查Temporal Worker Pod的CPU和内存使用情况均在限制范围内,且DynamoDB控制台中未出现写入请求限流情况。目前无法确定错误原因,寻求排查方向。
更新信息
在Temporal Activity的调用栈中发现以下错误信息:
coroutine root [blocked on chan-2.Receive]: go.temporal.io/sdk/internal.(*decodeFutureImpl).Get(0xc05ecce318, {0x18f01f8, 0xc0007f0300}, {0x13f11c0, 0xc05ecce180}) /go/src/app/vendor/go.temporal.io/sdk/internal/internal_workflow.go:1588 +0x3e github.com/twilio-internal/comms-api-broadcast-internal-api/internal/temporal/workflows.IngressWorkflow({0x18f01f8?, 0xc0007f0240?}, {{0xc0001b8150, 0x22}, {0xc0001b8180, 0x2a}, {0xc0007a0060, 0x11}, {0xc00035c557, 0x5}}) /go/src/app/internal/temporal/workflows/ingress.go:44 +0x452 reflect.Value.call({0x1460c60?, 0x1746018?, 0x4158c5?}, {0x16d5a9a, 0x4}, {0xc0007f0270, 0x2, 0xc0007f0270?}) /usr/local/go/src/reflect/value.go:581 +0xca6 reflect.Value.Call({0x1460c60?, 0x1746018?, 0x7fefb9f2c208?}, {0xc0007f0270?, 0x46f49d?, 0x7ff000dfb878?}) /usr/local/go/src/reflect/value.go:365 +0xb9 go.temporal.io/sdk/internal.executeFunction({0x1460c60, 0x1746018}, {0xc0006b0480, 0x2, 0x1466620?}) /go/src/app/vendor/go.temporal.io/sdk/internal/internal_worker.go:1940 +0x26b go.temporal.io/sdk/internal.(*workflowEnvironmentInterceptor).ExecuteWorkflow(0xc0006c8190, {0x18f01f8, 0xc0007f0210}, 0xc0006a24f8) /go/src/app/vendor/go.temporal.io/sdk/internal/workflow.go:619 +0x150 go.temporal.io/sdk/interceptor.(*tracingWorkflowInboundInterceptor).ExecuteWorkflow(0xc000517860, {0x18f03f0, 0xc00050a600}, 0xc0006a24f8) /go/src/app/vendor/go.temporal.io/sdk/interceptor/tracing_interceptor.go:449 +0x2ca go.temporal.io/sdk/internal.(*workflowExecutor).Execute(0xc0007e2180, {0x18f03f0, 0xc00050a600}, 0xc000802440) /go/src/app/vendor/go.temporal.io/sdk/internal/internal_worker.go:835 +0x28b go.temporal.io/sdk/internal.(*syncWorkflowDefinition).Execute.func1({0x18f01f8, 0xc000517920}) /go/src/app/vendor/go.temporal.io/sdk/internal/internal_workflow.go:556 +0xc6
不确定该调用栈在当前场景下的含义。
内容的提问来源于stack exchange,提问作者Sai Krishna
相关产品推荐
相关产品推荐

