Google Cloud Dataprep挑战实验室Task3 Dataflow作业执行失败求助

问题概述 #

在完成Google Cloud挑战实验室《Perform Foundational Data, ML, and AI Tasks in Google Cloud: Challenge Lab》时，仅Task3「运行简单Dataprep作业」无法完成：

• 选择Trifacta Photon执行作业可正常运行，但Qwik Labs测试要求必须使用Dataflow运行，不认可该配置
• 严格遵循1个月前的YouTube教程步骤操作仍失败，推测是近期Dataprep、GCP或实验室环境更新引发的兼容性问题
• Dataflow执行时的错误日志：

Dataflow job execution failed: S01:PTableStoreTransformGCS3/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)/Impulse+PTableStoreTransformGCS3/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)+PTableStoreTransformGCS3-TextIO-Write-WriteFiles-GatherTempFileResults-Reify-ReifyViewInGlobalWindow3/PairWithRestriction+PTableStoreTransformGCS3-TextIO-Write-WriteFiles-GatherTempFileResults-Reify-ReifyViewInGlobalWindow3/SplitWithSizing failed., The job failed because a work item has failed 4 times. Look in previous log entries for the cause of each one of the 4 failures. If the logs only contain generic timeout errors related to accessing external resources, such as MongoDB, verify that the worker service account has permission to access the resource's subnetwork. The work item was attempted on these workers: Open logs

解决建议 #

1. 检查Dataflow Worker服务账号权限

   • 确认实验室分配的Dataflow Worker服务账号，是否拥有作业涉及的GCS存储桶读写权限，以及对应子网的访问权限
   • 手动为服务账号添加roles/storage.objectAdmin（针对作业使用的GCS桶）和roles/compute.networkUser（针对子网）权限

2. 调整Dataflow作业配置

   • 尝试提升Worker机器规格，比如从默认的n1-standard-1改为n1-standard-2，增加资源配额
   • 延长作业超时时间，避免因资源初始化缓慢导致的超时失败

3. 重置实验室环境

   • 重新启动挑战实验室实例，清除可能存在的环境残留配置后再次尝试

4. 反馈至Qwik Labs支持

   • 提交问题至Qwik Labs官方支持，说明是近期平台更新导致原有步骤失效，请求修复实验室配置

内容的提问来源于stack exchange，提问作者dimitri.marinho