You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

AWS Batch S3 Lambda任务返回JSON错误及相关技术咨询

S3 Batch任务执行问题及批量文件复制方案咨询

问题背景

需要将大量(最多10万份)文件从S3的一个位置复制到另一个位置,最初考虑使用S3 Batch Copy操作,但它无法修改输出路径,而这对归集文件供Athena后续处理至关重要。因此转而使用S3 Batch Lambda操作,通过Lambda对每个条目发起S3 CopyObject请求,操作速度快且运行正常,但每个条目在报告中都因Lambda返回的JSON格式错误而报错。

构建的Lambda响应JSON结构

{    
    "invocationSchemaVersion": "1.0",    
    "invocationId": "AAAAAAAAAAGQaOzzJcNEbxZje2JpMRjdxh4CsFXUr0+Hdc+gVHtCRyXFH9oR5nxKv7uNh4jTCj8Gb7f/QUZmk2DIagbWXK1Km73jlZ2qKWQnJT5Avg4hVRixjjW+aOrhLj1GQPYmeuVXoiT5L6TgPwrOrRasNLwRTUrwoEiCCuE3zHsOPi8Jm1ai4tLDpzLIzx2Rd5Ye8as1+W2d7h/oviX1Nc1+FsJqjn+RLXChbDdqYcRvqrOfn7UU9zTeoCZFEgJW54xPf3+bVcjfXEOyHkzps/leo0CS4C6KVTsoC+sFPUen4s7S3oxBlbVfZvGntX2XTfLnhLy9y8m3OCJGaDvZu9US22sFaTniIcNOdyDGpcmVkolYZwlLoC8dA8EK80aXdWYgy+x4Nmrlbkjt8c/xWl5lfgTtL4HDcGXcobpsdio9eC4pgGia16+FzoH2CZQam1qO0PT2LTDO8/rioJsyuHvD3f7R9wTGN1JRhuOurbrK/ribHs09eryBrJXaSIeY4e2nk4wFRZuElk4fPlojpCtR5EMVJvl+kA6ZnHom2lFUFaPdeswkUXV1E3MfPqDjUF39IWR2A15im/9JMDkFW1m2129tiL5QyFEUMXkmwZuUEdAQUaBwQzExOgukoLAFxXlsy/b/FfoAATfqqb5q3z3lSN3ioC1uHfSFpTRxPBJ6+HuGYItoYSRXlKPJEHWbaIyP63pdVQJQ2Jq7dcNX7ptm1hbgQ+JAMc0239/MGIY+bRVz6TpH1blu3BDIna7OBJweOZBj3qHH7tDolHInOytmKavoBF6vbunOAjQvEXBEj+6rMuiiiyyw0V5zvinof7YxwmdtyjIetD+IEVqxiXtqc1Pyb3sVEsRuRFLbYawJ9uTLClRSJCUFY4vLQHfSzL7bWDH+D3lJ04w5ujXdPMnG0z4aKy2UioP4xfUku0afWe4pTurZbeTnKKZ358iIxI/1qG0y8FEYv7jrMjpo3dh2ivcCQrmEBgMlDQTS5shNsnUj02M6RNwZlNC5NWSNKBpSsexe9B+LrP5vUBwj8eQxIrYVn9hETgqb8u+SGKZ8H8q2IWhocdps7JdMtP+WP29pH5kPh3VKQEjykDJr8s1Zr++Y3oWWhxCuWUZhrqYje0Bx/D8HW/dVqqTHClTeWC1yg5ff+jKnZ0RKUnO1XzEFt87UXIDbd56p+RgfrEIZovCftoBnHOi1XzHtvlfKhplvHLsGfQN/jgDZV4ooge0QrM8plsRPfuaEA1uHY5UOKcaqKj9Jpp2l6ctJ4OWvQnUzxEyuD2SCg7mFWFCgjsqn783Tib7tZMo+qjLGgRWc4riuAW7SKb+YAAI0rZxH3Z1RuK7KbFCdszMl9xNSptMF6kJfUl/ocRK0AUn3jbguSGPdSe08D4Tx1pqhBiJHpCE1T04PsZEdykte9sei5CZnxcZ5XdEpbADNHe2y9u6vQwxh3QrpJasQJRBhJuKHXntjab5IgZsDzc2Ez0iQlfiHVz9Gp9zkM8NTgC4Auv3T9iYi7rxff3LEoOZKEU1tizGIEP/gThfE8thMytWsLW3d2hgw/B74grX4IzwrM2tyvR2oa3alOLEkDwpKFSDWFYXFMokeMKLTk14re8TpD9cJ0Jf4vN8PNQeTCWFyEp/XISn6f4T8jm6rf1nf7A2OsfZA6gdvrWc7u61F8CMBeBDc8lUp1Rv8zDM5K80xpQt39x5PaBMj66gnM/01onH7g9O3xZN50eiiuT5LVkARbPtiTJi4Dxruu+UVf2e86CjGVQRu1cDtCqlQZ2Y+ylTQXepg/rxK/34KZ3olTXvo0L0yfymbO8e5IX0eXlMJ1OlilBJbssET3ohCTqjMrJDR3IF5raz1xeDoJMTvuZAbIC84ksNl90JJDz1zjkrN2TiUMtD19M0B+TjbyLn/4Tw/bjHrBkpmx2c7DIk0e7L7wG9rThcBT7hjPnHbEk7XNpysssgrtp3m/LR3X+TY+7P8ALMSUAkVMymhyZ2Ur6zmgQyP5VGzJzbx66VZPCPM4+hfBLr1teivvEgAompM2uMxrOx0nv3/29LL/4hlbXSwQO2dk85KshJqzCnWdTbGDUzZIvRMlTo3MgAFyH1vXu/FM/oP8gyBZh+sbcA69UVXMFGSnpouP3wa0t2tBE7hOe9jIHYnb+p+Ng9cMwyh13WqNIstvp5qHb8GDiQGzzb/Z4p7PHALL5JWIhC+zhlnp2TElKC1oii2zRF1/FGre1qVpj4AZOWPmzt+qTfNk9DbdNq66334vMi3tKoDrTuKRmz9pr5ZlQ==",    
    "treatMissingKeysAs": "PermanentFailure",    
    "results": [        
        {            
            "taskId": "AAAAAAAAAAGWxTheY+PQE987jdKDGe9EI4x7Njgb9+4ogcLMgjqtAXk7YuyLCZx138AOKrT9C8huBaHNg4kzvhEJnAIFtvkBUZ6uZ6igTYN0vnsl1DitqineddsGvD6Nt1jldgrshZuEe+fFMt/ak/56ESDW1hE8652hcxrZS6icV/uRtwtj2k3Xmc5IoNZugtx6mw5BhPqErv9yjZMBYzbZ8vVUQo7ccalfVVzGlv2oO8nEVjamBbIdpu+xrzZuD/x4X4epphXqCzuNQpdRUXkL465K+yVs9J0y+bktcQ8au85FCPM+6Ts7fAeUm2DSFKVJRXE9XDeHXdH6wvPDrcU9oC99vyouCGn5lkPwdug6N32Y5/pE9vMlRTSpnAOOWBO698ci/34OVgLxcuLl9rm1Z5e4B2+zZoAJSVmR7+/masL/brpJakUM9tU5hdkp4kAkNR7sxH4z2DwMaM9qfKPPmNhybfQqXVMmqEOtRAAJdUwZNgKtwR0fjH7uGAxxcha8cpCVAMBjU6VJw/bKBjmjFxsoi3oC+Dj2toaxvc8LT3WegdCPKOfZcvZLTq3kajfXQGK0aiBpLL+A2q2IzobafpBUEaApqa4m90LLxOP9TbKreaLOBqbyQKEVEskoYBXlbi6nvCxN4qIIwfXM4QsiMT2spPuen/odWo5HajyiaI/7WSsupY/je2y5opufO4Em/6pw8PXZMwcP3ati+ocTyYIjSX3gpuS9GFUT9Sov8jAGQI71H4pPbBIxopK20BKeb6DT9VRlONECNfQaDFr6ovedTiBMPAjZeeX27VmvRpBMgAY/1fjtX6gEkmrkU7V8vzdvX/T3mInk1TzoI/K2eIE24asmrEsIkECgUB+b54SiXZaZYCVmaiaDZgEnLqq4q+IYo8l6e6f0cZNCbLD/IKslRU7QRJufvCHBMw/YeUv7XG1St170fqgzqfc/gvxlSYxBtewXRdCLk47CtE+wFAfYNXjDT3PRG6DAOl3wyDjjCuJRijoIOkAArnsx5AdoKjhI2JTazl6jdwhk2bmFdC1D2gujNJbqCLjoGQinb/aJMw+qYe8DPSV3YhAFsHQDAUnew5M2/qgIIv7Kps3Z9X6FspzQUFi+AG2yu+qrhJzzVCPPmRY4P8FuRLG8BuA1AhIPTRK1hZsXi/p+4n3S3w199I6TFY/BxDCV9lIu7coHIEv6lNQHh7wS7EWS9or0wz/02lMzf1+j04nOdokwW49IXFTXStXej3ZhdA==",            
            "resultCode": "Succeeded",            
            "resultString": "asdasdasd"        
        }    
    ]
}

收到的错误信息

Invalid JSON returned in Lambda payload: "{"invocationSchemaVersion":"1.0","invocationId":"AAAAAAAAAAG5pvT5OiQJUWXIYVazaz4A04maxTVgE+ByVOSrNfLYyVakaNnKgyxewmpNGtixsr3SV9Aw+txterQjFisOmpnq5BJ/v/TjMW0/L+svjPFW8Vl7N8Ey50HR66rx6h4BGak91Tapr7Xvfk3bPY7T8sECPmcpk5oD2UwTd64uYej5WrZBnlrOIiv1lKSAugx1kxCESMae2ukD31ZzMVZyYknG5vnfoJETiO4F+/iq4peMGVkwzcbuJ0KNbkyEvBbuOkbt3EOuE/721mbB0qcbJIBFpXgc45yw7IsJUyWllYls5l5uL33IBtFastnPHUMqUtv/Y3q6/xx8yjEvyNktxfKyhJf4CgnuC1BkbmaQ0xoruiR4uLkV/zgW2WoanYZEyWtBDGCqKSdGBhsfPTRvLkp8dJapNm/SYEVJ1MuL2NKZWSIaFYCBGa4gc2SSYT8Tp4MFG+zughNUx6yBh7QxKO8+0ChtHjnirJggdxtcdVYaLyerdVI9miEWQKTue9DCLsE0fCzLDFrcgQxi/P87yDDlEH3Ij5L/Go1QsqG+LhAjhZh7170wySWNoiC5L8zytly7QjfdJsT9I4P/kSL4j/Ap8qq9L9cUazLbuGBZBNpZRmNyBYFB3emIotwpTNcFII6xXZHadskMxal4xuDuGmU1GzUtf1KpdHjWTbCoYPgh5ZHTS4mvNgGLUhG+6X5SoX9uL/7vqK7b/yNaaKjP2rdz4EJdfmLKtAn6KCfyeds2yAZKxwKe643jXb0YtJuPGi9C50kb8rokt1Wcgo9W0j4kunADdMWxT3Ta67/yQ5s6kxewsVBHkcgAJiTU0OnelSI5pCPPOfQBx4BxwiXp7Avam9aaruWnx4Q3IlD9YuutEjdM9b4cHgaUhplxDW2Y0ujiRqze9eZsoNCXS5ok98

自定义的C#输入类型

public class S3BatchEvent{
    [JsonProperty("invocationSchemaVersion")]
    public string InvocationSchemaVersion { get; set; }

    [JsonProperty("invocationId")]
    public string InvocationId { get; set; }

    [JsonProperty("job")]
    public Job Job { get; set; }

    [JsonProperty("tasks")]
    public List<Task> Tasks { get; set; }
}

public class Task{
    [JsonProperty("taskId")]
    public string TaskId { get; set; }

    [JsonProperty("s3BucketArn")]
    public string S3BucketArn { get; set; }

    [JsonProperty("s3Key")]
    public string S3Key { get; set; }

    [JsonProperty("s3VersionId")]
    public object S3VersionId { get; set; }
}

核心问题

  1. 构建Lambda响应时存在哪些错误?
  2. .NET SDK中是否有S3 Batch任务对应的现成类型?

其他高效批量复制方案咨询

是否有其他高效方案可实现带动态输出路径的S3批量文件复制?要求任务远程执行且性能优异,避免本地资源不足导致的性能瓶颈。


内容的提问来源于stack exchange,提问作者KorsaR

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.08.17 11:20:37