AWS DataPipeline报错：S3提前生成空$folder$文件如何解决？

Let’s break down your problem and walk through actionable steps to fix this annoying conflict:

First: Figure out what’s creating the premature $folder$ file #

In Amazon S3, $folder$ objects are just 0-byte placeholder files created when a tool or process explicitly "makes a directory" (since S3 doesn’t use real directories—only object keys with slashes). To find the source:

• Use AWS CloudTrail to track the PutObject event for the problematic $folder$ file:
  • Go to the CloudTrail console, filter for S3 events where the event name is PutObject
  • Match the object key to #{myOutputS3Loc}/dynamoDB/[YYYY-MM-dd-HH-mm-ss]/$folder$ (use a timestamp from a recent problematic run)
  • Check the userIdentity field in the event details to see if it’s DataPipeline itself, a Lambda function, a QuickSight process, or another IAM entity.

Second: Fix based on the source #

Case 1: The $folder$ is created by your DataPipeline #

Some DataPipeline activities automatically create target directories as a pre-processing step, which generates this placeholder. Here’s how to fix it:

• Option 1: Skip pre-creating directories: S3 doesn’t require pre-made paths—it auto-generates them when you upload files. Check your pipeline config for any steps that explicitly create the S3 path and remove them.
• Option 2: Add a cleanup step before export: Insert a ShellCommandActivity right before your DynamoDB export step to delete the $folder$ file. Use this command (replace <your-region> with your AWS region):

  aws s3 rm #{myOutputS3Loc}/dynamoDB/#{format(@scheduledStartTime, 'YYYY-MM-dd-HH-mm-ss')}/$folder$ --region <your-region>
  

  Make sure the IAM role assigned to your DataPipeline has the s3:DeleteObject permission for the target S3 path.

Case 2: The $folder$ is created by another process #

• If it’s a Lambda function or other automated task: Modify that process to either skip creating the $folder$ placeholder, or schedule it to run after your DataPipeline export completes (UTC 2:00:28+).
• If it’s a QuickSight-related process: Check your QuickSight dataset refresh schedules or data source settings. Disable any auto-path-creation features, or adjust the refresh time to run post-export.

Quick Reminder #

S3 doesn’t depend on $folder$ objects to work—deleting these placeholders won’t break legitimate file uploads or access. The path will still be recognized when your export file is uploaded.

内容的提问来源于stack exchange，提问作者jagadish Prabhakar