Linux下使用curl带凭据批量下载文件遇错误求助

Hey Alyssa, let's walk through troubleshooting this error step by step—curl's "Failed writing body" error usually points to issues with writing files to your system, not just your .netrc credentials. Let's break down the most likely causes and fixes:

1. First, Verify Your Credentials Work for a Single File #

Before diving into bulk downloads, let's confirm your .netrc and cookie setup actually works for one file. Grab a single URL from your Test.txt and run:

curl -v -LJO -n -c ~/.urs_cookies -b ~/.urs_cookies "PASTE_YOUR_SINGLE_URL_HERE"

• The -v flag shows detailed logs. Look for HTTP/1.1 200 OK to confirm you're authenticated successfully. If you see 401 Unauthorized:
  • Double-check your .netrc entry: make sure machine urs.earthdata.nasa.gov login mylogin password mypassword has no typos (especially your username/password).
  • Confirm .netrc permissions are strictly 0600: run ls -l ~/.netrc—it should show -rw------- (only you can read/write it; curl ignores .netrc if permissions are too open).

2. Check Disk Space and Directory Permissions #

The #1 cause of "Failed writing body" is not having enough space or permission to write files:

• Check disk space: Run df -h . to see how much free space is left in your current directory. 1000 files (~1MB each) need at least 1GB of free space—if you're low, clean up or move to a directory with more space.
• Check directory write access: Run ls -ld . to view permissions for your current folder. You need the w (write) permission—if you don't see it, run chmod u+w . to add write access for your user.

3. Fix Filename Issues (Common with CGI URLs) #

Your URLs are CGI endpoints with long, encoded parameters. When using -O, curl tries to extract a filename from the URL, which can result in:

• Filenames that are too long (most Linux filesystems limit names to 255 characters)
• Invalid characters like &, %, or = that cause file creation failures

Instead, use the LABEL parameter from your URLs as the filename (it's already a valid, descriptive name). Replace your bulk command with this loop:

cat Test.txt | tr -d '
' | while read -r url; do
  # Extract the LABEL value from the URL
  filename=$(echo "$url" | sed -n 's/.*LABEL=\([^&]*\).*/\1/p')
  # Download to the extracted filename
  curl -L -n -c ~/.urs_cookies -b ~/.urs_cookies -o "$filename" "$url"
done

This ensures you get clean, valid filenames every time.

4. Limit Concurrent Downloads to Avoid Server Throttling #

NASA's Earthdata servers may block too many simultaneous requests. If you're using xargs, add the -P flag to limit concurrency (e.g., 3 concurrent downloads):

cat Test.txt | tr -d '
' | xargs -n 1 -P 3 bash -c '
  filename=$(echo "$1" | sed -n "s/.*LABEL=\([^&]*\).*/\1/p")
  curl -L -n -c ~/.urs_cookies -b ~/.urs_cookies -o "$filename" "$1"
' _

This prevents overwhelming the server and reduces failed requests.

Final Troubleshooting Tip #

If you still see errors, run the curl command with -v in your loop to see exactly where it's failing. This will tell you if it's an authentication issue, a server error, or a file-writing problem.

内容的提问来源于stack exchange，提问作者Alyssa.r