Python爬虫程序实现爬取文件直接上传至FTP服务器技术求助

Hey there! Let's dive into why your FTP upload isn't working—your code has a key mistake in how you're handling the binary PDF content from the web request. Here's what's going wrong and how to fix it:

The Core Problem #

This line is the culprit:

ftp.storbinary('STOR ' + '/uh/backup/' + fileName + '.pdf',open(r.content))

The open() function expects a file path string, but you're passing r.content—which is raw binary data from the PDF request. That's why you're getting errors related to invalid file paths or encoding mismatches.

The Fix: Use BytesIO to Wrap Binary Content #

You need to convert the binary content into a file-like object that storbinary can read. Python's io.BytesIO module does exactly this. Here's how to adjust your code:

First, add the import at the top of your script:

import io

Then modify the PDF upload block:

for i in range(0, len(a), 2):
    url = defaultInformation['gooktoHome'] + a[i].attrs['href']
    r = requests.get(url, allow_redirects=True)
    fileName = datetime.now().strftime('%Y%m%d%H%M%S')
    
    if a[i].attrs['href'][-3:] == 'pdf':
        full_file_name = f"{fileName}.pdf"
        # Wrap the binary content in a BytesIO object
        pdf_file = io.BytesIO(r.content)
        # Upload to FTP
        ftp.storbinary(f"STOR /uh/backup/{full_file_name}", pdf_file)

Extra Checks to Avoid Future Issues #

• Verify FTP Connection: Make sure you've properly logged into your FTP server before running this code (e.g., ftp.login("your_username", "your_password") after connecting).
• Check Directory Permissions: Confirm that the /uh/backup/ directory exists on the FTP server and your account has write access to it.
• Handle Large Files: If you're scraping very large PDFs, consider streaming the request directly to the FTP server instead of loading the entire file into memory. You can do this by passing r.raw to storbinary (just make sure stream=True is set in requests.get()).
• Avoid Duplicate Filenames: While using timestamps helps, if you're scraping multiple PDFs quickly, there's a tiny chance of filename collisions. You could add a random suffix or hash part of the URL to make filenames unique.

内容的提问来源于stack exchange，提问作者하욱현