Scrapy框架添加代理后出现TypeError错误求助

问题描述 #

为Python Scrapy爬虫添加代理功能，设置meta={'proxy':'103.42.162.50:8080'}后运行抛出TypeError，错误提示to_bytes must receive a str or bytes object, got NoneType，完整错误信息如下：

File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 279, in _get_agent
    proxyScheme, proxyNetloc, proxyHost, proxyPort, proxyParams = _parse(proxy)
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\scrapy\core\downloader\webclient.py", line 39, in _parse
    return _parsed_url_args(parsed)
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\scrapy\core\downloader\webclient.py", line 20, in _parsed_url_args
    host = to_bytes(parsed.hostname, encoding="ascii")
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\scrapy\utils\python.py", line 108, in to_bytes
    raise TypeError('to_bytes must receive a str or bytes '
TypeError: to_bytes must receive a str or bytes object, got NoneType
2023-03-12 02:47:32 [scrapy.core.scraper] ERROR: Error downloading <GET https://dvlaregistrations.dvla.gov.uk/search/results.html?search=N11CKY&amp;action=index&amp;pricefrom=0&amp;priceto=&amp;prefixmatches=&amp;currentmatches=&amp;limitprefix=&amp;limitcurrent=&amp;limitauction=&amp;searched=true&amp;openoption=&amp;language=en&amp;prefix2=Search&amp;super=&amp;super_pricefrom=&amp;super_priceto=>

爬虫代码如下：

import scrapy
from scrapy.crawler import CrawlerProcess
import pandas as pd
import scrapy_xlsx

itemList=[]
class plateScraper(scrapy.Spider):
    name = 'scrapePlate'
    allowed_domains = ['dvlaregistrations.dvla.gov.uk']
    FEED_EXPORTERS = {'xlsx': 'scrapy_xlsx.XlsxItemExporter'}
    custom_settings = {'FEED_EXPORTERS' :FEED_EXPORTERS,'FEED_FORMAT': 'xlsx','FEED_URI': 'output_r00.xlsx', 'LOG_LEVEL':'INFO','DOWNLOAD_DELAY': 0}
    DOWNLOADER_MIDDLEWARES = {
        'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 1
    }

    def start_requests(self):
        df=pd.read_excel('data.xlsx')
        columnA_values=df['PLATE']
        for row in columnA_values:
            global  plate_num_xlsx
            plate_num_xlsx=row
            base_url =f"https://dvlaregistrations.dvla.gov.uk/search/results.html?search={plate_num_xlsx}&action=index&pricefrom=0&priceto=&prefixmatches=&currentmatches=&limitprefix=&limitcurrent=&limitauction=&searched=true&openoption=&language=en&prefix2=Search&super=&super_pricefrom=&super_priceto="
            url=base_url
            yield scrapy.Request(url,callback=self.parse, cb_kwargs={'plate_num_xlsx': plate_num_xlsx},meta={'proxy':'103.42.162.50:8080'})

    def parse(self, response, plate_num_xlsx=None):
        plate = response.xpath('//div[@class="resultsstrip"]/a/text()').extract_first()
        price = response.xpath('//div[@class="resultsstrip"]/p/text()').extract_first()

        try:
            a = plate.replace(" ", "").strip()
            if plate_num_xlsx == plate.replace(" ", "").strip():
                item = {"plate": plate_num_xlsx, "price": price.strip()}
                itemList.append(item)
                print(item)
                yield item
            else:
                item = {"plate": plate_num_xlsx, "price": "-"}
                itemList.append(item)
                print(item)
                yield item
        except:
            item = {"plate": plate_num_xlsx, "price": "-"}
            itemList.append(item)
            print(item)
            yield item

process = CrawlerProcess()
process.crawl(plateScraper)
process.start()

import winsound
winsound.Beep(555,333)

解决方案 #

核心原因 #

Scrapy的HttpProxyMiddleware解析代理地址时，要求地址必须包含协议前缀（http://或https://）。如果只写IP:端口，Scrapy无法正确解析出hostname，导致parsed.hostname为None，进而触发to_bytes的类型错误。

具体修复步骤 #

1. 给代理地址添加协议前缀
   将meta={'proxy':'103.42.162.50:8080'}修改为：

   meta={'proxy':'http://103.42.162.50:8080'}
   

   若代理支持HTTPS，也可以写https://前缀，两者都能正常工作。

2. 移除不必要的中间件配置
   HttpProxyMiddleware是Scrapy默认启用的中间件，无需手动在DOWNLOADER_MIDDLEWARES中设置优先级，删除以下代码避免潜在冲突：

   DOWNLOADER_MIDDLEWARES = {
       'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 1
   }
   

3. 额外建议

   • 免费代理稳定性差，建议在使用前先通过简单的请求校验代理是否可用；
   • 若目标站点是HTTPS，优先选择支持HTTPS的代理，避免出现加密层解析问题；
   • 可以考虑使用代理池自动切换有效代理，提升爬虫稳定性。

内容的提问来源于stack exchange，提问作者xlmaster