Scrapy报TypeError: to_bytes需str/bytes对象，传入NoneType的问题排查

报错信息 #

TypeError: to_bytes must receive a str or bytes object, got NoneType.

爬虫脚本 #

# lbc_ads_to_scrap.py

import scrapy


class LbcAdsToScrapSpider(scrapy.Spider):
    name = 'lbc_ads_to_scrap'
    allowed_domains = ['aia-immobilier.fr']
    
    def start_requests(self):
        yield scrapy.Request(
            'https://aia-immobilier.fr/fr/',
            meta={
                'proxy':'85.135.67.143'
            }

        )

    def parse(self, response):
        yield {
            'balise_h1': response.xpath('//h1')
        }

完整错误栈 #

2024-01-04 17:31:36 [scrapy.core.scraper] ERROR: Error downloading <GET https://aia-immobilier.fr/fr/>
Traceback (most recent call last):
File "C:\Users\kgonc\anaconda3\envs\scraping\lib\site-packages\twisted\internet\defer.py", line 1656, in _inlineCallbacks
result = current_context.run(
File "C:\Users\kgonc\anaconda3\envs\scraping\lib\site-packages\twisted\python\failure.py", line 489, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "C:\Users\kgonc\anaconda3\envs\scraping\lib\site-packages\scrapy\core\downloader\middleware.py", line 49, in process_request
return (yield download_func(request=request, spider=spider))
File "C:\Users\kgonc\anaconda3\envs\scraping\lib\site-packages\scrapy\utils\defer.py", line 67, in mustbe_deferred
result = f(*args, **kw)
File "C:\Users\kgonc\anaconda3\envs\scraping\lib\site-packages\scrapy\core\downloader\handlers\__init__.py", line 75, in download_request
return handler.download_request(request, spider)
File "C:\Users\kgonc\anaconda3\envs\scraping\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 65, in download_request
return agent.download_request(request)
File "C:\Users\kgonc\anaconda3\envs\scraping\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 327, in download_request
agent = self._get_agent(request, timeout)
File "C:\Users\kgonc\anaconda3\envs\scraping\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 279, in _get_agent
proxyScheme, proxyNetloc, proxyHost, proxyPort, proxyParams = _parse(proxy)
File "C:\Users\kgonc\anaconda3\envs\scraping\lib\site-packages\scrapy\core\downloader\webclient.py", line 40, in _parse
return _parsed_url_args(parsed)
File "C:\Users\kgonc\anaconda3\envs\scraping\lib\site-packages\scrapy\core\downloader\webclient.py", line 21, in _parsed_url_args
host = to_bytes(parsed.hostname, encoding="ascii")
File "C:\Users\kgonc\anaconda3\envs\scraping\lib\site-packages\scrapy\utils\python.py", line 107, in to_bytes
raise TypeError('to_bytes must receive a str or bytes '
TypeError: to_bytes must receive a str or bytes object, got NoneType

问题描述 #

已检查目标URL和代理地址，未发现明显问题，请求定位错误原因。

错误原因及解决方案 #

错误原因 #

Scrapy的代理配置必须包含协议头（http://或https://）。当前代理地址85.135.67.143缺少协议头，导致Scrapy内部解析代理URL时无法识别主机名（hostname），返回None，最终触发to_bytes的类型错误。

解决方案 #

在代理地址前添加协议头，若代理为HTTP协议则用http://，HTTPS协议则用https://，修改后的start_requests方法如下：

def start_requests(self):
    yield scrapy.Request(
        'https://aia-immobilier.fr/fr/',
        meta={
            'proxy':'http://85.135.67.143'
        }
    )

如果代理需要指定端口，需一并添加（例如http://85.135.67.143:8080），否则会使用对应协议的默认端口。

内容的提问来源于stack exchange，提问作者kévin Goncalves