如何获取拒绝GET请求的网页数据？Tesco油价爬取遇阻求助

我是编程新手，正在学习网页爬取，当前项目是收集超市燃油价格，尝试爬取Tesco的燃油价格JSON数据失败，用curl调试发现服务器拦截请求，以下是相关信息及可行解决方案：

curl调试输出 #

⏺ ~ > curl -v "https://www.tesco.com/fuel_prices/fuel_prices_data.json" --http1.1
*   Trying 96.16.109.100:443...
* Connected to www.tesco.com (96.16.109.100) port 443
* ALPN: curl offers http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: C=GB; L=Welwyn Garden City; jurisdictionCountryName=GB; O=Tesco PLC; businessCategory=Private Organization; serialNumber=00445790; CN=www.tesco.com
*  start date: Mar 20 13:08:28 2024 GMT
*  expire date: Apr 20 13:08:27 2025 GMT
*  subjectAltName: host "www.tesco.com" matched cert's "www.tesco.com"
*  issuer: C=US; O=Entrust, Inc.; OU=See www.entrust.net/legal-terms; OU=(c) 2014 Entrust, Inc. - for authorized use only; CN=Entrust Certification Authority - L1M
*  SSL certificate verify ok.
* using HTTP/1.1
> GET /fuel_prices/fuel_prices_data.json HTTP/1.1
> Host: www.tesco.com
> User-Agent: curl/8.4.0
> Accept: */*

* Recv failure: Connection reset by peer
* LibreSSL SSL_read: LibreSSL/3.3.6: error:02FFF036:system library:func(4095):Connection reset by peer, errno 54
* Closing connection
* Send failure: Broken pipe
* Send failure: Broken pipe
curl: (56) Recv failure: Connection reset by peer

Python爬取代码 #

import requests

URL = "https://www.tesco.com/fuel_prices/fuel_prices_data.json"

page = requests.get(URL)

print(page.text)

运行报错信息 #

Traceback (most recent call last):
  File "/Users/abjohnson/coding/pylab/beautiful_web_scraper.py", line 6, in <module>
    page = requests.get(URL)
  File "/Users/abjohnson/Library/Python/3.9/lib/python/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/Users/abjohnson/Library/Python/3.9/lib/python/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/abjohnson/Library/Python/3.9/lib/python/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/abjohnson/Library/Python/3.9/lib/python/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/Users/abjohnson/Library/Python/3.9/lib/python/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', TimeoutError(60, 'Operation timed out'))

可行解决方案 #

• 添加浏览器请求头：服务器通常通过User-Agent识别爬虫，给请求添加常见浏览器的UA即可绕过基础检测。示例代码：

  import requests
  
  URL = "https://www.tesco.com/fuel_prices/fuel_prices_data.json"
  headers = {
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
  }
  
  response = requests.get(URL, headers=headers)
  print(response.text)
  

• 使用会话保持连接：部分服务器会验证会话一致性，用requests.Session()维持会话可以提高请求成功率：

  import requests
  
  URL = "https://www.tesco.com/fuel_prices/fuel_prices_data.json"
  headers = {
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
  }
  
  session = requests.Session()
  session.headers.update(headers)
  response = session.get(URL)
  print(response.text)
  

• 补充必要Cookies：如果上述方法无效，可手动访问目标页面，通过浏览器开发者工具（F12）获取请求中的Cookies，加入到请求头或会话中：

  import requests
  
  URL = "https://www.tesco.com/fuel_prices/fuel_prices_data.json"
  headers = {
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
      "Cookie": "your-cookie-content-here"
  }
  
  response = requests.get(URL, headers=headers)
  print(response.text)
  

• 使用代理IP：如果当前IP被服务器封禁，可尝试使用代理IP分散请求来源：

  import requests
  
  URL = "https://www.tesco.com/fuel_prices/fuel_prices_data.json"
  headers = {
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
  }
  proxies = {
      "http": "http://proxy-ip:port",
      "https": "https://proxy-ip:port"
  }
  
  response = requests.get(URL, headers=headers, proxies=proxies)
  print(response.text)
  

• 遵守网站规则：先查看网站的robots.txt文件，确认目标JSON路径是否允许爬取，避免违反网站的爬虫协议。

内容的提问来源于stack exchange，提问作者j4son