使用BeautifulSoup获取avamet表格时region参数异常问题求助

问题 #

我尝试从avamet网站获取两个数据表，使用Google Colab中的BeautifulSoup Python库。目标URL为：https://www.avamet.org/mx-consulta-diaria.php?id=%%25%%2525&ini=2024-02-27&fin=2024-02-28&token=s0141%21，该页面可见两个表格。

我编写了如下代码：

import requests
import pandas as pd
from bs4 import BeautifulSoup

def get_avamet_data(START_DATE=None, END_DATE=None, region='%%25%%25'):

  # All stations: %%25%%25  -------------------------------------------> Doesn't work
  # Only the station 15: c15%25 ---------------------------------------> It works!!!

  url = 'https://www.avamet.org/mx-consulta-diaria.php?id=' + region + '25&ini=' + START_DATE + '&fin=' + END_DATE + '&token=s0141%21'

  print('Getting data from: ' + url)

  response = requests.get(url)

  web_html = BeautifulSoup(response.content, 'html.parser')

  selector = 'table'
  tables = web_html.find_all(selector)

  table_1 = tables[0]
  table_2 = tables[1]

  data_1 = []
  for row in table_1.find_all('tr'):
    cols = row.find_all(['td', 'th'])
    cols = [ele.text.strip() for ele in cols]
    data_1.append([ele for ele in cols if ele])

  data_2 = []
  for row in table_2.find_all('tr'):
    cols = row.find_all(['td', 'th'])
    cols = [ele.text.strip() for ele in cols]
    data_2.append([ele for ele in cols if ele])

  return([pd.DataFrame(data_1), pd.DataFrame(data_2)])

调用代码：

a = get_avamet_data(START_DATE='2024-02-27', END_DATE='2024-02-28')

a[0]
a[1]

但tables变量返回空列表。不过当将region参数从'%%25%%25'改为'c15%25'时，代码可以正常运行。请问问题出在哪里？

分析与解决 #

1. URL参数拼接错误 #

你构造URL时，给region='%%25%%25'额外拼接了'25'，最终生成的id参数值是%%25%%2525——这和目标URL中的参数一致，但服务器对全站点请求的参数解析逻辑可能不同，或者该参数本身就无效。

对比单站点场景：region='c15%25'拼接'25'后得到c15%2525，刚好是服务器能识别的单站点参数格式，因此能正常返回表格内容。

2. 请求头缺失触发反爬限制 #

requests.get默认使用python-requests/x.x.x作为User-Agent，很多网站会识别这类非浏览器请求，返回无实际内容的页面（自然找不到表格）。而单站点请求数据量小，未触发服务器的反爬拦截，所以能正常工作。

修正后的代码 #

import requests
import pandas as pd
from bs4 import BeautifulSoup

def get_avamet_data(START_DATE=None, END_DATE=None, region='%25%25'):
    # 直接使用全站点的正确参数值，无需额外拼接25
    url = f'https://www.avamet.org/mx-consulta-diaria.php?id={region}&ini={START_DATE}&fin={END_DATE}&token=s0141%21'
    print('Getting data from: ' + url)

    # 添加浏览器请求头模拟正常访问
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
    }
    response = requests.get(url, headers=headers)

    web_html = BeautifulSoup(response.content, 'html.parser')
    tables = web_html.find_all('table')

    # 增加容错判断，避免索引越界报错
    if len(tables) < 2:
        print("未找到预期的两个表格")
        return [pd.DataFrame(), pd.DataFrame()]

    # 解析表格数据逻辑不变
    data_1 = []
    for row in tables[0].find_all('tr'):
        cols = [ele.text.strip() for ele in row.find_all(['td', 'th'])]
        data_1.append([ele for ele in cols if ele])

    data_2 = []
    for row in tables[1].find_all('tr'):
        cols = [ele.text.strip() for ele in row.find_all(['td', 'th'])]
        data_2.append([ele for ele in cols if ele])

    return [pd.DataFrame(data_1), pd.DataFrame(data_2)]

调用示例：

a = get_avamet_data(START_DATE='2024-02-27', END_DATE='2024-02-28')
print(a[0])
print(a[1])

内容的提问来源于stack exchange，提问作者sermomon