Web Scraping技术问题：如何关联主页面与子页面的对应数据？

我在对某网站做Web Scraping时，遇到了子页面数据无法和主页面地区数据关联的问题。主页面有类似「Alabama Trucking Companies」的地区名称，点进对应链接的子页面能看到Abbeville、Adamsville等城市信息，需要把每个城市的名称、链接和所属地区绑定。

目前的代码能分别抓取主页面的地区列表和子页面的城市列表，但没法建立两者的对应关系，原代码如下：

start_time = datetime.now()

url = 'https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php'

page_country = requests.get(url).content
soup_country = BeautifulSoup(page_country, 'lxml')
countries = soup_country.find('div',{'class':'col-xs-12 col-sm-9'})

countries_list = []
country_info = countries.find_all('div',{'class':'col-md-4 column'})
for i in country_info:
    title_country = i.text.strip()
    href_country = i.find('a', href=True)['href']
    countries_list.append({'Country Title':title_country, 'Link':(f'https://www.quicktransportsolutions.com//carrier//{href_country}')})

countries_links = []
for i in pd.DataFrame(countries_list)['Link']:
    page_city = requests.get(i).content
    soup_city = BeautifulSoup(page_city, 'lxml')
    city = soup_city.find('div',{'align':'center','class':'table-responsive'})
    countries_links.append(city)

cities_list = []
for i in countries_links:
    city_info = i.find_all('td',"")
    for i in city_info:
        title_city = i.text.strip()
        try:
            href_city = i.find('a', href=True)['href']
        except:
            continue
        cities_list.append({'City Title':title_city,'City Link':href_city})

end_time = datetime.now()
print(f'Duration: {end_time - start_time}')

df = pd.DataFrame(cities_list)
df = df.loc[df['City Link']!= '#'].drop_duplicates().reset_index(drop=True)
df

问题分析 #

原代码的问题在于把地区链接和城市数据分开处理，遍历地区链接时没有保留对应的地区名称，导致后续抓取城市时无法关联所属地区。

修改后的代码 #

核心思路是在遍历每个地区的同时，直接抓取该地区下的城市，并将地区名称嵌入到每条城市数据中：

from datetime import datetime
import requests
from bs4 import BeautifulSoup
import pandas as pd

start_time = datetime.now()

url = 'https://www.quicktransportsolutions.com/carrier/usa-trucking-companies.php'

page_country = requests.get(url).content
soup_country = BeautifulSoup(page_country, 'lxml')
countries = soup_country.find('div',{'class':'col-xs-12 col-sm-9'})

countries_list = []
country_info = countries.find_all('div',{'class':'col-md-4 column'})
for i in country_info:
    title_country = i.text.strip()
    href_country = i.find('a', href=True)['href']
    countries_list.append({'所属地区': title_country, '地区链接': f'https://www.quicktransportsolutions.com/carrier/{href_country}'})

# 直接在遍历地区时抓取城市并关联地区名称
cities_list = []
for region in countries_list:
    region_name = region['所属地区']
    region_link = region['地区链接']
    
    page_city = requests.get(region_link).content
    soup_city = BeautifulSoup(page_city, 'lxml')
    city_container = soup_city.find('div', {'align':'center','class':'table-responsive'})
    
    if not city_container:
        continue
    
    city_info = city_container.find_all('td', "")
    for city_item in city_info:
        title_city = city_item.text.strip()
        try:
            href_city = city_item.find('a', href=True)['href']
        except:
            continue
        
        # 跳过无效链接
        if href_city == '#':
            continue
        
        cities_list.append({
            '所属地区': region_name,
            '城市名称': title_city,
            '城市链接': href_city
        })

end_time = datetime.now()
print(f'Duration: {end_time - start_time}')

df = pd.DataFrame(cities_list).drop_duplicates().reset_index(drop=True)
print(df.head())

修改说明 #

1. 合并了地区遍历与城市抓取的逻辑，在处理每个地区时，直接携带地区名称去抓取子页面的城市
2. 每条城市数据中新增了所属地区字段，直接关联对应的地区名称
3. 提前过滤了无效的#链接，避免后续重复处理
4. 优化了变量命名，让代码可读性更强

内容的提问来源于stack exchange，提问作者Mahmoud Badr