Python网页爬取：提取指定Div内链接及实现翻页爬取

一、精准提取指定区域内的学院链接 #

你无法提取id="ContentPlaceHolder1_pnl_collegelist" Div内的链接，核心问题是没有精准定位到目标容器再提取子链接。用lxml的XPath可以直接锁定目标区域，再提取其中的所有学院链接，同时注意处理相对路径转绝对路径：

from lxml import html
import requests

base_url = "https://www.indiacollegeshub.com"
list_page_url = "https://www.indiacollegeshub.com/colleges/"
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
}

# 发起请求并解析页面
response = requests.get(list_page_url, headers=headers)
tree = html.fromstring(response.content)

# 先锁定目标Div，再提取内部所有a标签的href属性
college_links = tree.xpath('//div[@id="ContentPlaceHolder1_pnl_collegelist"]//a/@href')
# 转换为绝对URL（过滤非站点内的链接）
absolute_links = [base_url + link for link in college_links if link.startswith('/')]

二、实现2000+页面的翻页爬取 #

首先观察站点翻页URL的规律：该站点翻页参数为page，第一页默认无参数，第二页为https://www.indiacollegeshub.com/colleges/?page=2，以此类推。可以通过以下步骤实现批量翻页：

1. 获取最大页码 #

先爬取第一页，从分页控件中提取最大页码（需根据实际页面结构调整XPath）：

def get_max_page():
    first_page_url = "https://www.indiacollegeshub.com/colleges/"
    response = requests.get(first_page_url, headers=headers)
    tree = html.fromstring(response.content)
    # 假设分页栏倒数第二个a标签是最大页码（需根据实际页面结构调整）
    max_page_text = tree.xpath('//div[contains(@class, "pagination")]//a[last()-1]/text()')[0]
    return int(max_page_text)

2. 循环爬取所有页面 #

构造每个页面的URL，批量提取学院链接：

import time

max_page = get_max_page()
all_college_links = []

for page in range(1, max_page + 1):
    page_url = f"https://www.indiacollegeshub.com/colleges/?page={page}"
    print(f"爬取第 {page} 页...")
    try:
        response = requests.get(page_url, headers=headers)
        response.raise_for_status()  # 捕获HTTP请求错误
        tree = html.fromstring(response.content)
        current_links = tree.xpath('//div[@id="ContentPlaceHolder1_pnl_collegelist"]//a/@href')
        absolute_links = [base_url + link for link in current_links if link.startswith('/')]
        all_college_links.extend(absolute_links)
        time.sleep(1)  # 控制请求频率，避免触发反爬
    except Exception as e:
        print(f"第 {page} 页爬取失败: {str(e)}")
        continue

# 保存所有链接到本地
with open('college_links.txt', 'w', encoding='utf-8') as f:
    for link in all_college_links:
        f.write(link + '\n')
print(f"共提取 {len(all_college_links)} 个学院链接")

三、爬取单个学院页面数据 #

拿到所有学院链接后，循环进入每个链接提取目标数据（示例提取名称和地址，需根据页面结构调整XPath）：

import json

def crawl_college_detail(url):
    try:
        response = requests.get(url, headers=headers)
        response.raise_for_status()
        tree = html.fromstring(response.content)
        # 示例提取字段，需根据实际页面结构修改
        college_name = tree.xpath('//h1/text()')[0].strip()
        college_address = tree.xpath('//div[@class="college-address"]/text()')[0].strip()
        return {
            'name': college_name,
            'address': college_address,
            'url': url
        }
    except Exception as e:
        print(f"爬取 {url} 失败: {str(e)}")
        return None

# 批量爬取所有学院数据
college_data = []
for link in all_college_links:
    detail = crawl_college_detail(link)
    if detail:
        college_data.append(detail)
    time.sleep(0.5)

# 保存为JSON文件
with open('college_data.json', 'w', encoding='utf-8') as f:
    json.dump(college_data, f, ensure_ascii=False, indent=2)

关键注意事项 #

• 反爬规避：如果遇到403或请求失败，可增加延时（如time.sleep(2)），或使用代理IP；
• 结构变化：站点可能更新页面布局，需定期检查XPath是否有效；
• 数据存储：2000+页面的数据量建议用SQLite/MySQL存储，而非纯文本文件。

内容的提问来源于stack exchange，提问作者Pratik