动态航班网站爬取求助：弹窗内航班组构数据爬取及代码实现方案

Python 版本分步计划 #

1. 整理已分析的请求核心信息 #

• 把抓包得到的航班列表接口和组构信息接口的URL、请求方法（GET/POST）、必填请求头（如User-Agent、Cookie、X-Requested-With）、参数（如日期范围、航班ID）整理成清晰清单
• 确认接口是否需要动态签名、token等参数，单独记录其生成逻辑（如果有）

2. 编写通用请求工具函数 #

用requests库实现可复用的请求函数，处理请求发送、异常捕获和响应解析：

import requests

def make_request(url, method='GET', params=None, headers=None, data=None):
    try:
        if method.upper() == 'GET':
            resp = requests.get(url, params=params, headers=headers)
        else:
            resp = requests.post(url, headers=headers, data=data)
        resp.raise_for_status()
        return resp.json()  # 接口返回JSON用这个，非JSON则返回resp.text
    except requests.exceptions.RequestException as e:
        print(f"请求失败: {e}")
        return None

3. 获取航班日期与对应ID列表 #

调用航班列表接口，提取每个航班的日期和唯一标识ID，存储为结构化数据：

# 替换为你抓包到的真实接口和参数
flight_list_url = "https://example.com/api/flights"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36...",
    "Cookie": "你的Cookie内容",
}
flight_params = {
    "start_date": "2024-05-01",
    "end_date": "2024-05-10",
}

flight_data = make_request(flight_list_url, params=flight_params, headers=headers)
flight_items = []
if flight_data:
    for item in flight_data.get('data', []):
        flight_items.append({
            "date": item.get('flight_date'),
            "flight_id": item.get('flight_id')
        })

4. 循环请求组构信息接口 #

遍历航班列表，用每个航班的ID调用组构接口，关联日期存储结果：

# 替换为真实组构信息接口
seat_map_url = "https://example.com/api/seat-map"
seat_map_results = []

for flight in flight_items:
    seat_params = {
        "flight_id": flight['flight_id'],
        "date": flight['date']
    }
    seat_data = make_request(seat_map_url, params=seat_params, headers=headers)
    if seat_data:
        seat_map_results.append({
            "flight_date": flight['date'],
            "flight_id": flight['flight_id'],
            "seat_map_info": seat_data.get('seat_map', {})
        })

5. 数据保存与异常优化 #

• 将结果保存为JSON或CSV文件：

import json

with open("flight_seat_map.json", "w", encoding="utf-8") as f:
    json.dump(seat_map_results, f, ensure_ascii=False, indent=4)

• 给请求添加重试机制（如失败后重试2次，每次间隔1秒），避免单个请求失败中断脚本

PHP 版本分步计划 #

1. 整理请求元数据 #

• 列出航班列表、组构信息接口的URL、请求方法、必填请求头、参数，重点标记Cookie、Referer等影响请求合法性的字段
• 记录动态参数（如token）的生成规则（如果有）

2. 编写通用cURL请求函数 #

实现可复用的请求函数，处理GET/POST请求、请求头和响应解析：

function makeRequest($url, $method = 'GET', $params = [], $headers = []) {
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // 生产环境建议开启SSL验证

    // 格式化请求头
    $curlHeaders = [];
    foreach ($headers as $key => $value) {
        $curlHeaders[] = "$key: $value";
    }
    curl_setopt($ch, CURLOPT_HTTPHEADER, $curlHeaders);

    if ($method === 'POST') {
        curl_setopt($ch, CURLOPT_POST, true);
        curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($params));
    } else {
        $url .= '?' . http_build_query($params);
        curl_setopt($ch, CURLOPT_URL, $url);
    }

    $response = curl_exec($ch);
    if (curl_errno($ch)) {
        echo "请求失败: " . curl_error($ch);
        curl_close($ch);
        return null;
    }
    curl_close($ch);
    return json_decode($response, true); // 非JSON响应直接返回$response
}

3. 获取航班日期与ID列表 #

调用航班列表接口，解析响应并提取核心数据：

// 替换为真实接口和参数
$flightListUrl = "https://example.com/api/flights";
$headers = [
    "User-Agent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36...",
    "Cookie" => "你的Cookie内容",
];
$flightParams = [
    "start_date" => "2024-05-01",
    "end_date" => "2024-05-10",
];

$flightData = makeRequest($flightListUrl, 'GET', $flightParams, $headers);
$flightItems = [];
if ($flightData && isset($flightData['data'])) {
    foreach ($flightData['data'] as $item) {
        $flightItems[] = [
            "date" => $item['flight_date'],
            "flight_id" => $item['flight_id']
        ];
    }
}

4. 批量请求组构信息 #

遍历航班数组，调用组构接口并关联日期存储结果：

// 替换为真实组构信息接口
$seatMapUrl = "https://example.com/api/seat-map";
$seatMapResults = [];

foreach ($flightItems as $flight) {
    $seatParams = [
        "flight_id" => $flight['flight_id'],
        "date" => $flight['date']
    ];
    $seatData = makeRequest($seatMapUrl, 'GET', $seatParams, $headers);
    if ($seatData) {
        $seatMapResults[] = [
            "flight_date" => $flight['date'],
            "flight_id" => $flight['flight_id'],
            "seat_map_info" => $seatData['seat_map'] ?? []
        ];
    }
}

5. 数据持久化与优化 #

• 将结果保存为JSON文件：

file_put_contents("flight_seat_map.json", json_encode($seatMapResults, JSON_UNESCAPED_UNICODE | JSON_PRETTY_PRINT));

• 添加重试逻辑，如请求失败时延迟1秒后重试2次，降低被反爬拦截的概率

内容的提问来源于stack exchange，提问作者Help