Google Colab中Python调用Redfin爬虫获取Property ID无返回求助

我在Google Colaboratory中编写并运行Python代码，想借助Apify的Redfin Search爬虫从给定地址列表提取Property ID。代码逻辑是先把地址转成经纬度，再定位房产获取ID，已经填入API token，但始终没有任何信息返回。测试地址用的是“10725 Lone Pine Ln, Fort Worth, TX 76108”，预期能拿到Property ID等信息，想知道是代码问题还是Colab无法运行这类代码？

以下是我的代码：

import requests
from geopy.geocoders import Nominatim
import json
import time

APIFY_API_URL = 'https://api.apify.com/v2/acts/tri_angle~redfin-search/runs?token=***'  # Plug in the provided API URL

def geocode_address(address):
    geolocator = Nominatim(user_agent="redfin_scraper", timeout=10)  # Increased timeout to 10 seconds
    try:
        location = geolocator.geocode(address)
        if location:
            return location.latitude, location.longitude
    except Exception as e:
        print(f"Error geocoding address {address}: {e}")
    return None, None

def search_redfin(lat, lng):
    search_url = f"https://www.redfin.com/stingray/do/location-autocomplete?location={lat},{lng}"
    headers = {'User-Agent': 'Mozilla/5.0'}
    response = requests.get(search_url, headers=headers)
    if response.status_code == 200:
        try:
            json_text = response.text.split('&&')[1]
            return json.loads(json_text)
        except (json.JSONDecodeError, IndexError) as e:
            print("Error decoding JSON response:", e)
            print("Response text:", response.text)
            return None
    return None

def extract_home_id(search_results):
    try:
        payload = search_results.get('payload')
        if not payload:
            return None
        region_views = payload.get('regionViews')
        if not region_views:
            return None
        root = region_views.get('__root')
        if not root:
            return None
        for result in root:
            if 'home' in result:
                return result['home']['id']
    except KeyError as e:
        print(f"KeyError: {e}")
    return None

def construct_redfin_detail_url(state, city, street, zip_code, home_id):
    street_formatted = street.replace(' ', '-')
    city_formatted = city.replace(' ', '-')
    return f"https://www.redfin.com/{state}/{city_formatted}/{street_formatted}/home/{home_id}"

def query_apify_redfin_scraper(detail_url):
    api_url = APIFY_API_URL
    payload = {
        "detailUrls": [{"url": detail_url}],
        "debugLog": False
    }
    headers = {
        "Content-Type": "application/json"
    }
    response = requests.post(api_url, headers=headers, data=json.dumps(payload))
    if response.status_code == 201:
        try:
            return response.json()['data']['id']
        except json.JSONDecodeError as e:
            print("Error decoding JSON response:", e)
            print("Response text:", response.text)
            return None
    return None

def get_apify_scraper_results(run_id):
    api_url = f"https://api.apify.com/v2/acts/tri_angle~redfin-search/run-sync-get-dataset-items?token=***"
    while True:
        response = requests.get(api_url)
        if response.status_code == 200:
            try:
                data = response.json()
                if data:
                    return data
            except json.JSONDecodeError as e:
                print("Error decoding JSON response:", e)
                print("Response text:", response.text)
                return None
        time.sleep(10)  # Wait for 10 seconds before retrying
    return None

def main(addresses):
    results = []
    for address in addresses:
        lat, lng = geocode_address(address)
        if lat and lng:
            search_results = search_redfin(lat, lng)
            if search_results:
                home_id = extract_home_id(search_results)
                if home_id:
                    parts = address.split(',')
                    street = parts[0].strip()
                    city = parts[1].strip()
                    state_zip = parts[2].strip().split(' ')
                    state = state_zip[0]
                    zip_code = state_zip[1]
                    detail_url = construct_redfin_detail_url(state, city, street, zip_code, home_id)
                    run_id = query_apify_redfin_scraper(detail_url)
                    if run_id:
                        scraper_results = get_apify_scraper_results(run_id)
                        results.append(scraper_results)
    return results

addresses = [
    "10725 Lone Pine Ln, Fort Worth, TX 76108",
    # Add more addresses here
]

detail_urls = main(addresses)
print(json.dumps(detail_urls, indent=2))

排查与解决建议 #

1. Colab兼容性排除 #

Colab完全支持这类Python网络请求代码，不存在无法运行的情况，问题大概率出在代码逻辑或API调用细节上。

2. 代码关键问题点 #

• Apify结果接口错误：get_apify_scraper_results函数的API URL未传入run_id，正确格式应为f"https://api.apify.com/v2/acts/tri_angle~redfin-search/runs/{run_id}/dataset/items?token=***"，原代码固定URL导致无法获取对应任务的结果。
• Redfin接口解析失效：Redfin的location-autocomplete接口返回格式可能已变更，split('&&')[1]的解析方式可能无法正确提取JSON，建议打印完整响应文本确认结构。
• 地址拆分逻辑脆弱：通过split(',')拆分地址的方式在地址含逗号时会出错，建议使用usaddress等专业地址解析库。
• 错误日志缺失：多数分支仅返回None但无错误提示，无法定位失败环节，需在每个返回None的分支添加打印语句（如“经纬度获取失败”“Redfin搜索无结果”）。

3. 调试步骤 #

• 在main函数的每一步添加打印，输出经纬度、Redfin搜索结果、home_id、detail_url、run_id等，确认哪一步返回None。
• 单独测试geocode_address函数，验证Nominatim能否正确返回目标地址的经纬度。
• 手动访问构造的detail_url，确认Redfin页面正常且包含目标房产信息。
• 检查Apify账户的API token权限与剩余额度，确保爬虫可正常触发。

内容的提问来源于stack exchange，提问作者austinmb