技术咨询：如何从https://maroof.sa/businesses/获取所有店铺URL？

这个站点的店铺数据是动态加载的，普通静态爬虫无法获取完整内容，以下是两种可行方案：

方案一：使用Playwright模拟浏览器（推荐） #

Playwright能完整渲染JS动态内容，还能自动处理滚动加载，步骤如下：

1. 安装依赖：

pip install playwright
playwright install chromium

2. 编写Python代码：

from playwright.sync_api import sync_playwright

def get_all_business_urls():
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=False)  # 调试时可设为False查看浏览器操作
        page = browser.new_page()
        page.goto("https://maroof.sa/businesses/")
        
        # 滚动页面加载所有店铺内容
        last_height = page.evaluate("document.body.scrollHeight")
        while True:
            page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
            page.wait_for_timeout(3000)  # 等待3秒确保内容加载
            new_height = page.evaluate("document.body.scrollHeight")
            if new_height == last_height:
                break
            last_height = new_height
        
        # 提取所有店铺URL并去重
        business_links = page.locator("a[href*='/business/']").all()
        urls = [link.get_attribute("href") for link in business_links]
        unique_urls = list(set(urls))
        
        print(f"共提取到 {len(unique_urls)} 个店铺URL")
        browser.close()
        return unique_urls

if __name__ == "__main__":
    get_all_business_urls()

方案二：抓取站点API接口（更高效） #

打开浏览器开发者工具（F12）切换到Network标签，刷新页面并滚动，可找到加载店铺数据的API接口（类似https://maroof.sa/api/v1/businesses?page=xxx）。直接请求API绕过浏览器渲染，速度更快：

示例Python代码：

import requests
import time

def get_businesses_via_api():
    base_url = "https://maroof.sa/api/v1/businesses"
    page = 1
    all_urls = []
    
    while True:
        params = {"page": page, "per_page": 20}  # 实际参数需通过开发者工具确认
        response = requests.get(base_url, params=params)
        if response.status_code != 200:
            break
        
        data = response.json()
        if not data.get("data"):
            break
        
        # 从返回的JSON中提取店铺URL
        for business in data["data"]:
            business_url = f"https://maroof.sa/business/{business['slug']}"
            all_urls.append(business_url)
        
        page += 1
        time.sleep(1)  # 添加间隔避免触发反爬
    
    unique_urls = list(set(all_urls))
    print(f"共提取到 {len(unique_urls)} 个店铺URL")
    return unique_urls

if __name__ == "__main__":
    get_businesses_via_api()

注意事项 #

• 不要高频请求，可添加请求间隔避免IP被封禁
• API参数可能随时变化，需实时通过开发者工具确认最新接口和参数
• 模拟浏览器时，需根据网络状况调整等待时间，确保内容完全加载

内容的提问来源于stack exchange，提问作者ahmed sayed