使用Playwright实现无限滚动加载时遇TimeoutError问题求助

我需要使用Playwright加载网页https://www.justdial.com/Mumbai/Tourist-Attraction/nct-10596038上的全部景点数据，但目前仅能获取前10条数据，且执行代码时抛出TimeoutError。我尝试过通过循环调用page.mouse.wheel()结合time.sleep()的无限滚动方案，但未解决问题。

核心代码 #

from playwright.sync_api import sync_playwright
import pandas as pd

def main():
    with sync_playwright() as p:
        chromium_path = r"C:\Users\pranit\AppData\Local\ms-playwright\chromium-1076\chrome-win\chrome.exe"
        browser = p.chromium.launch(executable_path=chromium_path, headless=False)

        page_url = f'https://www.justdial.com/Mumbai/Tourist-Attraction/nct-10596038'

        page = browser.new_page()
        page.goto(page_url, timeout=600000)

        attractions = page.locator('//div[@class="jsx-a3a43292cc1e6428 results_listing_container"]/div').all()
        print(f'There are: {len(attractions)} attractions.')

        attractions_list =[]
        for attraction in attractions:
            attraction_dict = {} 
            attraction_dict['place'] = attraction.locator('//div[@class="jsx-4d407376001b01ad"]/h2/a').inner_text()


            attractions_list.append(attraction_dict)
        df = pd.DataFrame(attractions_list)
        df.to_excel('attractions_list.xlsx', index=False)
        df.to_csv('attractions_list.csv', index=False)

        browser.close()

if __name__ == '__main__':
    main()

尝试过的无限滚动代码 #

for i in range(5): #make the range as long as needed
        page.mouse.wheel(0, 15000)
        time.sleep(2)
        i += 1
    
    time.sleep(15)

报错信息 #

PS C:\Users\pranit\OneDrive\Desktop\BeautifulSoup tutorial> python -u "c:\Users\pranit\OneDrive\Desktop\BeautifulSoup tutorial\TravelAdvisorAttractions.py"
There are: 10 attractions.
Traceback (most recent call last):
  File "c:\Users\pranit\OneDrive\Desktop\BeautifulSoup tutorial\TravelAdvisorAttractions.py", line 31, in <module>
    main()
  File "c:\Users\pranit\OneDrive\Desktop\BeautifulSoup tutorial\TravelAdvisorAttractions.py", line 20, in main
    attraction_dict['place'] = attraction.locator('//div[@class="jsx-4d407376001b01ad"]/h2/a').inner_text()
  File "C:\Users\pranit\AppData\Local\Programs\Python\Python39\lib\site-packages\playwright\sync_api\_generated.py", line 17228, in inner_text
    self._sync(self._impl_obj.inner_text(timeout=timeout))
  File "C:\Users\pranit\AppData\Local\Programs\Python\Python39\lib\site-packages\playwright\_impl\_sync_base.py", line 109, in _sync
    return task.result()
  File "C:\Users\pranit\AppData\Local\Programs\Python\Python39\lib\site-packages\playwright\_impl\_locator.py", line 444, in inner_text
    return await self._frame.inner_text(
  File "C:\Users\pranit\AppData\Local\Programs\Python\Python39\lib\site-packages\playwright\_impl\_frame.py", line 619, in inner_text
  File "C:\Users\pranit\AppData\Local\Programs\Python\Python39\lib\site-packages\playwright\_impl\_connection.py", line 482, in wrap_api_call
    return await cb()
  File "C:\Users\pranit\AppData\Local\Programs\Python\Python39\lib\site-packages\playwright\_impl\_connection.py", line 97, in inner_send
    result = next(iter(done)).result()
playwright._impl._api_types.TimeoutError: Timeout 30000ms exceeded.
=========================== logs ===========================
waiting for locator("xpath=//div[@class=\"jsx-a3a43292cc1e6428 results_listing_container\"]/div").first.locator("xpath=//div[@class=\"jsx-4d407376001b01ad\"]/h2/a")

内容的提问来源于stack exchange，提问作者Pranit Utekar