Selenium爬取RSC文献时遭遇TimeoutException问题求助

问题详情 #

使用Selenium爬取RSC corpus文献时，以下代码行始终抛出TimeoutException：

wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, '.capsule.capsule--article')))

已尝试EC.presence_of_element_located和EC.visibility_of_element_located两种等待策略，均无效。确认CSS选择器正确，URL可正常加载查询结果，但无法定位到解决异常的方法。

堆栈跟踪信息 #

Traceback (most recent call last):
  File "rsc.py", line 40, in <module>
    main(url, query, page, location)
  File "rsc.py", line 22, in main
    dois = scraper.get_doi(query=query, page=page)
  File "/home/ssarrouf/Documents/GitHub/WaterRemediationParser/batterydataextractor-main/batterydataextractor/scrape/rsc.py", line 62, in get_doi
    _ = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, '.capsule.capsule--article')))
  File "/home/ssarrouf/.pyenv/versions/3.8.16/lib/python3.8/site-packages/selenium/webdriver/support/wait.py", line 95, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: 
Stacktrace:
#0 0x55e10fae5d93 <unknown>
#1 0x55e10f8b42d7 <unknown>
#2 0x55e10f8f0caa <unknown>
#3 0x55e10f8f0db1 <unknown>
#4 0x55e10f92e8f4 <unknown>
#5 0x55e10f91461d <unknown>
#6 0x55e10f92c619 <unknown>
#7 0x55e10f914353 <unknown>
#8 0x55e10f8e3e40 <unknown>
#9 0x55e10f8e5038 <unknown>
#10 0x55e10fb398be <unknown>
#11 0x55e10fb3d8f0 <unknown>
#12 0x55e10fb1df90 <unknown>
#13 0x55e10fb3eb7d <unknown>
#14 0x55e10fb0f578 <unknown>
#15 0x55e10fb63348 <unknown>
#16 0x55e10fb634d6 <unknown>
#17 0x55e10fb7d341 <unknown>
#18 0x7f0fad5a9b43 <unknown>

相关代码片段 #

get_doi方法 #

def get_doi(self, query, page):
      """
        Get a list of dois from query massages and the exact page.
        :param query: the query text (e.g. battery materials)
        :param page: the number of page
        :return: a list of dois of the relevant query text and page.
        """
        
        if self.driver is None:
            driver = webdriver.Chrome()
        else:
            driver = self.driver

        if self.url is None:
            url = "http://pubs.rsc.org/en/results?searchtext="
            url = url + query
        else:
            url = self.url
        driver.get(url)
        wait = WebDriverWait(driver, self.max_wait_time)
        # To make sure we don't overload the server
        sleep(1)
        next_button = wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a[class^=paging__btn]")))[1]
        page_string = """document.querySelectorAll("a[class^=paging__btn]")[1].setAttribute("data-pageno", \\""""\
                      + str(page) + """\\")"""
        driver.execute_script(page_string)
        next_button.click()
        sleep(1)
        _ = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, '.capsule.capsule--article')))
        doi_lists = driver.find_elements(By.PARTIAL_LINK_TEXT, 'https://doi.org')
        dois = [doi.text for doi in doi_lists]
        return dois

文献下载方法 #

def download_doi(self, doi, file_location):
    """
    Download the html paper of the doi
    :param doi: doi of the paper
    :param file_location: the saving location
    :return:
    """
    doi = doi.split("org/")[-1]
    r = requests.get('http://doi.org/' + doi, headers={'User-Agent': 'Mozilla/5.0'})
    result = re.findall(r'https://pubs.rsc.org/en/content/articlehtml/.*?"', r.text)
    url = result[0][:-1]
    web_content = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}).content

    result = self.get_rsc_abstract(web_content)
    exact_date = result['date'].split("/")
    doi = result['doi'].replace("/", "_")
    if len(exact_date) == 3:
        name = exact_date[0] + exact_date[1] + exact_date[2] + '_' + doi
    else:
        name = result['online_date'].replace("/", "") + '_' + doi

    with open(file_location + name + '.html', 'wb') as f:
        f.write(web_content)
    return

爬虫主方法 #

def main(url, query, page, file_location):
    """
    RSC web-scraper runner
    :param url: the scraping url (or default)
    :param query: query text (e.g. battery materials)
    :param page: the page number of the query pages
    :param file_location: saving location
    :return:
    """
    scraper = RSCWebScraper(url=url)
    dois = scraper.get_doi(query=query, page=page)
    for doi in dois:
        try:
            scraper.download_doi(doi, file_location)
        # Some papers don't have html access
        except:
            continue
    return


if __name__ == "__main__":
    # Download papers within a certain date range
    url = "https://pubs.rsc.org/en/results/all?Category=All&AllText=water%20remediation&IncludeReference=false&Select" \
          "Journal=false&DateRange=true&SelectDate=true&DateToYear={}&DateFromYear={}&DateFromMonth={}&DateTo" \
          "Month={}&PriceCode=False&OpenAccess=false".format("2022", "2021", "06", "01")
    query = "water remediation"
    location = r"/home/ssarrouf/Documents/webscrape/to_date_papers/rsc/"
    for page in range(1, 120):
        main(url, query, page, location)

已尝试的无效操作 #

• 修改等待方式（presence_of_element_located/visibility_of_element_located）
• 调整查询条件和日期范围

1. 排查iframe嵌套问题 #

RSC部分页面会通过iframe加载搜索结果，导致目标元素不在主DOM树中。在等待目标元素前执行以下代码切换到对应iframe：

# 遍历所有iframe，尝试切换到包含目标元素的iframe
for iframe in driver.find_elements(By.TAG_NAME, 'iframe'):
    try:
        driver.switch_to.frame(iframe)
        if len(driver.find_elements(By.CSS_SELECTOR, '.capsule.capsule--article')) > 0:
            break
    except Exception:
        driver.switch_to.default_content()
# 若未找到目标iframe，切回主文档
driver.switch_to.default_content()

2. 绕过反爬检测 #

RSC可能通过浏览器自动化特征识别爬虫，配置ChromeDriver模拟真实浏览器：

from selenium.webdriver.chrome.options import Options

options = Options()
# 禁用自动化检测标志
options.add_argument("--disable-blink-features=AutomationControlled")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)

driver = webdriver.Chrome(options=options)
# 覆盖navigator.webdriver属性
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")

点击分页后触发页面滚动，确保动态内容加载：

next_button.click()
# 滚动至页面底部触发内容加载
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(2)

3. 优化等待策略 #

延长超时时间并等待多个目标元素出现，提升稳定性：

# 将超时时间调整为30秒
wait = WebDriverWait(driver, 30)
# 等待至少一个结果胶囊加载完成
_ = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '.capsule.capsule--article')))

4. 重构分页逻辑 #

当前修改data-pageno的方式不可靠，直接构造分页URL跳转更稳定：
RSC搜索结果的分页URL通常可通过在原URL后拼接&page=页码实现，替换原分页相关代码：

# 移除原next_button相关操作，直接访问目标分页URL
driver.get(f"{url}&page={page}")
# 等待结果加载
_ = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, '.capsule.capsule--article')))

5. 确保页面完全加载 #

在执行元素定位前，等待页面DOM加载完成：

wait.until(lambda driver: driver.execute_script('return document.readyState') == 'complete')

内容的提问来源于stack exchange，提问作者Chloe C