Selenium爬取findmasa.com时遇InvalidSelectorException错误求助

问题描述 #

爬取https://findmasa.com/city/los-angeles/中壁画子页面信息时，前4个页面正常，第5个链接https://findmasa.com/view/map#1456a64a抛出错误：

selenium.common.exceptions.InvalidSelectorException: Message: invalid selector: An invalid or illegal selector was specified (Session info: chrome=114.0.5735.199)

错误原因 #

CSS选择器语法规则中，ID选择器不能直接以数字开头。第5个壁画的ID是1456a64a，原代码使用li#1456a64a作为选择器，违反了语法规范，导致Selenium无法解析该选择器。

解决方案 #

改用属性选择器匹配元素的ID属性，这种方式不受ID开头字符的限制；同时修复原代码未关闭浏览器实例的资源泄漏问题，优化数据提取逻辑避免索引越界。

修改核心要点 #

1. 将等待元素的CSS选择器从li#{id}替换为li[id="{id}"]，兼容数字开头的ID
2. 在parse_mural函数末尾添加driver.quit()，避免残留大量Chrome进程
3. 优化数据提取逻辑，增加元素长度判断，防止索引越界报错
4. 调整CSV保存时机，爬取完成后统一保存，提升效率

修改后的完整代码 #

from requests_html import HTMLSession
import warnings
import csv

from selenium.webdriver import Chrome
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
import selenium.webdriver.support.expected_conditions as EC


warnings.filterwarnings("ignore", category=DeprecationWarning)

s = HTMLSession()

def get_mural_links(page):
    url = f'https://findmasa.com/city/los-angeles/{page}'
    links = []
    r = s.get(url)
    artworks = r.html.find('ul.list-works-cards div.top p')
    for item in artworks:
        links.append(item.find('a', first=True).attrs['href'])
    return links


def parse_mural(url):
    spl = '#'
    mural_id = url.partition(spl)[2]

    driver = Chrome()
    driver.get(url)

    # 使用属性选择器定位li元素，兼容数字开头的ID
    li_element = WebDriverWait(driver, 60).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, f'li[id="{mural_id}"]'))
    )

    data_lat = li_element.get_attribute('data-lat')
    data_lng = li_element.get_attribute('data-lng')
    link = url

    # 初始化默认值，避免索引越界
    artist = 'No Data'
    address = 'No Data'
    city = 'No Data'
    
    p_elements = li_element.find_elements(By.TAG_NAME, 'p')
    if len(p_elements) >= 3:
        city = p_elements[2].text
    if len(p_elements) >= 2:
        address = p_elements[1].text
    
    try:
        artist = li_element.find_element(By.TAG_NAME, 'a').text
    except:
        pass

    info = {
        'ID': mural_id,
        'ARTIST': artist,
        'LOCATION': address,
        'CITY': city,
        'LATITUDE': data_lat,
        'LONGITUDE': data_lng,
        'LINK': link,
    }
    
    driver.quit()  # 关闭浏览器，释放资源
    return info


def save_csv(results):
    keys = results[0].keys()
    with open('LAmural_MASA.csv', 'w', newline='', encoding='utf-8') as f:
        dict_writer = csv.DictWriter(f, keys)
        dict_writer.writeheader()
        dict_writer.writerows(results)


def main():
    results = []
    for x in range(1, 3):
        urls = get_mural_links(x)
        for url in urls:
            results.append(parse_mural(url))
    save_csv(results)  # 爬取完成后统一保存，提升效率


if __name__ == '__main__':
    main()

内容的提问来源于stack exchange，提问作者Jessie H