Python网页爬虫运行卡顿并出现两类错误的技术求助
问题描述
我正在完成学校项目,尝试用Python爬取网站数据,参考了Edureka的教程。以下是我的代码:
from selenium import webdriver from bs4 import BeautifulSoup import pandas as pd driver = webdriver.Chrome("D:\COSC2625_Team_Blue\chromedriver") products=[] #List to store name of the product prices=[] #List to store price of the product ratings=[] #List to store rating of the product driver.get("""<a href="https://www.flipkart.com/laptops/">https://www.flipkart.com/laptops/</a>~buyback-guarantee-on-laptops-/pr?sid=6bo%2Cb5g&amp;amp;amp;amp;amp;amp;amp;amp;amp;uniq""") content = driver.page_source soup = BeautifulSoup(content) for a in soup.findAll('a',href=True, attrs={'class':'_31qSD5'}): name=a.find('div', attrs={'class':'_3wU53n'}) price=a.find('div', attrs={'class':'_1vC4OE _2rQ-NK'}) rating=a.find('div', attrs={'class':'hGSR34 _2beYZw'}) products.append(name.text) prices.append(price.text) ratings.append(rating.text) df = pd.DataFrame({'Product Name':products,'Price':prices,'Rating':ratings}) df.to_csv('products.csv', index=False, encoding='utf-8')
运行代码后弹出空白浏览器,终端先出现这条错误:
PS D:\COSC2625_Team_Blue> [8556:2544:0809/132114.260:ERROR:device_event_log_impl.cc(214)] [13:21:14.260] USB: usb_device_handle_win.cc:1048 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
查了下知道这个不影响运行,等了一会儿又出现第二条错误:
[4488:14404:0809/132312.207:ERROR:gpu_init.cc(486)] Passthrough is not supported, GL is disabled, ANGLE is
之后程序彻底卡顿,等了一小时还是只有这两条错误,浏览器一直空白,请问这是什么原因?
排查与解决建议
- 修正URL格式:
driver.get()中的URL包含了HTML<a>标签,导致Chrome无法解析正确地址,直接卡在加载。改成纯链接:driver.get("https://www.flipkart.com/laptops/~buyback-guarantee-on-laptops-/pr?sid=6bo%2Cb5g&uniq") - 匹配ChromeDriver版本:确保ChromeDriver版本和本地Chrome浏览器版本完全一致,版本不兼容会引发启动异常或卡顿。
- 添加页面等待:网站加载需要时间,直接获取
page_source可能拿到空内容,添加显式等待确保元素加载完成:from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC # 最长等待10秒,直到目标元素出现 WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, "_31qSD5"))) content = driver.page_source - 禁用GPU加速:GPU初始化错误可通过添加Chrome参数解决,避免浏览器卡顿:
options = webdriver.ChromeOptions() options.add_argument("--disable-gpu") driver = webdriver.Chrome(r"D:\COSC2625_Team_Blue\chromedriver", options=options) - 修复路径转义:Windows路径中的反斜杠需转义,或使用原始字符串避免解析错误:
driver = webdriver.Chrome(r"D:\COSC2625_Team_Blue\chromedriver")
内容的提问来源于stack exchange,提问作者Xiaolong Guo
相关产品推荐
相关产品推荐

