Visual Studio运行报错但命令行正常，原因何在？

我的Python爬虫代码在命令行运行时能正常抓取所有表格，但在Visual Studio中运行时大量表格缺失，抛出ValueError: No tables found错误。以下是完整代码和VS中的报错信息，请求排查原因。

完整代码 #

from io import StringIO

import pandas as pd
import requests
from bs4 import BeautifulSoup
import xlsxwriter

headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36"
}


def scrape_finviz_data(version):
    url = f"https://finviz.com/screener.ashx?v={version}&r={{page}}&f=all&c=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70&f=ind_stocksonly&o=-marketcap"

    page = 1

    screen = requests.get(url.format(page=page, version=version), headers=headers)
    soup = BeautifulSoup(screen.text, features="lxml")
    pages = int(soup.find_all("a", {"class": "screener-pages"})[-1].text)

    data = []
    
    #for page in range(1, 200, 20):
    for page in range(1, 20 * pages, 20):  
        # print(version, page, url.format(page=page, version=version))
        screen = requests.get(
            url.format(page=page, version=version), headers=headers
        ).text
        tables = pd.read_html(StringIO(screen))
        data.append(tables[-2])

    return pd.concat(data, ignore_index=True)


if __name__ == "__main__":
    version = "152"
    df = scrape_finviz_data(version)
    print(df)

writer = pd.ExcelWriter('Data.xlsx', engine='xlsxwriter')
df.to_excel(writer, sheet_name='Sheet1')
workbook = writer.book
workbook.filename = 'Data.xlsm'
workbook.add_vba_project('./vbaProject.bin')
writer.close()

Visual Studio报错信息 #

File "c:\Users\jkru0\OneDrive\Desktop\vince.py", line 30, in scrape_finviz_data
    tables = pd.read_html(StringIO(screen))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jkru0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\pandas\io\html.py", line 1245, in read_html
    return _parse(
           ^^^^^^^
  File "C:\Users\jkru0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\pandas\io\html.py", line 1008, in _parse
    raise retained
  File "C:\Users\jkru0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\pandas\io\html.py", line 988, in _parse
    tables = p.parse_tables()
             ^^^^^^^^^^^^^^^^
  File "C:\Users\jkru0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\pandas\io\html.py", line 248, in parse_tables
    tables = self._parse_tables(self._build_doc(), self.match, self.attrs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jkru0\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\pandas\io\html.py", line 603, in _parse_tables
    raise ValueError("No tables found")
ValueError: No tables found

1. 请求频率触发反爬拦截 #

Finviz对高频请求有反爬机制，VS环境下代码执行速度可能比命令行更快，导致请求间隔过短被拦截，返回的页面不含表格内容。

解决方法：

• 在循环中添加随机延迟，模拟真实用户访问间隔：

  import time
  import random
  
  # 循环内请求后添加
  time.sleep(random.uniform(1, 3))
  

2. Python环境依赖版本不一致 #

命令行和VS使用的Python解释器或依赖包（如pandas、lxml）版本不同，会导致pd.read_html解析页面的结果存在差异。

解决方法：

• 对比命令行和VS终端的Python版本：执行python --version确认环境一致。
• 统一依赖包版本，重新安装指定版本：

  pip install pandas==2.1.4 lxml==4.9.3 requests==2.31.0
  

3. 请求头被识别为爬虫 #

代码中的User-Agent版本过旧，或VS环境下请求头被修改，导致被网站判定为爬虫，返回无表格的页面。

解决方法：

• 更新User-Agent为当前主流浏览器标识：

  headers = {
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
  }
  

• 添加更多模拟浏览器的请求头字段：

  headers = {
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
      "Accept-Language": "en-US,en;q=0.5",
      "Referer": "https://finviz.com/"
  }
  

4. 页面解析逻辑存在隐患 #

代码直接通过索引tables[-2]获取表格，若部分页面结构变化，会导致索引错误或找不到目标表格。

解决方法：

• 先检查表格数量，避免索引越界：

  tables = pd.read_html(StringIO(screen))
  if len(tables) >= 2:
      data.append(tables[-2])
  else:
      print(f"页面 {page} 表格数量不足，跳过")
  

• 用BeautifulSoup先定位目标表格，再传入pd.read_html解析，提高准确性：

  soup = BeautifulSoup(screen, features="lxml")
  target_table = soup.find("table", class_="screener-table")
  if target_table:
      tables = pd.read_html(StringIO(str(target_table)))
      data.append(tables[0])
  

内容的提问来源于stack exchange，提问作者user3444610