如何解决StreamLit滑块触发网页爬虫重复执行的问题？

核心问题是每次交互触发脚本重跑时，爬虫逻辑未被缓存或持久化，导致重复执行。以下是具体修复方案：

方案1：使用@st.cache_data缓存爬取结果 #

StreamLit的@st.cache_data装饰器可自动缓存函数返回结果，相同参数调用时直接返回缓存数据，避免重复爬取。

修改后的代码： #

import streamlit as st
import requests
from bs4 import BeautifulSoup as bs
import re
import pandas as pd

# 用@st.cache_data缓存爬虫函数，避免重复执行
@st.cache_data
def dataframe_creation(linker):
    br = requests.get(linker)
    page = br.content
    soup = bs(page, 'html.parser')
    
    # 提取价格
    advert = soup.find_all("div", attrs={"class": "advert__content-header"})
    price_list = []
    html_code = str(advert)
    matches = re.findall(r'<b>(€)</b>(.*?)</span>', html_code)
    for match in matches:
        number = match[1].split(" ")[0].replace(".", "")
        price_list.append(number)
    
    # 提取城市
    city = soup.find_all("div", attrs={"class": "advert__content-place"})
    city2 = str(city)
    city3 = re.findall(r'>(.*?)</div>', city2)
    city_list = [z.split(",")[0] for z in city3]
    
    # 创建DataFrame并返回
    finale1 = pd.DataFrame({
        "Prices": price_list,
        "Cities": city_list
    })
    finale1['Prices'] = finale1['Prices'].astype(int)
    return finale1

starter = st.checkbox('Begin Data Gathering')
# 替换成你的目标房产网站链接
target_link = "你的目标链接"

if starter:
    # 调用缓存后的函数，仅第一次执行爬取
    finale1 = dataframe_creation(target_link)
    # 把数据存入session_state，确保重跑时能获取
    st.session_state['finale1'] = finale1

# 检查session_state中是否有数据，再渲染滑块和筛选结果
if 'finale1' in st.session_state:
    slider = st.slider("Price in Euros", min_value=200, max_value=5000, step=10)
    filtered_df = st.session_state['finale1'][st.session_state['finale1']['Prices'] <= slider]
    st.dataframe(filtered_df)

方案2：仅用st.session_state持久化数据 #

如果不想用缓存装饰器，也可通过session_state手动存储爬取后的DataFrame，确保仅在第一次勾选时执行爬取：

修改后的代码： #

import streamlit as st
import requests
from bs4 import BeautifulSoup as bs
import re
import pandas as pd

def dataframe_creation(linker):
    br = requests.get(linker)
    page = br.content
    soup = bs(page, 'html.parser')
    
    # 提取价格
    advert = soup.find_all("div", attrs={"class": "advert__content-header"})
    price_list = []
    html_code = str(advert)
    matches = re.findall(r'<b>(€)</b>(.*?)</span>', html_code)
    for match in matches:
        number = match[1].split(" ")[0].replace(".", "")
        price_list.append(number)
    
    # 提取城市
    city = soup.find_all("div", attrs={"class": "advert__content-place"})
    city2 = str(city)
    city3 = re.findall(r'>(.*?)</div>', city2)
    city_list = [z.split(",")[0] for z in city3]
    
    finale1 = pd.DataFrame({
        "Prices": price_list,
        "Cities": city_list
    })
    finale1['Prices'] = finale1['Prices'].astype(int)
    return finale1

starter = st.checkbox('Begin Data Gathering')
target_link = "你的目标链接"

# 仅当session_state中无数据且勾选了checkbox时，执行爬取
if starter and 'finale1' not in st.session_state:
    st.session_state['finale1'] = dataframe_creation(target_link)

# 有数据时才渲染滑块和筛选结果
if 'finale1' in st.session_state:
    slider = st.slider("Price in Euros", min_value=200, max_value=5000, step=10)
    filtered_df = st.session_state['finale1'][st.session_state['finale1']['Prices'] <= slider]
    st.dataframe(filtered_df)

关键修复点说明 #

• 函数返回值：原代码中dataframe_creation函数未添加return语句，导致无法获取生成的DataFrame，修复后明确返回处理好的数据集。
• 持久化存储：通过st.cache_data或st.session_state存储爬取后的DataFrame，确保脚本重跑时直接使用已有数据，不再重复执行爬虫逻辑。
• 条件判断：仅在首次勾选checkbox且未存储数据时，才执行爬取操作，避免重复触发。

内容的提问来源于stack exchange，提问作者Me_and_Michael