Google App Engine中Flask服务Selenium无效会话ID问题求助

在Google App Engine的Flask服务中运行Selenium时，循环获取子元素环节始终抛出「invalid session id」错误，已排查多日未找到原因。该代码在Cloud Run/Jupyter环境可正常运行，已打印循环前后的webdriver会话ID，显示一致，但问题仍存在。

Driver初始化代码 #

import chromedriver_binary

# User agent for desktop
user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36'

options = Options()

# specify headless mode
options.add_argument('--headless')

options.add_argument(f'user-agent={user_agent}')

options.add_argument("--window-size=1920,1080")

options.add_argument("--disable-gpu")
options.add_argument("--no-sandbox")
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)

options.add_argument('--disable-dev-shm-usage')     

wd = webdriver.Chrome(options=options)

出错的业务代码片段 #

问题出在循环获取子元素环节：

@app.route("/scrape-stuff")
def scrape_stuff():
    
    try:
        wd.get(".....")
        time.sleep(3)
    except Exception as e:
        send_sms("Critical error")
        sys.stdout.write("Critical error" + str(e))

    # this array holds the result of the scrape
    result_array=[]
    # accept cookies button
    try:
        wd.execute_script('...').click()
    except:
        pass

    for i in range(1,2):
        # Select from the dropdown 
        try:
            wd.execute_script(f"document.querySelector('....').click()")
            time.sleep(2)
        except Exception as e:
            send_sms("Critical error: cannot select from dropdown"+str(e))
            sys.stdout.write("ERROR")
        
        # Press all Load more -until there is nothing left 
        view_more_button=1
        while view_more_button == 1:
            try:
                wd.execute_script('document.querySelector("....").click()')
                time.sleep(5)
            except :
                view_more_button = 0
                pass
        
        # Get all the items 
        elements= wd.find_elements('xpath','//div[@class="....."]') 

        # wd.save_screenshot("scraper.png")
        # return send_file("scraper.png")   <----  SHOWS EXPECTED OUTPUT (also ,I am not blocked or asked for Captcha)
    
        for element in elements :
            # *此处始终抛出InvalidSessionIdException*
            sub_element=element.find_element('xpath','//div[@class="...."]')
 
    return "OK"

错误堆栈信息 #

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2073, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1518, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1516, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1502, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/app/main.py", line 360, in scrape_stuff
sub_element = elements[element_index].find_element('xpath',".//span[@class='....']").get_attribute('innerHTML')
File "/usr/local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 634, in execute_script
return self.execute(command, {
File "/usr/local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.InvalidSessionIdException: Message: invalid session id

会话ID打印信息 #

2022-08-24 11:46:32.000 CEST
SESSION ID BEFORE LOOP d361399a553f0ce677665c0326156c23
2022-08-24 11:46:32.000 CEST
WEBDRIVER OBJECT <selenium.webdriver.chrome.webdriver.WebDriver (session="d361399a553f0ce677665c0326156c23")>
2022-08-24 11:46:32.000 CEST
SESSION ID IN THE LOOP d361399a553f0ce677665c0326156c23

内容的提问来源于stack exchange，提问作者Peter Malik