Docker中Selenium报element not interactable但本地运行正常的问题

背景 #

本地运行正常的Python房地产爬虫脚本，Docker化后Selenium操作areaMin输入框时抛出ElementNotInteractableException，Cookie按钮可正常点击，仅后续输入框操作失败。

目标元素HTML #

<input autocomplete="off" inputmode="decimal" name="areaMin" type="text" id="areaMin" placeholder="Powierzchnia od" aria-label="Powierzchnia od" data-cy="search-form--field--areaMin"> 

容器错误日志 #

File "/app/./scrapping_nieruchomosci.py", line 38, in <module>
2024-01-15 13:48:02     initialize_search(driver=driver, min=MIN, max=MAX, city=CITY_PHRASE)
2024-01-15 13:48:02   File "/app/utils/helper_functions.py", line 30, in initialize_search
2024-01-15 13:48:02     min_pow.send_keys(min)
2024-01-15 13:48:02   File "/usr/local/lib/python3.10/site-packages/selenium/webdriver/remote/webelement.py", line 231, in send_keys
2024-01-15 13:48:02     self._execute(
2024-01-15 13:48:02   File "/usr/local/lib/python3.10/site-packages/selenium/webdriver/remote/webelement.py", line 403, in _execute
2024-01-15 13:48:02     return self._parent.execute(command, params)
2024-01-15 13:48:02   File "/usr/local/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py", line 440, in execute
2024-01-15 13:48:02     self.error_handler.check_response(response)
2024-01-15 13:48:02   File "/usr/local/lib/python3.10/site-packages/selenium/webdriver/remote/errorhandler.py", line 245, in check_response
2024-01-15 13:48:02     raise exception_class(message, screen, stacktrace)
2024-01-15 13:48:02 selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable
2024-01-15 13:48:02   (Session info: chrome-headless-shell=120.0.6099.216)

Dockerfile内容 #

# Use an official Python runtime as a parent image
FROM python:3.10

# Make port 80 available to the world outside this container
EXPOSE 80

# Set the working directory in the container
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

RUN pip install --upgrade pip

RUN pip install --trusted-host pypi.python.org -r requirements.txt

RUN apt-get update && apt-get install -y wget unzip && \
    wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb && \
    apt install -y ./google-chrome-stable_current_amd64.deb && \
    rm google-chrome-stable_current_amd64.deb && \
    apt-get clean

# Set display port as an environment variable
ENV DISPLAY=:99

# Run app.py when the container launches
CMD ["python", "./scrapping_nieruchomosci.py"]

核心代码 #

初始化搜索函数 #

def initialize_search(driver, min, max, city):
    driver.get("https://www.otodom.pl/")

    time.sleep(5)
    print("Checking the cookies!!!!!!!!!!!!!")
    try:
        cookies_button = driver.find_element(By.XPATH, value='//*[@id="onetrust-accept-btn-handler"]')
        cookies_button = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="onetrust-accept-btn-handler"]')))
        cookies_button.click()
        print("Clicked on cookies")
    except:
        print("No cookies")
        print(driver.current_url)
    print(driver.current_url)
    #print(driver.page_source)

    time.sleep(10)
    min_pow = driver.find_element(By.ID, value="areaMin")
    #this prints the input element of html from above (the same  prints when run locally)
    print(min_pow.get_attribute("outerHTML"))
    min_pow.send_keys(min)

主脚本 #

import time
import json
from datetime import datetime

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By

from utils.dicts import css_selectors_dict
from utils.helper_functions import get_links_list, get_row_dict, initialize_search

CITY_PHRASE = "Plock"
MIN = 50
MAX = 55
# Get current time and format it as a string in "year-month-day-hour-minute" format
current_time = datetime.now()
time_string = current_time.strftime("%Y-%m-%d-%H-%M")

# Chrome options to run with selenium
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage') # /uses tmp instead of shared memory
chrome_options.add_argument('--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36')


driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=chrome_options)

initialize_search(driver=driver, min=MIN, max=MAX, city=CITY_PHRASE)

已尝试操作 #

替换为显式等待WebDriverWait(driver, 30).until(EC.element_to_be_clickable((By.ID, "areaMin")))，结果超时，无交互错误但元素无法被定位为可点击状态。

解决方案 #

1. 重置Headless窗口尺寸 #

Headless模式默认窗口极小，会导致元素处于不可视区域，添加窗口大小参数：

chrome_options.add_argument('--window-size=1920,1080')

2. 强制滚动到元素位置 #

即使元素存在，也可能因不在可视区域无法交互，用ActionChains滚动到目标元素：

from selenium.webdriver.common.action_chains import ActionChains

# 先等待元素加载完成
min_pow = WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.ID, "areaMin")))
# 滚动到元素可见位置
ActionChains(driver).move_to_element(min_pow).perform()
# 再执行输入操作
min_pow.send_keys(min)

3. 排查iframe/Shadow DOM #

如果搜索框被嵌套在iframe或Shadow DOM中，常规定位方法会失效：

• iframe处理：先切换到对应iframe再定位元素：

  WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, "iframe[src*='search']")))
  

• Shadow DOM处理：通过JS直接获取元素：

  min_pow = driver.execute_script('return document.getElementById("areaMin")')
  min_pow.send_keys(min)
  

4. 对齐Chrome与Driver版本 #

容器内Chrome版本和webdriver-manager下载的Driver版本可能不匹配，固定版本解决：

• 修改Dockerfile安装指定版本Chrome：

  RUN apt-get update && apt-get install -y wget unzip && \
      wget https://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_120.0.6099.216-1_amd64.deb && \
      apt install -y ./google-chrome-stable_120.0.6099.216-1_amd64.deb && \
      rm google-chrome-stable_120.0.6099.216-1_amd64.deb && \
      apt-get clean
  

• 代码中指定对应Driver版本：

  driver = webdriver.Chrome(service=Service(ChromeDriverManager(driver_version="120.0.6099.216").install()), options=chrome_options)
  

5. 绕过网站自动化检测 #

部分网站会拦截Selenium请求，添加参数绕过检测：

chrome_options.add_argument('--disable-blink-features=AutomationControlled')
chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
chrome_options.add_experimental_option('useAutomationExtension', False)

内容的提问来源于stack exchange，提问作者Marcin