使用Scrapy Playwright点击SSO登录按钮无响应问题求助

问题场景 #

使用Scrapy Playwright测试Tealium平台SSO登录功能时，输入错误邮箱并点击提交按钮后，未触发预期的“邮箱不正确”文本错误提示，现有两种实现方式均未达到预期效果：

• 基于PageMethod的流程执行后无任何反馈
• 使用evaluate方法仅能单独完成输入邮箱或点击按钮操作，无法依次执行并触发错误提示

问题分析 #

现有代码存在以下关键问题：

• customer_settings拼写错误，应为custom_settings，导致自定义请求头未生效
• PageMethod中waitForEvent("click")无实际意义，未针对错误提示元素设置等待逻辑
• evaluate方法执行后未等待错误提示渲染，且表单事件触发不完整

修复方案 #

方案1：修正PageMethod执行流程 #

核心调整为等待错误提示元素加载，同时修复配置项拼写错误：

import scrapy
from scrapy_playwright.page import PageMethod
from path import Path

class telSpider(scrapy.Spider):
    name = 'tel'
    start_urls = 'https://my.tealiumiq.com/login/sso/'

    # 修正配置项名称
    custom_settings = {
        'USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15',
        'CONNECTION': 'keep-alive',
        'ACCEPT': 'application/json, text/javascript, */*; q=0.01'
    }

    def start_requests(self):
        yield scrapy.Request(
            self.start_urls,
            meta=dict(
                playwright=True,
                playwright_include_page=True,
                playwright_page_methods=[
                    PageMethod('waitForLoadState', state='domcontentloaded'),
                    # 填充错误邮箱
                    PageMethod("fill", selector='#email', value="incorrect@email.com"),
                    # 等待提交按钮可用
                    PageMethod('wait_for_selector', selector="#submitBtn", state='enabled'),
                    # 简化点击参数，保留必要延迟
                    PageMethod("click", selector="#submitBtn", delay=500, force=True),
                    # 等待错误提示元素出现（需根据页面实际DOM调整选择器）
                    PageMethod('wait_for_selector', selector='.error-message', state='visible'),
                    # 截图验证结果
                    PageMethod("screenshot", path=Path(__file__).parent / "tealium_error.png", full_page=True),
                ]
            ),
            callback=self.parse
        )

    def parse(self, response):
        # 提取并打印错误提示文本
        error_msg = response.css('.error-message::text').get()
        print(f"错误提示内容: {error_msg}")

方案2：修复evaluate方法并添加等待逻辑 #

完善表单事件触发，确保执行JS后等待错误提示渲染：

import scrapy
from scrapy_playwright.page import PageMethod
from path import Path

class telSpider(scrapy.Spider):
    name = 'tel'
    start_urls = 'https://my.tealiumiq.com/login/sso/'

    custom_settings = {
        'USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15',
    }

    def start_requests(self):
        yield scrapy.Request(
            self.start_urls,
            meta=dict(
                playwright=True,
                playwright_include_page=True,
                playwright_page_methods=[
                    PageMethod('waitForLoadState', state='domcontentloaded'),
                    # 执行JS完成输入并点击，补充完整表单事件
                    PageMethod("evaluate", """
                        (function() {
                            const input = document.querySelector('#email');
                            input.value = "incorrect@email.com";
                            // 触发input和change事件，确保表单校验逻辑被触发
                            input.dispatchEvent(new Event('input', { bubbles: true }));
                            input.dispatchEvent(new Event('change', { bubbles: true }));
                            document.querySelector("#submitBtn").click();
                        }());
                    """),
                    # 等待错误提示元素加载
                    PageMethod('wait_for_selector', selector='.error-message', state='visible'),
                    # 截图验证
                    PageMethod("screenshot", path=Path(__file__).parent / "tealium_error_eval.png", full_page=True),
                ]
            ),
            callback=self.parse
        )

    def parse(self, response):
        error_msg = response.css('.error-message::text').get()
        print(f"错误提示内容: {error_msg}")

关键注意事项 #

• 需根据页面实际DOM结构，替换代码中.error-message为真实的错误提示元素选择器
• 避免无意义的事件等待，直接针对目标元素（错误提示）设置等待逻辑
• 确保自定义配置项名称拼写正确，否则请求头无法生效

内容的提问来源于stack exchange，提问作者Dollar Tune-bill