Scrapy_Playwright登录失败但单独Playwright可登录问题求助

问题概述 #

使用Scrapy结合scrapy_playwright组件登录目标网站时，系统提示登录信息错误，但单独用Playwright编写的登录脚本可成功登录，两段代码登录凭据一致（已脱敏），且未收到Scrapy返回的错误信息，需排查原因。

Scrapy爬虫代码 #

class BiosSpider(scrapy.Spider):
    name = "bios"

    def start_requests(self):
        yield (scrapy.Request(url="https://mysite.com.tr/LoginPage",
                              callback=self.parse,
                              meta={"playwright": True,
                                    "playwright_include_page": True,
                                    "playwright_page_methods": [
                                           PageMethod(
                                            "fill", '//div[@class="card-body"]/form[@id="uyeGirisFormBayi"]/div/input[@name="bayiKodu"]',  value="aaaaa"),
                                        PageMethod(
                                            "fill", '//div[@class="card-body"]/form[@id="uyeGirisFormBayi"]/div/input[@name="kullaniciAdi"]', value="aaaaaab"),
                                        PageMethod(
                                            "fill", '//div[@class="card-body"]/form[@id="uyeGirisFormBayi"]/div/input[@name="sifre"]', value="ccccccc"),
                                        PageMethod(
                                            "wait_for_timeout", 5*1000),
                                        PageMethod(
                                            "click", '//div[@class="card-body"]/form[@id="uyeGirisFormBayi"]/button'),
                                    ]
                                    }))

    async def parse(self, response):
        print("QQQQQQQQQ", response)
        print("UUUUUUUUUU", response.request.headers['User-Agent'])
        print("HHHHHHHHHHH", response.request.headers)
        page = response.meta['playwright_page']
        await page.wait_for_timeout(50*1000)

纯Playwright登录代码 #

from playwright.sync_api import sync_playwright


with sync_playwright() as p:
    bwo = p.chromium.launch(headless=False, slow_mo=50)
    page = bwo.new_page()
    page.goto("https://mysite.com.tr/LoginPage")
    page.locator(
        '//div[@class="card-body"]/form[@id="uyeGirisFormBayi"]/div/input[@name="bayiKodu"]'
    ).fill("aaaa"),
    page.locator(
        '//div[@class="card-body"]/form[@id="uyeGirisFormBayi"]/div/input[@name="kullaniciAdi"]').fill("bbbbbb")
    page.locator(
        '//div[@class="card-body"]/form[@id="uyeGirisFormBayi"]/div/input[@name="sifre"]').fill("ccccccc")
    page.locator(
        '//div[@class="card-body"]/form[@id="uyeGirisFormBayi"]/button').click()
    page.wait_for_timeout(500*1000)
    page.close()

Scrapy配置文件 #

USER_AGENT = None
REQUEST_FINGERPRINTER_IMPLEMENTATION = "2.7"
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
FEED_EXPORT_ENCODING = "utf-8"


PLAYWRIGHT_LAUNCH_OPTIONS = {"headless": False}

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

# timeout zamanı
PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT = 600*1000

排查步骤 #

• 核对凭据一致性
  检查两段代码中bayiKodu、kullaniciAdi的真实值：Scrapy代码里是aaaaa和aaaaaab，纯Playwright里是aaaa和bbbbbb，确认真实环境中两者完全一致，排除笔误。

• 添加页面状态日志
  在Scrapy的parse方法中添加代码，获取页面错误提示和当前状态，明确错误细节：

  async def parse(self, response):
      page = response.meta['playwright_page']
      # 获取登录错误提示（替换为实际错误元素选择器）
      error_msg = await page.locator('.login-error').text_content()
      print("登录错误提示:", error_msg)
      # 保存页面截图和HTML到本地
      await page.screenshot(path='login_result.png')
      html = await page.content()
      with open('login_result.html', 'w', encoding='utf-8') as f:
          f.write(html)
      await page.wait_for_timeout(50*1000)
  

• 调整操作时序与等待逻辑
  纯Playwright用了slow_mo=50模拟慢操作，Scrapy仅用固定等待可能导致元素未就绪就执行操作：

  • 替换固定等待为等待元素可交互：

    PageMethod("wait_for_selector", '//div[@class="card-body"]/form[@id="uyeGirisFormBayi"]/div/input[@name="bayiKodu"]', state="visible"),
    PageMethod("fill", '//div[@class="card-body"]/form[@id="uyeGirisFormBayi"]/div/input[@name="bayiKodu"]', value="aaaaa"),
    

  • 点击按钮后等待页面跳转，而非固定等待：

    PageMethod("click", '//div[@class="card-body"]/form[@id="uyeGirisFormBayi"]/button'),
    PageMethod("wait_for_navigation", wait_until="networkidle"),
    

• 统一浏览器环境参数
  同步与异步Playwright环境可能存在差异，统一配置：

  • 在Scrapy配置中添加slow_mo参数：

    PLAYWRIGHT_LAUNCH_OPTIONS = {"headless": False, "slow_mo": 50}
    

  • 显式设置与Playwright一致的User-Agent（可从纯Playwright脚本中获取真实UA）：

    USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36"
    

• 逐步骤调试登录操作
  在Scrapy的parse方法中重新执行登录步骤，观察每一步的执行结果：

  async def parse(self, response):
      page = response.meta['playwright_page']
      # 清空并重新填充输入框
      await page.locator('//div[@class="card-body"]/form[@id="uyeGirisFormBayi"]/div/input[@name="bayiKodu"]').fill("")
      await page.locator('//div[@class="card-body"]/form[@id="uyeGirisFormBayi"]/div/input[@name="bayiKodu"]').fill("aaaaa")
      await page.locator('//div[@class="card-body"]/form[@id="uyeGirisFormBayi"]/div/input[@name="kullaniciAdi"]').fill("")
      await page.locator('//div[@class="card-body"]/form[@id="uyeGirisFormBayi"]/div/input[@name="kullaniciAdi"]').fill("aaaaaab")
      await page.locator('//div[@class="card-body"]/form[@id="uyeGirisFormBayi"]/div/input[@name="sifre"]').fill("")
      await page.locator('//div[@class="card-body"]/form[@id="uyeGirisFormBayi"]/div/input[@name="sifre"]').fill("ccccccc")
      await page.wait_for_timeout(1000)
      await page.locator('//div[@class="card-body"]/form[@id="uyeGirisFormBayi"]/button').click()
      await page.wait_for_timeout(50*1000)
  

内容的提问来源于stack exchange，提问作者Gurhan Goktas