Zerostep Playwright点击Login误触Register的调试咨询

当使用Zerostep Playwright操作Web应用时，执行await ai('Click on the Login link in the header', { page, test })指令，实际点击了Register按钮，以下是具体调试定位方法：

调试定位步骤 #

• 验证DOM元素标识匹配
  从日志可见，hover和click操作都指向了ID为73的元素。需确认该ID对应的实际元素：

  1. 在浏览器开发者工具中执行document.getElementById('73')，查看元素是Login还是Register按钮。
  2. 检查页面DOM结构，确认Login和Register按钮是否存在ID混淆、嵌套覆盖或选择器边界模糊问题。

• 优化AI指令精确性
  Zerostep依赖自然语言解析，指令模糊易导致模型误判：

  1. 补充元素特征，例如修改指令为Click on the Login link in the header, located to the [left/right] of the Register button（根据页面实际布局调整方位描述）。
  2. 直接指定元素文本或属性，例如Click on the link with exact text 'Login' in the header。

• 查看完整DOM快照
  当前日志中DOM内容被截断，需获取完整快照：
  调整日志输出配置，确保完整打印Zerostep发送的snapshot.dom内容，分析模型接收的页面信息是否准确，比如Login按钮的文本、位置是否在快照中正确呈现。

• 对比页面状态变化
  检查hover操作后的页面状态：
  hover可能触发菜单展开或元素位置偏移，导致后续click目标错误。可在hover后添加await page.waitForTimeout(1000)等待页面稳定，或在click指令中明确指定hover后的目标元素。

• 降级为原生Playwright验证
  用原生Playwright选择器直接定位Login按钮，验证元素是否可正常点击：

  // 替换AI指令，用原生方式测试
  await page.click('text=Login');
  // 或使用更精确的选择器
  await page.click('header >> text=Login');
  

  如果原生方式可正常点击，说明问题出在Zerostep的AI解析环节；如果同样点击错误，说明页面本身存在元素定位冲突。

• 升级Zerostep版本
  当前使用的@zerostep/playwright版本为0.1.5，属于较早版本，新版本可能修复了元素识别bug：
  执行npm update @zerostep/playwright升级到最新稳定版后重新测试。

用户提供的测试代码 #

test('has title', async ({ page }) => {
  await page.goto('http://eaapp.somee.com/');

  /* Expect a title "to contain" a substring. */
  await ai('Verify that title contains Execute Automation', {page, test})
  await ai('Hover on to Login link', {page, test})
  await ai('Click on the Login link in the header', {page, test})
  await ai('Fill the fields with realistic values', {page, test})

});

package.json内容 #

{ "devDependencies": { "@playwright/test": "^1.40.1", "@types/node": "^20.10.4", "@zerostep/playwright": "^0.1.5", "playwright": "1.40.1" }, "scripts": { "test": "npx playwright test" } }

AI函数运行日志 #

ws send: {"type":"task-start","packageVersion":"v0.1.5","taskId":"018c5db5-004b-7bfe-ad1c-47850643b9a8","task":"Verify that title contains Execute Automation","snapshot":{"dom":"{"documents":[{"documentURL":0,"title":1,"baseURL":0,"contentLanguage":
< ws recv: {"type":"task-complete","taskId":"018c5db5-004b-7bfe-ad1c-47850643b9a8","model":"gpt-3.5-turbo","wasSuccessful":true,"result":{"assertion":false}}
ws send: {"type":"task-start","packageVersion":"v0.1.5","taskId":"018c5db5-1381-740c-86b6-981c04b62564","task":"Hover on to Login link","snapshot":{"dom":"{"documents":[{"documentURL":0,"title":1,"baseURL":0,"contentLanguage":-1,"encodingName":2,

< ws recv: {"type":"command-request","taskId":"018c5db5-1381-740c-86b6-981c04b62564","index":0,"name":"hoverElement","arguments":{"id":"73"}}
ws send: {"type":"command-response","packageVersion":"v0.1.5","taskId":"018c5db5-1381-740c-86b6-981c04b62564","index":0,"result":"null"}
< ws recv: {"type":"task-complete","taskId":"018c5db5-1381-740c-86b6-981c04b62564","model":"gpt-3.5-turbo","wasSuccessful":true,"result":{"actions":["Hover"]}}
ws send: {"type":"task-start","packageVersion":"v0.1.5","taskId":"018c5db5-1b0c-7508-9555-141536f29a1b","task":"Click on the Login link in the header","snapshot":{"dom":"{"documents":[{"documentURL":0,"title":1,"baseURL":0,"contentLanguage":-1,"enc
< ws recv: {"type":"command-request","taskId":"018c5db5-1b0c-7508-9555-141536f29a1b","index":1,"name":"clickElement","arguments":{"id":"73"}}
ws send: {"type":"command-response","packageVersion":"v0.1.5","taskId":"018c5db5-1b0c-7508-9555-141536f29a1b","index":1,"result":"null"}
< ws recv: {"type":"task-complete","taskId":"018c5db5-1b0c-7508-9555-141536f29a1b","model":"gpt-3.5-turbo","wasSuccessful":true,"result":{"actions":["Click"]}}
ws send: {"type":"task-start","packageVersion":"v0.1.5","taskId":"018c5db5-24f0-7453-b031-966813fefde3","task":"Fill the fields with realistic values","snapshot":{"dom":"{"documents":[{"documentURL":0,"title":1,"baseURL":0,"contentLanguage":-1,"enc

内容的提问来源于stack exchange，提问作者Muhammad Tufail Khan