Python中用Selenium/Beautiful Soup获取动态表格单元格数据遇阻求助

Hey there, let's break down why you're having trouble grabbing that dynamic policy number from the table cell, and fix it step by step.

先搞清楚问题出在哪 #

Looking at your HTML snippet, the dynamic value you want is stored in an <input> element with id="policy_number". The issues with your current approaches are:

1. Selenium XPath方法: You might have been targeting the wrong element (like the parent div or td instead of the input itself), or relying on unstable time.sleep() instead of waiting for the element to fully load. Also, input elements store their content in the value attribute—not innerText or text.
2. BeautifulSoup方法: BS can't execute JavaScript, so if you fetch the page source before the JS finishes rendering the input's value, you'll get empty or unpopulated HTML. Using lxml didn't help because the root problem is timing, not the parser.

Here's your target HTML for reference:

方案1：用Selenium正确获取动态值（推荐） #

Ditch the fixed time.sleep() and use explicit waits to ensure the input element is loaded and populated. Here's the corrected code:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# 打开页面
crom_driver.get("https://cms.bharti-axagi.co.in/home/CMS/com/bagi/cms/Loginforms/CMS_LoginScreen.caf")

# 显式等待input元素出现并可访问（最多等10秒，避免不稳定的sleep）
policy_input = WebDriverWait(crom_driver, 10).until(
    EC.presence_of_element_located((By.ID, "policy_number"))
)

# 获取input的value属性，这就是你要的政策号
previous_Claim_Policy = policy_input.get_attribute("value")
print("Policy Number:", previous_Claim_Policy)

为什么这个能行？ #

• Explicit waits wait until the element is actually present in the DOM, so you don't get None because the element wasn't loaded yet.
• We're targeting the <input> directly, not its parent elements, so we can access its value attribute correctly.

方案2：如果一定要用BeautifulSoup #

If you need to use BS, you still need Selenium to wait for the page to fully render before grabbing the source. Here's how:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup as bs

# 打开页面并等待元素加载
crom_driver.get("https://cms.bharti-axagi.co.in/home/CMS/com/bagi/cms/Loginforms/CMS_LoginScreen.caf")
WebDriverWait(crom_driver, 10).until(
    EC.presence_of_element_located((By.ID, "policy_number"))
)

# 现在获取已经渲染完成的页面源码
content = crom_driver.page_source
soup = bs(content, "html.parser")

# 定位到input元素并获取value
policy_input = soup.find("input", {"id": "policy_number"})
if policy_input:
    policy_number = policy_input.get("value")
    print("Policy Number:", policy_number)
else:
    print("Could not find the policy number input element")

为什么之前用lxml没结果？ #

You probably fetched the page source too early, before the JS populated the input. Waiting for the element first ensures the value is present in the HTML when you parse it.

额外排查点 #

• Check for iframes: If the table is inside an iframe, you need to switch to it first with crom_driver.switch_to.frame("iframe_id_or_name") before locating elements.
• Stable selectors: If the id changes dynamically, use a more reliable XPath like //td/div[@class='fieldsbox']/input[@xql='tns:CHDRNUM'] to target the element.

内容的提问来源于stack exchange，提问作者Hietsh Kumar