爬虫时用NA填充缺失文本值：BeautifulSoup与DataFrame对齐问题排查

用BeautifulSoup抓取数据生成两个不同类别的DataFrame，第一个DataFrame存在部分缺失值，合并时出现行对齐问题。尝试用if not语句处理缺失值，但仍触发'get_text() is empty'错误，相关代码如下：

soup = bs(response.text, 'html5lib')
for x in soup.find_all("div", {"class": "details_table"}):
    S = x.find("span", {"class": "search_line_s details_table_data"}).get_text(strip=True)
    if not x.find("span", {"class": "search_line_a__ details_table_data"}).get_text(strip=True):
        A = "N/A"
    else:
        A = x.find("span", {"class": "search_line_a__ details_table_data"}).get_text(strip=True)
    App = x.find("span", {"class": "search_line_app details_table_data"}).get_text(strip=True)
    df3.loc[len(df3.index)] = [S, A, App]
for items in soup.find_all("a", {"class": "e_link"}):
    item_at = items.attrs
    list_of_dict_values = item_at.values()
    good_objects = [True, False, True, False, True, False, True, True, False, True, False, False, False]
    property_asel = [val for is_good, val in zip(good_objects, list_of_dict_values) if is_good]
    link = property_asel[0]
    type = property_asel[1]
    name = property_asel[2]
    category = property_asel[3]
    sub_category = property_asel[4]
    price = property_asel[5]
    df.loc[len(df.index)] = [name, category, sub_category, type, price, link]
    fac.append(items.get_text(strip=True))
    result = pd.concat([df, df3], axis=1)

1. 解决get_text()报错问题 #

报错核心原因是直接调用get_text()前未检查标签是否存在：如果x.find()返回None，后续调用get_text()会直接报错。同时，即使标签存在，get_text(strip=True)可能返回空字符串，也需要处理。优化后的代码逻辑如下：

soup = bs(response.text, 'html5lib')

# 处理df3的逻辑优化
for x in soup.find_all("div", {"class": "details_table"}):
    # 处理S字段：先判断标签是否存在，不存在则设为N/A
    s_tag = x.find("span", {"class": "search_line_s details_table_data"})
    S = s_tag.get_text(strip=True) if s_tag else "N/A"
    
    # 处理A字段：先找标签，再判断文本是否为空
    a_tag = x.find("span", {"class": "search_line_a__ details_table_data"})
    if a_tag:
        a_text = a_tag.get_text(strip=True)
        A = a_text if a_text else "N/A"
    else:
        A = "N/A"
    
    # 处理App字段：同样先检查标签存在性
    app_tag = x.find("span", {"class": "search_line_app details_table_data"})
    App = app_tag.get_text(strip=True) if app_tag else "N/A"
    
    df3.loc[len(df3.index)] = [S, A, App]

# 处理df的逻辑优化，增加长度判断避免索引越界
for items in soup.find_all("a", {"class": "e_link"}):
    item_at = items.attrs
    list_of_dict_values = item_at.values()
    good_objects = [True, False, True, False, True, False, True, True, False, True, False, False, False]
    property_asel = [val for is_good, val in zip(good_objects, list_of_dict_values) if is_good]
    
    # 确保property_asel有足够元素再取值，避免索引报错
    if len(property_asel) >= 6:
        link = property_asel[0]
        type = property_asel[1]
        name = property_asel[2]
        category = property_asel[3]
        sub_category = property_asel[4]
        price = property_asel[5]
        df.loc[len(df.index)] = [name, category, sub_category, type, price, link]
        fac.append(items.get_text(strip=True))

# 合并操作移到循环外，避免重复拼接
result = pd.concat([df, df3], axis=1)

2. 解决合并行对齐问题 #

• 检查元素数量匹配：确认soup.find_all("div", {"class": "details_table"})和soup.find_all("a", {"class": "e_link"})返回的元素数量一致，这是行对齐的前提。如果数量不匹配，需要调整抓取逻辑，确保每个条目对应一组数据。
• 用索引合并替代直接拼接：给两个DataFrame添加相同的序号列，再用pd.merge按序号合并，避免因行数不一致导致的错位：

  # 给df添加序号列
  df['id'] = range(len(df))
  # 给df3添加序号列
  df3['id'] = range(len(df3))
  # 按id合并
  result = pd.merge(df, df3, on='id', how='outer')
  

内容的提问来源于stack exchange，提问作者Munrock