从爬虫所得字典创建DataFrame报could not broadcast错误如何解决

报错原因 #

你遇到的ValueError: could not broadcast input array from shape (3) into shape (1)错误，以及代码存在的其他问题如下：

• 未导入pandas库，无法直接使用pd相关方法
• 存储岗位信息的字典在循环内被反复覆盖，最终仅保留最后1条岗位数据，没有收集全量爬取结果
• 爬取的字段多为BeautifulSoup的Tag对象，未提取为纯文本；find_all返回的列表直接存入字典会导致维度不匹配，触发广播错误
• 异常处理逻辑仅打印空值，未给缺失字段赋值默认值，会出现字段复用前一条数据的错误
• 无效链接过滤逻辑错误，无法正确剔除带#的无效链接

修复后完整代码 #

import requests
from bs4 import BeautifulSoup
import pandas as pd # 导入pandas库

baseurl = "https://www.olx.com.eg/"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36"
}

product_links = []
for x in range(1,13):
    r = requests.get(f"https://www.olx.com.eg/jobs/?page={x}", headers=headers)
    soup = BeautifulSoup(r.content, "lxml")
    product_list = soup.findAll("div", class_="ads__item")
    for item in product_list:
        for link in item.findAll("a",href=True):
            if link['href'] != '#': # 提前过滤无效链接
                product_links.append(link['href'])

# 初始化空列表存储全量岗位数据
all_jobs = []

for link in product_links:
    r = requests.get(link, headers=headers)
    soup = BeautifulSoup(r.content, "lxml")
    # 提取字段纯文本，缺失时赋值为空字符串
    job_title = soup.find('h1',class_="brkword").text.strip() if soup.find('h1',class_="brkword") else ''
    job_location = soup.find('strong',class_="c2b").text.strip() if soup.find('strong',class_="c2b") else ''
    job_date = soup.find('span',class_="pdingleft10 brlefte5").text.strip() if soup.find('span',class_="pdingleft10 brlefte5") else ''
    
    value_tds = soup.find_all('td',class_='value')
    seniority = value_tds[0].text.strip() if len(value_tds)>=1 else ''
    full_or_part = value_tds[1].text.strip() if len(value_tds)>=2 else ''
    education_level = value_tds[2].text.strip() if len(value_tds)>=3 else ''
    sector = value_tds[3].text.strip() if len(value_tds)>=4 else ''
    
    # 把多个描述p标签的文本拼接为单个字符串
    desc_p = soup.find_all('p',class_='pding10')
    description = '\n'.join([p.text.strip() for p in desc_p]) if desc_p else ''

    # 单条岗位数据
    job_info = {
        "Job Title" : job_title,
        "Job Location" : job_location,
        "Post Date" : job_date,
        "Seniority Level" : seniority,
        "Full or Part time" : full_or_part,
        "Educational Level" : education_level,
        "Sector" : sector,
        "Job Description" : description
    }
    # 追加到全量列表
    all_jobs.append(job_info)

# 全量数据转DataFrame
job_data = pd.DataFrame(all_jobs)
# 导出为CSV，encoding用utf-8-sig避免Windows下打开乱码
job_data.to_csv("olx_egypt_jobs.csv", index=False, encoding='utf-8-sig')

验证说明 #

代码运行后会在当前目录生成olx_egypt_jobs.csv文件，所有爬取的岗位数据会按行存储，可直接用于后续数据分析。

内容的提问来源于stack exchange，提问作者Seif Mahdi