BeautifulSoup网页爬取异常：数据仅显示单列，zip_longest未解决问题

问题描述 #

使用BeautifulSoup爬取网页后，导出的CSV文件所有数据挤在一列，未按预期分配到对应列。已尝试使用zip_longest和解包操作实现标题对应列、详情按行匹配，但问题仍未解决。

代码如下：

import requests
from bs4 import BeautifulSoup
import csv
from itertools import zip_longest

job_title = []
company_name = []
location_name = []
skills = []

#2 use requests to fetch the url
result = requests.get ("https://wuzzuf.net/search/jobs/?q=programmer&a=hpb")

#3 save page content/markup
src = result.content

#4 create soup object to parse content
soup = BeautifulSoup (src, "lxml")

#5 find the elements containing info we need 
#-- job titles, job skills, company names, location names 

job_titles = soup.find_all("h2",{"class":"css-m604qf"})
company_names = soup.find_all("a",{"class":"css-17s97q8"})
locations_names = soup.find_all("span",{"class":"css-5wys0k"})
job_skills = soup.find_all("div",{"class":"css-y4udm8"})

#6 loop over returned lists to extract needed info other lists

for i in range(len(job_titles)):
    job_title.append(job_titles[i].text)
    company_name.append(company_names[i].text)
    location_name.append(locations_names[i].text)
    skills.append(job_skills[i].text)

#7 create csv file and fill it with values
file_list = [job_title, company_name, location_name, skills]
exported = zip_longest(*file_list)

with open("job.csv", "w" ) as myfile:
    wr = csv.writer(myfile)
    wr.writerow(["job title", "company name", "location", "skills"])
    wr.writerows(exported)

问题原因与解决方法 #

核心问题 #

CSV数据挤在一列的主要原因是文件编码未适配Windows环境，默认编码下Excel等工具无法正确识别列分隔符；其次是爬取的文本包含换行、多余空格等特殊字符，干扰CSV的行结构。

修正后的代码 #

import requests
from bs4 import BeautifulSoup
import csv
from itertools import zip_longest

job_title = []
company_name = []
location_name = []
skills = []

# 请求网页
result = requests.get("https://wuzzuf.net/search/jobs/?q=programmer&a=hpb")
src = result.content
soup = BeautifulSoup(src, "lxml")

# 定位元素（简化class写法）
job_titles = soup.find_all("h2", class_="css-m604qf")
company_names = soup.find_all("a", class_="css-17s97q8")
locations_names = soup.find_all("span", class_="css-5wys0k")
job_skills = soup.find_all("div", class_="css-y4udm8")

# 提取并清洗数据
for i in range(len(job_titles)):
    job_title.append(job_titles[i].text.strip())
    company_name.append(company_names[i].text.strip())
    location_name.append(locations_names[i].text.strip())
    # 替换技能文本中的换行符，避免破坏CSV行结构
    skills.append(job_skills[i].text.replace("\n", " ").strip())

# 导出CSV
file_list = [job_title, company_name, location_name, skills]
exported = zip_longest(*file_list)

# 指定UTF-8带BOM编码+控制换行符，解决列错位和空行问题
with open("job.csv", "w", encoding="utf-8-sig", newline="") as myfile:
    wr = csv.writer(myfile)
    wr.writerow(["job title", "company name", "location", "skills"])
    wr.writerows(exported)

关键改动说明 #

• 编码适配：使用encoding="utf-8-sig"生成带BOM的UTF-8文件，确保Windows平台的表格工具能正确识别列分隔符，避免所有数据挤在一列。
• 文本清洗：通过.strip()去除文本首尾空格，用.replace("\n", " ")替换技能文本中的换行符，防止特殊字符破坏CSV的行结构。
• 换行符控制：添加newline=""参数，让csv.writer自动处理换行逻辑，避免Windows下导出的CSV出现多余空行。

额外优化建议 #

如果爬取的文本包含大量逗号，可以在初始化csv.writer时指定制表符作为分隔符，进一步避免列错位：

wr = csv.writer(myfile, delimiter="\t")

内容的提问来源于stack exchange，提问作者Lpython