使用Selenium爬取Google Jobs存入DataFrame时字段错乱求助

我是StackOverflow新手，若帖子结构不佳请见谅。我正在学习Python网页爬虫，作为个人项目，尝试用Selenium爬取Google Jobs（无限滚动加载，每次滚动加载10条）的数据并存入Pandas DataFrame。目前已实现滚动加载并提取条目文本，但每个职位条目包含3-6行不同属性（如职位名称、公司名等），用\n分割后得到长度不一的列表，导致生成的DataFrame字段顺序错乱。以下是我的代码，恳请帮助生成结构规整的DataFrame：

#imports
import pandas as pd
import numpy as np
from serpapi import GoogleSearch
import requests
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

#using selenium to launch and scroll through the Google Jobs page
url = "https://www.google.com/search?q=google+jobs+data+analyst&oq=google+jobs+data+analyst&aqs=chrome..69i57j69i59j0i512j0i22i30i625l4j69i60.4543j0j7&sourceid=chrome&ie=UTF-8&ibp=htl;jobs&sa=X&ved=2ahUKEwjXsv-_iZP9AhVPRmwGHX5xDEsQutcGKAF6BAgPEAU&sxsrf=AJOqlzWGHNISzgpAUCZBmQA1mWXXt3I7gA:1676311105893#htivrt=jobs&htidocid=GS94rKdYQqQAAAAAAAAAAA%3D%3D&fpstate=tldetail"
driver = webdriver.Chrome()
driver.get(url)
joblist =[]

#pointing to the html element to scroll to
elementxpath = '//*[@id="immersive_desktop_root"]/div/div[3]/div[1]/div[1]/div[3]/ul/li[10]'
element = driver.find_element(By.XPATH,elementxpath)
driver.execute_script('arguments[0].scrollIntoView(true)',element)
datas = driver.find_elements(By.XPATH,'//*[@id="immersive_desktop_root"]/div/div[3]/div[1]/div[1]/div[3]/ul/li') 
joblist.append([da.text for da in datas])

#adding 3s delay for website to load after scrolling before executing code
time.sleep(3)

#capturing all the job list objects in the second set of 10 results loaded after 1st scroll down
elementxpath = '//*[@id="VoQFxe"]/div/div/ul/li[10]'
element = driver.find_element(By.XPATH,elementxpath)
driver.execute_script('arguments[0].scrollIntoView(true)',element)
datas = driver.find_elements(By.XPATH,'//*[@id="VoQFxe"]/div/div/ul/li')
joblist.append([da.text for da in datas])
x=2
time.sleep(3)

#using a while loop to scroll and capture for the remaining scroll downs as element xpath is in iterable format unlike th previous 2 xpaths
while True:
    elementxpath = '//*[@id="VoQFxe"]/div['+str(1*x)+']/div/ul/li[10]'
    element = driver.find_element(By.XPATH,elementxpath)
    driver.execute_script('arguments[0].scrollIntoView(true)',element)
    x+=1
    time.sleep(3)
    datas = driver.find_elements(By.XPATH,'//*[@id="VoQFxe"]/div['+str(1*x)+']/div/ul/li')
    joblist.append([da.text for da in datas])
    if x>1000:
        break
    else:
        continue

#unpacking and cleaning captured values from joblist to a newlist of lists in the desired format for creating a dataframe
jlist = []
for n in joblist:
    for a in range(0,len(n)-1):
        if n[a]!='':
            jlist.append(n[a].split('\n'))

jobdf = pd.DataFrame(jlist)
jobdf.columns = ['Logo','Role', 'Company', 'Source','Posted','Full / Part Time', 'Waste']
jobdf

1. 精准定位每个字段元素，避免文本分割错位 #

不要直接提取整个列表项的文本再拆分，而是针对每个职位卡片里的具体字段单独定位元素，这样每个字段对应明确，不会因为条目行数不同导致顺序混乱。Google Jobs的职位卡片有固定的类名，可以基于这些类名定位：

• 职位名称：h2.BjJfJf.PUpOsf
• 公司名称：div.vNEEBe > span:first-child
• 来源平台：div.vNEEBe > span.EimVq.eFXRqe
• 发布时间：span.LL4CDc
• 工作类型：div.Qk80Jf

2. 优化滚动加载逻辑，避免XPATH索引失效 #

原代码依赖固定索引的XPATH容易因为页面结构变化失效，改成通过滚动页面底部触发加载，同时记录当前职位数量，当数量不再增加时停止滚动，避免无效循环。

修改后的代码 #

import pandas as pd
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# 初始化浏览器
url = "https://www.google.com/search?q=google+jobs+data+analyst&oq=google+jobs+data+analyst&aqs=chrome..69i57j69i59j0i512j0i22i30i625l4j69i60.4543j0j7&sourceid=chrome&ie=UTF-8&ibp=htl;jobs&sa=X&ved=2ahUKEwjXsv-_iZP9AhVPRmwGHX5xDEsQutcGKAF6BAgPEAU&sxsrf=AJOqlzWGHNISzgpAUCZBmQA1mWXXt3I7gA:1676311105893#htivrt=jobs&htidocid=GS94rKdYQqQAAAAAAAAAAA%3D%3D&fpstate=tldetail"
driver = webdriver.Chrome()
driver.get(url)
wait = WebDriverWait(driver, 10)

# 存储职位数据的列表
job_data = []

# 滚动加载逻辑：循环滚动直到没有新内容
previous_count = 0
while True:
    # 等待职位卡片加载完成
    job_cards = wait.until(EC.presence_of_all_elements_located((By.CLASS_NAME, "MjjYud")))
    current_count = len(job_cards)
    
    # 如果没有新职位加载，退出循环
    if current_count == previous_count:
        break
    previous_count = current_count
    
    # 遍历每个职位卡片，提取字段
    for card in job_cards:
        job_info = {}
        # 提取职位名称
        try:
            job_info["Role"] = card.find_element(By.CSS_SELECTOR, "h2.BjJfJf.PUpOsf").text
        except:
            job_info["Role"] = None
        # 提取公司名称
        try:
            job_info["Company"] = card.find_element(By.CSS_SELECTOR, "div.vNEEBe > span:first-child").text
        except:
            job_info["Company"] = None
        # 提取来源平台
        try:
            job_info["Source"] = card.find_element(By.CSS_SELECTOR, "div.vNEEBe > span.EimVq.eFXRqe").text
        except:
            job_info["Source"] = None
        # 提取发布时间
        try:
            job_info["Posted"] = card.find_element(By.CSS_SELECTOR, "span.LL4CDc").text
        except:
            job_info["Posted"] = None
        # 提取工作类型
        try:
            job_info["Full / Part Time"] = card.find_element(By.CSS_SELECTOR, "div.Qk80Jf").text
        except:
            job_info["Full / Part Time"] = None
        
        # 避免重复添加
        if job_info not in job_data:
            job_data.append(job_info)
    
    # 滚动到页面底部触发加载
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(3)

# 生成DataFrame
jobdf = pd.DataFrame(job_data)
print(jobdf)

# 关闭浏览器
driver.quit()

代码说明 #

• 使用WebDriverWait替代固定sleep，更高效地等待元素加载
• 针对每个字段单独捕获，缺失字段填充None，保证DataFrame结构规整
• 滚动逻辑通过比较职位数量判断是否加载完成，避免无限循环
• 避免重复添加相同职位数据

内容的提问来源于stack exchange，提问作者Richard T Vetticad