如何遍历LinkedIn链接时为Pandas DataFrame动态新增列？

要实现每爬取一份LinkedIn资料就新增一列，核心思路是将单份资料的所有属性整理为一个序列（Series），再将这个序列作为新列横向拼接到DataFrame中，具体实现如下：

步骤1：初始化结构明确的空DataFrame #

先定义所有资料的固定属性作为行索引，确保后续每列的行顺序一致：

import pandas as pd

# 定义所有资料属性，作为DataFrame的行索引
row_attrs = [
    "name",
    "location",
    "company",
    "date",
    "job_location",
    "description",
    "college",
    "degree"
]

# 初始化空DataFrame，行是属性，列后续动态添加
profile_df = pd.DataFrame(index=row_attrs)

步骤2：遍历链接，爬取单份资料并添加为新列 #

遍历每个LinkedIn链接时，单独处理当前资料的所有信息，整理成Series后拼接到DataFrame：

from selenium import webdriver
from selenium.webdriver.common.by import By

# 假设你的链接列表为links
links = ["link1", "link2", "link3"]
driver = webdriver.Chrome()

for idx, link in enumerate(links, start=1):
    driver.get(link)
    
    # 1. 爬取基础信息
    name = driver.find_element(By.CSS_SELECTOR, "YOUR_NAME_SELECTOR").text  # 替换为你实际的name选择器
    location = driver.find_element(By.CSS_SELECTOR, "YOUR_LOCATION_SELECTOR").text  # 替换为实际的location选择器
    
    # 2. 爬取并整理工作经历
    exp = {"company": [], "date": [], "job_location": [], "description": []}
    jobs = driver.find_elements(By.CSS_SELECTOR, 'section:has(#experience)>div>ul>li')
    for job in jobs:
        # 用or ''处理空值，避免报错
        exp["company"].append(driver.execute_script('return arguments[0].querySelector("span[class=\'t-14 t-normal\']:nth-child(2) span")?.innerText', job) or '')
        exp["date"].append(driver.execute_script('return arguments[0].querySelector("span[class=\'t-14 t-normal t-black--light\']:nth-child(3) span")?.innerText', job) or '')
        exp["job_location"].append(driver.execute_script('return arguments[0].querySelector("span[class=\'t-14 t-normal t-black--light\']:nth-child(4) span")?.innerText', job) or '')
        exp["description"].append(driver.execute_script('return arguments[0].querySelector("ul li ul span[aria-hidden=true]")?.innerText', job) or '')
    
    # 将多份工作经历用逗号拼接成字符串
    company_str = ', '.join(filter(None, exp["company"]))
    date_str = ', '.join(filter(None, exp["date"]))
    job_loc_str = ', '.join(filter(None, exp["job_location"]))
    desc_str = ', '.join(filter(None, exp["description"]))
    
    # 3. 爬取并整理教育经历
    ed = {"College": [], "Degree": []}
    edu = driver.find_elements(By.CSS_SELECTOR, 'section:has(#education)>div>ul>li')
    for education in edu:
        ed["College"].append(education.find_element(By.CSS_SELECTOR, 'span[class=\'mr1 hoverable-link-text t-bold\'] span').text)
        ed["Degree"].append(driver.execute_script('return arguments[0].querySelector("span[class=\'t-14 t-normal\']:nth-child(2) span")?.innerText', education) or '')
    
    college_str = ', '.join(filter(None, ed["College"]))
    degree_str = ', '.join(filter(None, ed["Degree"]))
    
    # 4. 将当前资料整理为Series，作为新列
    current_profile = pd.Series(
        [name, location, company_str, date_str, job_loc_str, desc_str, college_str, degree_str],
        index=row_attrs,
        name=f'column{idx}'  # 列名按顺序命名为column1、column2...
    )
    
    # 5. 横向拼接新列到DataFrame
    profile_df = pd.concat([profile_df, current_profile], axis=1)

driver.quit()
# 查看最终结果
print(profile_df)

原代码问题说明 #

你之前的代码直接将所有资料的属性列表合并成大列表后赋值，会导致所有数据按行堆叠，无法区分单份资料。而新方案通过单份资料单独处理→生成对应Series→横向拼接的方式，自然实现了动态新增列的需求，无需预先定义列名。

内容的提问来源于stack exchange，提问作者Jesper Ezra