从ESCO展开页面提取技能至Pandas DataFrame（含父节点编码）

需要从ESCO网站的「处理和处置废物及危险材料」板块（已展开所有子项）提取技能名称和对应父节点编码，生成包含「Skills」和「Skills codes」列的Pandas DataFrame，且需支持批量处理多个URI。之前的尝试代码因元素选择器匹配错误未成功。

原尝试代码 #

import pandas as pd
from bs4 import BeautifulSoup

# Replace with your actual HTML file path
file_path = '/Users/federiconutarelli/Desktop/esco.html'

# Read the HTML file
with open(file_path, 'r', encoding='utf-8') as file:
   soup = BeautifulSoup(file, 'html.parser')

# Initialize lists to store skills and codes
skills = []
codes = []

# Find elements containing skills and their codes
# Replace 'skill_class_name' and 'code_class_name' with the actual class names or use other selectors based on your HTML structure
for skill_element in soup.find_all('div', class_='classification_item'):
   skill_name = skill_element.get_text(strip=True)
   skills.append(skill_name)
   
   # Assuming the code is in a close relation to the skill element, you might need to adjust the method of finding it
   code_element = skill_element.find_next_sibling('div', class_='main_item')
   if code_element:
       skill_code = code_element.get_text(strip=True)
       codes.append(skill_code)
   else:
       codes.append('')  # Append an empty string or None if no code is found

# Create a DataFrame
df = pd.DataFrame({
   'Skill': skills,
   'Code': codes
})

print(df)

问题分析 #

原代码核心问题是元素选择逻辑错误：

• 父节点编码（如S6.13.0）位于带有main_item类的父容器中，而非技能项的同级元素
• 技能名称是main_item下的子项，需按父板块分组提取，否则会出现编码与技能不匹配的情况

修正后的实现代码 #

import pandas as pd
import requests
from bs4 import BeautifulSoup

def extract_esco_skills(uri):
    # 请求目标URI页面，模拟浏览器避免被拦截
    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'}
    response = requests.get(uri, headers=headers)
    response.encoding = 'utf-8'
    soup = BeautifulSoup(response.text, 'html.parser')
    
    skills = []
    skill_codes = []
    
    # 遍历所有展开的主板块（包含编码和子技能）
    for main_section in soup.find_all('div', class_='main_item'):
        # 提取父节点编码（如S6.13.0）
        code_element = main_section.find('span', class_='code')
        if not code_element:
            continue
        parent_code = code_element.get_text(strip=True)
        
        # 提取当前板块下的所有技能项
        skill_items = main_section.find_all('div', class_='classification_item')
        for item in skill_items:
            skill_name = item.get_text(strip=True)
            skills.append(skill_name)
            skill_codes.append(parent_code)
    
    # 生成目标DataFrame
    return pd.DataFrame({
        'Skills': skills,
        'Skills codes': skill_codes
    })

# 单个URI测试
target_uri = 'http://data.europa.eu/esco/skill/f8c676de-c871-424f-9a65-77059d07910a'
df = extract_esco_skills(target_uri)
print(df)

# 批量处理多个URI
uri_list = [
    'http://data.europa.eu/esco/skill/f8c676de-c871-424f-9a65-77059d07910a',
    # 在此添加更多需要处理的URI
]
all_dfs = [extract_esco_skills(uri) for uri in uri_list]
combined_df = pd.concat(all_dfs, ignore_index=True)
print(combined_df)

关键调整说明 #

1. 直接请求网页：无需保存本地HTML，通过requests库实时获取页面内容，适配批量处理场景
2. 精准元素定位：
   • 父节点编码从main_item下的span.code元素提取，确保编码来源准确
   • 技能项作为main_item的子元素遍历，保证每个技能对应正确的父编码
3. 函数化封装：将提取逻辑封装为函数，方便重复调用多个URI，最后可通过pd.concat合并所有结果

内容的提问来源于stack exchange，提问作者Lusian