从不同子目录提取文本至Pandas DataFrame遇ValueError报错的解决方法

我尝试从不同子目录的文本文件中提取内容并存入Pandas DataFrame，但运行代码时触发ValueError: substring not found错误。

文本数据示例 #

"EXAMINATION: CHEST PA AND LAT INDICATION: History: F with shortness of breath TECHNIQUE: Chest PA and lateral COMPARISON: FINDINGS: The cardiac mediastinal and hilar contours are normal. Pulmonary vasculature is normal. Lungs are clear. No pleural effusion or pneumothorax is present. Multiple clips are again seen projecting over the left breast. Remote leftsided rib fractures are also re demonstrated. IMPRESSION: No acute cardiopulmonary abnormality."

报错信息 #

ValueError                                Traceback (most recent call last)
<ipython-input-108-bbeeb452bdef> in <module>
     48         df = pd.DataFrame(columns=keywords)
     49         # Extract text
---> 50         result = extract_text_using_keywords(text, keywords)
     51         # Append list of extracted text to the end of the pandas df
     52         df.loc[len(df)] = result

<ipython-input-108-bbeeb452bdef> in extract_text_using_keywords(clean_text, keyword_list)
     39             for prev_kw, current_kw in zip(keyword_list, keyword_list[1:]):
     40                 prev_kw_index = clean_text.index(prev_kw)
---> 41                 current_kw_index = clean_text.index(current_kw)
     42                 extracted_texts.append(clean_text[prev_kw_index + len(prev_kw) + 2:current_kw_index])
     43                 if current_kw == keyword_list[-1]:

ValueError: substring not found

运行代码 #

out = []
result = {}

for filename in glob.iglob('/content/sample_data/**/*.txt', recursive = True):
    
    out.append(filename)

print('File names: ',out)

for file in out:
      
        with open(file) as f:
          data = f.read()
          
    
        import re
        text = re.sub(r"[-_()\n\"#//@;<>{}=~|?,]*", "", data)
        text = re.sub(r'FINAL REPORT', '', text)
        text = re.sub(r'\s+', ' ', text)
        print(text)

        keywords = ["INDICATION", "TECHNIQUE", "COMPARISON", "FINDINGS", "IMPRESSION"]

        # Create function to extract text between each of the keywords
        # Assumption
        def extract_text_using_keywords(clean_text, keyword_list):
            extracted_texts = []
            for prev_kw, current_kw in zip(keyword_list, keyword_list[1:]):
                prev_kw_index = clean_text.index(prev_kw)
                current_kw_index = clean_text.index(current_kw)
                extracted_texts.append(clean_text[prev_kw_index + len(prev_kw) + 2:current_kw_index])
                if current_kw == keyword_list[-1]:
                    extracted_texts.append(clean_text[current_kw_index + len(current_kw) + 2:len(clean_text)])
            return extracted_texts

        # Create empty pandas df with keywords as column names
        df = pd.DataFrame(columns=keywords)
        # Extract text
        result = extract_text_using_keywords(text, keywords)
        # Append list of extracted text to the end of the pandas df
        df.loc[len(df)] = result

        #print(df)

        with pd.option_context('display.max_colwidth', None): # For diplaying full columns
          display(df)

错误原因 #

1. 部分文本文件中不存在keywords列表里的某个关键词，调用str.index()方法时找不到子串直接抛出ValueError
2. 循环内重复定义提取函数，效率低下
3. 提取逻辑不严谨：固定偏移2个字符（冒号+空格）适配性差，且处理最后一个关键词时会重复添加内容

修复后的代码 #

import glob
import re
import pandas as pd

def extract_text_using_keywords(clean_text, keyword_list):
    extracted_texts = []
    kw_positions = []
    
    # 收集所有存在的关键词及其位置，不存在的关键词先占位空字符串
    for kw in keyword_list:
        try:
            idx = clean_text.index(kw)
            kw_positions.append((kw, idx))
        except ValueError:
            extracted_texts.append("")
            continue
    
    # 处理相邻关键词间的内容
    for i in range(len(kw_positions)-1):
        prev_kw, prev_idx = kw_positions[i]
        curr_kw, curr_idx = kw_positions[i+1]
        # 截取内容并去除开头的冒号、空格等冗余字符
        content = clean_text[prev_idx + len(prev_kw):curr_idx].strip()
        content = re.sub(r'^[:\s]+', '', content)
        extracted_texts.append(content)
    
    # 处理最后一个关键词后的内容
    if kw_positions:
        last_kw, last_idx = kw_positions[-1]
        content = clean_text[last_idx + len(last_kw):].strip()
        content = re.sub(r'^[:\s]+', '', content)
        # 补全列表长度，确保和关键词列表数量一致
        while len(extracted_texts) < len(keyword_list):
            extracted_texts.append("")
        extracted_texts[-1] = content
    
    # 兜底补全空值，避免长度不匹配
    while len(extracted_texts) < len(keyword_list):
        extracted_texts.append("")
    
    return extracted_texts

# 获取所有目标文件路径
file_paths = [f for f in glob.iglob('/content/sample_data/**/*.txt', recursive=True)]
print('File names: ', file_paths)

# 初始化DataFrame
keywords = ["INDICATION", "TECHNIQUE", "COMPARISON", "FINDINGS", "IMPRESSION"]
df = pd.DataFrame(columns=keywords)

# 遍历处理每个文件
for file in file_paths:
    with open(file, 'r') as f:
        data = f.read()
    
    # 文本清洗
    text = re.sub(r"[-_()\n\"#//@;<>{}=~|?,]*", "", data)
    text = re.sub(r'FINAL REPORT', '', text)
    text = re.sub(r'\s+', ' ', text).strip()
    
    # 提取内容并添加到DataFrame
    result = extract_text_using_keywords(text, keywords)
    df.loc[len(df)] = result

# 显示完整列内容
with pd.option_context('display.max_colwidth', None):
    display(df)

关键改进点 #

• 函数移到循环外，避免重复定义浪费资源
• 用try-except捕获关键词不存在的情况，对应列存入空字符串，防止程序崩溃
• 优化内容截取逻辑：自动去除开头冗余符号，适配不同格式的文本
• 确保返回结果长度始终与关键词列表一致，避免DataFrame赋值时的长度不匹配问题

内容的提问来源于stack exchange，提问作者user10019553