基于上下文的长文本段落智能拆分技术需求及代码优化问询

我现在用的是固定每250词拆分长文本的方法，只是按句子拆分后凑够词数，属于无上下文的机械拆分，会出现截断关联信息、拆分结果并非真正语义段落的问题。现在需要一种基于上下文的智能段落拆分方案，以下是我当前的实现代码：

import re

newtext = '''
Kendrick Lamar Duckworth is an American rapper, songwriter, and record producer. He is often cited as one of the most influential rappers of his generation. Aside from his solo career, he is also a member of the hip hop supergroup Black Hippy alongside his former Top Dawg Entertainment (TDE) labelmates Ab-Soul, Jay Rock, and Schoolboy Q. Raised in Compton, California, Lamar embarked on his musical career as a teenager under the stage name K.Dot, releasing a mixtape titled Y.H.N.I.C. (Hub City Threat Minor of the Year) that garnered local attention and led to his signing with indie record label TDE. He began to gain recognition in 2010 after his first retail release, Overly Dedicated. The following year, he independently released his first studio album, Section.80, which included his debut single "HiiiPoWeR". By that time, he had amassed a large online following and collaborated with several prominent rappers. He subsequently secured a record deal with Dr. Dre's Aftermath Entertainment, under the aegis of Interscope Records. Lamar's major-label debut album, Good Kid, M.A.A.D City, was released in 2012, garnering him widespread critical recognition and mainstream success. His third album To Pimp a Butterfly (2015), which incorporated elements of funk, soul, jazz, and spoken word, predominantly centred around the Black-American experience. It became his first number-one album on the US Billboard 200 and was an enormous critical success. His fourth album, Damn (2017), saw continued acclaim, becoming the first non-classical and non-jazz album to be awarded the Pulitzer Prize for Music. It also yielded his first number-one single, "Humble", on the US Billboard Hot 100. Lamar curated the soundtrack to the superhero film Black Panther (2018) and in 2022, released his fifth and last album with TDE, Mr. Morale & the Big Steppers, which received critical acclaim. Lamar has certified sales of over 70 million records in the United States alone, and all of his albums have been certified platinum or higher by the Recording Industry Association of America (RIAA). He has received several accolades in his career, including 14 Grammy Awards, two American Music Awards, six Billboard Music Awards, 11 MTV Video Music Awards, a Pulitzer Prize, a Brit Award, and an Academy Award nomination. In 2012, MTV named him the Hottest MC in the Game on their annual list. Time named him one of the 100 most influential people in the world in 2016. In 2015, he received the California State Senate's Generational Icon Award. Three of his studio albums were included on Rolling Stone's 2020 list of the 500 Greatest Albums of All Time. Kendrick Lamar Duckworth was born in Compton, California on June 17, 1987, the son of a couple from Chicago. Although not in a gang himself, he grew up around gang members, with his closest friends being Westside Piru Bloods and his father, Kenny Duckworth, being a Gangster Disciple. His first name was given to him by his mother in honor of singer-songwriter Eddie Kendricks of The Temptations. He grew up on welfare and in Section 8 housing. In 1995, at the age of eight, Lamar witnessed his idols Tupac Shakur and Dr. Dre filming the music video for their hit single "California Love", which proved to be a significant moment in his life. As a child, Lamar attended McNair Elementary and Vanguard Learning Center in the Compton Unified School District. He has admitted to being quiet and shy in school, his mother even confirming he was a "loner" until the age of seven. As a teenager, he graduated from Centennial High School in Compton, where he was a straight-A student. Kendrick Lamar has stated that Tupac Shakur, the Notorious B.I.G., Jay-Z, Nas and Eminem are his top five favorite rappers. Tupac Shakur is his biggest influence, and has influenced his music as well as his day-to-day lifestyle. In a 2011 interview with Rolling Stone, Lamar mentioned Mos Def and Snoop Dogg as rappers that he listened to and took influence from during his early years. He also cites now late rapper DMX as an influence: "[DMX] really [got me started] on music," explained Lamar in an interview with Philadelphia's Power 99. "That first album [It's Dark and Hell Is Hot] is classic, [so he had an influence on me]." He has also stated Eazy-E as an influence in a post by Complex saying: "I Wouldn't Be Here Today If It Wasn't for Eazy-E." In a September 2012 interview, Lamar stated rapper Eminem "influenced a lot of my style" and has since credited Eminem for his own aggression, on records such as "Backseat Freestyle". Lamar also gave Lil Wayne's work in Hot Boys credit for influencing his style and praised his longevity. He has said that he also grew up listening to Rakim, Dr. Dre, and Tha Dogg Pound. In January 2013, when asked to name three rappers that have played a role in his style, Lamar said: "It's probably more of a west coast influence. A little bit of Kurupt, [Tupac], with some of the content of Ice Cube." In a November 2013 interview with GQ, when asked "The Four MC's That Made Kendrick Lamar?", he answered Tupac Shakur, Dr. Dre, Snoop Dogg and Mobb Deep, namely Prodigy. Lamar professed to having been influenced by jazz trumpeter Miles Davis and Parliament-Funkadelic during the recording of To Pimp a Butterfly. Lamar has been branded as the "new king of hip hop" numerous times. Forbes said, on Lamar's placement as hip hop's "king", "Kendrick Lamar may or may not be the greatest rapper alive right now. He is certainly in the very short lists of artists in the conversation." Lamar frequently refers to himself as the "greatest rapper alive" and once called himself "The King of New York." On the topic of his music genre, Lamar has said: "You really can't categorize my music, it's human music." Lamar's projects are usually concept albums. Critics found Good Kid, M.A.A.D City heavily influenced by West Coast hip hop and 90s gangsta rap. His third studio album, To Pimp a Butterfly, incorporates elements of funk, jazz, soul and spoken word poetry. Called a "radio-friendly but overtly political rapper" by Pitchfork, Lamar has been a branded "master of storytelling" and his lyrics have been described as "katana-blade sharp" and his flow limber and dexterous. Lamar's writing usually includes references to racism, black empowerment and social injustice, being compared to a State of Union address by The Guardian. His writing has also been called "confessional" and controversial. The New York Times has called Lamar's musical style anti-flamboyant, interior and complex and labelled him as a technical rapper. Billboard described his lyricism as "Shakespearean".    
'''
# 约1100词

regex = r'([A-z][^.!?]*[.!?]*"?)'
for sens in re.findall(regex, newtext):
    newtext = newtext.replace(f'{sens}', f'{sens}<eos>')
    sentencess = newtext.split('<eos>')
sentences = sentencess

current_chunk = 0
chunks = []

for sentence in sentences:
    if len(chunks) == current_chunk + 1:
        if len(chunks[current_chunk]) + len(sentence.split(" ")) <= 250:
            chunks[current_chunk].extend(sentence.split(" "))
        else:
            current_chunk += 1
            chunks.append(sentence.split(" "))
    else:
        chunks.append(sentence.split(" "))
for chunk_id in range(len(chunks)):
    chunks[chunk_id] = " ".join(chunks[chunk_id])
print(chunks) # 打印所有拆分后的"段落"
print(chunks[0]) # 打印第一个"段落"

智能拆分方案 #

核心思路 #

放弃只凑词数的机械拆分，改为语义优先+词数限制的逻辑：

1. 先计算相邻句子的语义相似度，关联度高的句子归为同一个段落
2. 当段落词数接近250上限时，找最近的语义停顿点拆分，不硬砍语义块

具体实现（带语义判断） #

下面的代码用预训练模型计算句子语义相似度，兼顾语义连贯和词数限制：

import re
from sentence_transformers import SentenceTransformer, util

def smart_split_text(text, max_words=250):
    # 第一步：精准拆分句子
    regex = r'([A-Z][^.!?]*[.!?]")|([A-Z][^.!?]*[.!?])'
    # 提取有效句子并去除空内容
    raw_sentences = re.findall(regex, text)
    sentences = []
    for s in raw_sentences:
        if s[0]:
            sentences.append(s[0].strip())
        elif s[1]:
            sentences.append(s[1].strip())
    
    # 加载轻量级语义模型，计算句子相似度
    model = SentenceTransformer('all-MiniLM-L6-v2')
    embeddings = model.encode(sentences, convert_to_tensor=True)
    
    chunks = []
    # 初始化第一个段落
    current_chunk = [sentences[0]]
    current_word_count = len(sentences[0].split())
    
    for i in range(1, len(sentences)):
        # 计算当前句子与段落最后一句的语义相似度
        similarity = util.cos_sim(embeddings[i-1], embeddings[i]).item()
        next_word_count = len(sentences[i].split())
        
        # 两个判断条件：词数不超上限，且语义相似度足够高（0.5为经验阈值，可调整）
        if current_word_count + next_word_count <= max_words and similarity > 0.5:
            current_chunk.append(sentences[i])
            current_word_count += next_word_count
        else:
            # 保存当前段落，开启新段落
            chunks.append(" ".join(current_chunk))
            current_chunk = [sentences[i]]
            current_word_count = next_word_count
    
    # 加入最后一个段落
    if current_chunk:
        chunks.append(" ".join(current_chunk))
    
    return chunks

# 测试拆分效果
split_result = smart_split_text(newtext)
for idx, para in enumerate(split_result):
    print(f"段落 {idx+1}:\n{para}\n")

方案优势 #

• 不会把讲同一主题的句子拆到不同段落，避免信息截断
• 词数限制依然生效，拆分结果符合长度要求
• 模型轻量，运行速度快，适合日常文本处理

内容的提问来源于stack exchange，提问作者Zeke John