字符串分块分割优化：如何确保分块末尾为完整单词？

Your existing chunking function works perfectly for fixed-length splits, but as you’ve noticed, it often cuts words off mid-sentence. Let’s adjust it to make sure every chunk ends with a complete word—here’s a practical solution:

The Improved Function #

public static IList<string> SplitChunksWithCompleteWords(string text, int chunkSize)
{
    List<string> chunks = new List<string>();
    int offset = 0;
    int textLength = text.Length;

    while (offset < textLength)
    {
        // Start with the maximum allowed chunk size
        int endIndex = Math.Min(offset + chunkSize, textLength);

        // If we're not at the end of the text, backtrack to find a natural word boundary
        if (endIndex < textLength)
        {
            // Look for common word separators—add more (like ';', ':') if needed
            int lastSeparatorIndex = text.LastIndexOfAny(new[] { ' ', '.', ',', '!', '?' }, endIndex - 1);

            // Only adjust the end index if we found a separator after the current offset
            if (lastSeparatorIndex > offset)
            {
                endIndex = lastSeparatorIndex + 1; // Include the separator in the chunk for readability
            }
            // If no separator exists (e.g., a single word longer than chunkSize), stick to original end index
        }

        // Add the finalized chunk and move the offset forward
        chunks.Add(text.Substring(offset, endIndex - offset));
        offset = endIndex;
    }

    return chunks;
}

Key Changes Explained #

• Word Boundary Detection: Instead of strictly cutting at offset + chunkSize, we check backwards from that point to find the nearest natural separator (space, punctuation). This ensures we don’t split words.
• Edge Case Handling: If a single word is longer than chunkSize (e.g., a technical term like "electroencephalography"), we still split it at the fixed length to avoid infinite loops and ensure all text gets processed.
• Clean Chunk Ends: We include the separator in the chunk (hence lastSeparatorIndex + 1) so chunks end with natural punctuation or spaces, rather than mid-word fragments.

Example Test Run #

Let’s use your sample input to see it in action:

string input = "Hello Everyone. How are you doing today? I hope this helps!";
var chunks = SplitChunksWithCompleteWords(input, 15);

// Resulting chunks:
// 1. "Hello Everyone. "
// 2. "How are you doing "
// 3. "today? I hope this "
// 4. "helps!"

Now every chunk wraps neatly at complete words or sentence endings, no more broken fragments!

内容的提问来源于stack exchange，提问作者Ali