基于长度拼接无重复Unicode文本段的技术实现问询

Let’s walk through how to properly stitch together split Unicode text segments while avoiding duplicates, using segment length as our guide. Here’s a clear, actionable approach based on your pseudocode:

Core Strategy #

The goal is to balance segment lengths by merging shorter-than-average segments with their adjacent neighbors, while ensuring we never reprocess any segments:

• Calculate the average segment length to identify which segments need merging
• For each short segment, compare the length of its previous and next segments—merge it with the shorter neighbor to keep overall lengths balanced
• Track processed segments using a set to avoid duplicates in the final output

Refined Pseudocode #

Here’s the full, filled-out version of your pseudocode with edge-case handling:

# Initialize variables
i = 0
original_text = "your_full_unicode_text_here"
space_separator = " "
total_segments = len(segments_list)
avg_length = len(original_text) / total_segments
new_segments = []
processed = set()  # Track indices of already handled segments

while i < total_segments:
    if i in processed:
        i += 1
        continue
    
    current_segment = segments_list[i]
    current_len = len(current_segment)
    
    # Check if current segment needs merging
    if current_len < avg_length:
        # Get lengths of adjacent segments (handle start/end edge cases)
        prev_len = len(segments_list[i-1]) if i > 0 else float('inf')
        next_len = len(segments_list[i+1]) if i < total_segments - 1 else float('inf')
        
        # Decide which neighbor to merge with (pick the shorter one)
        if prev_len <= next_len and i > 0:
            # Merge current with previous segment
            merged = segments_list[i-1] + space_separator + current_segment
            new_segments.pop()  # Remove the unmerged previous segment
            new_segments.append(merged)
            processed.add(i-1)
            processed.add(i)
        elif next_len < prev_len and i < total_segments - 1:
            # Merge current with next segment
            merged = current_segment + space_separator + segments_list[i+1]
            new_segments.append(merged)
            processed.add(i)
            processed.add(i+1)
            i += 1  # Skip next segment since it's already merged
        else:
            # No valid neighbors (edge case), add current segment as-is
            new_segments.append(current_segment)
            processed.add(i)
    else:
        # Segment is average or longer, add directly
        new_segments.append(current_segment)
        processed.add(i)
    
    i += 1

# Combine all segments into the final text
final_text = space_separator.join(new_segments)

Key Details #

• Unicode Safety: Using len() in Python works for Unicode text (it counts code points, not bytes). If you need byte-length calculations, swap len(current_segment) with len(current_segment.encode('utf-8')).
• Duplicate Prevention: The processed set ensures we never merge or add the same segment twice, which is critical if your input segments have overlaps or duplicates.
• Edge Case Handling: We set neighbor lengths to infinity for segments at the start/end of the list, so we don’t attempt to merge with non-existent segments.

内容的提问来源于stack exchange，提问作者TheSoldier