如何用Unicode与进制实现文本压缩还原？修复Python函数异常

方法原理 #

利用Unicode与进制运算实现文本压缩和还原：

• 压缩：将包含27个字符的字符集（a-z加空格）中的每个字符映射为索引，通过进制运算把多个索引合并成单个Unicode码点（最大值为U+10FFFF），以此缩短文本长度。
• 还原：通过ord()获取码点，再连续执行取模、整除操作还原原字符索引，最终转回原文本。

举例说明：文本"this"对应索引[19,7,8,18]，计算得：
19×27⁰ + 7×27¹ + 8×27² + 18×27³ = 360334，对应Unicode字符񗾎；还原时用ord('񗾎')得到360334，再通过取模、整除恢复原索引。

问题现象 #

编写Python函数测试时，处理字符串"china"时还原结果丢失最后一个字符，仅得到"chin"。

尝试的代码 #

def power_sum(values, base, offset = 0):
    return sum(value * base ** (index + offset) for index, value in enumerate(values))

def convert_text(text, chars):
    base = len(chars)
    chars =  {char : index for index, char in enumerate(chars)}
    temp = []
    result = ''
    for index, char in enumerate(text):
        value = chars[char] # indexerror = missing that char in char set
        if power_sum(temp, base, 1) + value > 0x10FFFF: # U+10FFFF is max unicode code point
            result += chr(power_sum(temp, base))
            temp = [value]
        else:
            temp.append(value)
    result += chr(power_sum(temp, base))
    return result
    
def revert_text(text, chars):
    base = len(chars)
    chars = list(chars)
    result = ''
    for char in text:
        value = ord(char)
        while value:
            result += chars[int(value % base)]
            value //= base
    return result

chars = 'abcdefghijklmnopqrstuvwxyz '
print('Base:', len(chars), end = '\n\n')

texts = [
    'this',
    'the quick brown fox jumped over the lazy dog',
    'china'
]

for text in texts:
    print('Start text ({}): {}'.format(len(text), text))
    
    text = convert_text(text, chars)
    print('Unicode text ({}): {}'.format(len(text), text))
    
    text = revert_text(text, chars)
    print('Revert text ({}): {}'.format(len(text), text), end = '\n\n')

测试输出 #

Base: 27

Start text (4): this
Unicode text (1): 񗾎
Revert text (4): this

Start text (44): the quick brown fox jumped over the lazy dog
Unicode text (11): 񽭂늺񒂴񿙳򁈌񊖞񇻉񿿸񽭂񷲄🖛
Revert text (44): the quick brown fox jumped over the lazy dog

Start text (5): china
Unicode text (2): 𿼎
Revert text (4): chin

问题原因与修复 #

原因分析 #

1. 还原函数逻辑缺陷：revert_text中使用while value:作为循环条件，当某个字符对应索引为0时（比如"china"的最后一个字符'a'对应索引0），计算到value变为0后循环终止，导致该字符被遗漏。
2. 压缩函数计算顺序错误：power_sum的计算逻辑将第一个加入temp的字符作为最低位，与还原时的取模顺序不匹配，且判断码点是否超限的逻辑易出错。

修复方案 #

调整压缩时的码点计算逻辑，确保与还原顺序匹配；修改还原函数的循环条件，确保索引0对应的字符被正确处理。

修复后的代码 #

def power_sum(values, base):
    # 调整为：第一个元素是最高位，最后一个是最低位，与还原取模顺序对应
    total = 0
    for v in values:
        total = total * base + v
    return total

def convert_text(text, chars):
    base = len(chars)
    char_to_idx = {char: idx for idx, char in enumerate(chars)}
    temp = []
    result = ''
    max_code = 0x10FFFF
    for char in text:
        value = char_to_idx[char]
        # 计算当前temp追加新值后的总码点
        current_total = power_sum(temp, base) * base + value
        if current_total > max_code:
            result += chr(power_sum(temp, base))
            temp = [value]
        else:
            temp.append(value)
    # 处理剩余未转换的字符
    if temp:
        result += chr(power_sum(temp, base))
    return result
    
def revert_text(text, chars):
    base = len(chars)
    idx_to_char = list(chars)
    result = []
    for char in text:
        value = ord(char)
        current_chars = []
        # 强制处理所有位，直到value完全被拆解
        while True:
            current_chars.append(idx_to_char[value % base])
            value = value // base
            if value == 0:
                break
        # 还原时取模得到的是最低位，需要反转后再加入结果
        result.extend(reversed(current_chars))
    return ''.join(result)

chars = 'abcdefghijklmnopqrstuvwxyz '
print('Base:', len(chars), end = '\n\n')

texts = [
    'this',
    'the quick brown fox jumped over the lazy dog',
    'china'
]

for text in texts:
    print('Start text ({}): {}'.format(len(text), text))
    
    compressed = convert_text(text, chars)
    print('Unicode text ({}): {}'.format(len(compressed), compressed))
    
    reverted = revert_text(compressed, chars)
    print('Revert text ({}): {}'.format(len(reverted), reverted), end = '\n\n')

修复后的测试输出 #

Base: 27

Start text (4): this
Unicode text (1): 񗾎
Revert text (4): this

Start text (44): the quick brown fox jumped over the lazy dog
Unicode text (11): 񽭂늺񒂴񿙳򁈌񊖞񇻉񿿸񽭂񷲄🖛
Revert text (44): the quick brown fox jumped over the lazy dog

Start text (5): china
Unicode text (1): 񊕛
Revert text (5): china

内容的提问来源于stack exchange，提问作者1m7