TensorFlow TextVectorization处理UTF-8文本时出现编码错误求助

问题描述 #

执行以下代码尝试用UTF-8编码文本适配TensorFlow的TextVectorization层时：

encoder = tf.keras.layers.TextVectorization(
    max_tokens=VOCAB_SIZE,
    standardize="lower",
)

encoder.adapt(train.map(
    lambda doc, label : doc
))

触发如下编码错误：

UnicodeEncodeError: 'ascii' codec can't encode character '\u2122' in position 49: ordinal not in range(128)

输入文本为UTF-8编码，按预期应与TensorFlow兼容。

环境信息 #

• TensorFlow版本：2.10.0
• Python版本：3.10.8
• 平台：Linux

完整回溯信息 #

InvalidArgumentError                      Traceback (most recent call last)
Cell In [10], line 8
      1 VOCAB_SIZE = 5000
      3 encoder = tf.keras.layers.TextVectorization(
      4     max_tokens=VOCAB_SIZE,
      5     standardize="lower",
      6 )
----> 8 encoder.adapt(train.map(
      9     lambda doc, label : doc
     10 ))

File ~/.local/lib/python3.10/site-packages/keras/layers/preprocessing/text_vectorization.py:467, in TextVectorization.adapt(self, data, batch_size, steps)
    417 def adapt(self, data, batch_size=None, steps=None):
    418     """Computes a vocabulary of string terms from tokens in a dataset.
    419 
    420     Calling `adapt()` on a `TextVectorization` layer is an alternative to
   (...)
    465           argument is not supported with array inputs.
    466     """
--> 467     super().adapt(data, batch_size=batch_size, steps=steps)

File ~/.local/lib/python3.10/site-packages/keras/engine/base_preprocessing_layer.py:258, in PreprocessingLayer.adapt(self, data, batch_size, steps)
    256 with data_handler.catch_stop_iteration():
    257     for _ in data_handler.steps():
--> 258         self._adapt_function(iterator)
    259         if data_handler.should_sync:
    260             context.async_wait()

File ~/.local/lib/python3.10/site-packages/tensorflow/python/util/traceback_utils.py:153, in filter_traceback.<locals>.error_handler(*args, **kwargs)
    151 except Exception as e:
    152   filtered_tb = _process_traceback_frames(e.__traceback__)
--> 153   raise e.with_traceback(filtered_tb) from None
    154 finally:
    155   del filtered_tb

File ~/.local/lib/python3.10/site-packages/tensorflow/python/eager/execute.py:54, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     52 try:
     53   ctx.ensure_initialized()
--> 54   tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     55                                       inputs, attrs, num_outputs)
     56 except core._NotOkStatusException as e:
     57   if name is not None:

InvalidArgumentError: Graph execution error:

2 root error(s) found.
  (0) INVALID_ARGUMENT:  UnicodeEncodeError: 'ascii' codec can't encode character '\xae' in position 401: ordinal not in range(128)
Traceback (most recent call last):

  File "/home/moss/.local/lib/python3.10/site-packages/tensorflow/python/ops/script_ops.py", line 279, in __call__
    return [self._convert(x) for x in ret]

  File "/home/moss/.local/lib/python3.10/site-packages/tensorflow/python/ops/script_ops.py", line 279, in <listcomp>
    return [self._convert(x) for x in ret]

  File "/home/moss/.local/lib/python3.10/site-packages/tensorflow/python/ops/script_ops.py", line 237, in _convert
    return result.astype(np.bytes_)

UnicodeEncodeError: 'ascii' codec can't encode character '\xae' in position 401: ordinal not in range(128)


     [[{{node PyFunc}}]]
     [[IteratorGetNext]]
     [[UniqueWithCounts/_6]]
  (1) INVALID_ARGUMENT:  UnicodeEncodeError: 'ascii' codec can't encode character '\xae' in position 401: ordinal not in range(128)
Traceback (most recent call last):

  File "/home/moss/.local/lib/python3.10/site-packages/tensorflow/python/ops/script_ops.py", line 279, in __call__
    return [self._convert(x) for x in ret]

  File "/home/moss/.local/lib/python3.10/site-packages/tensorflow/python/ops/script_ops.py", line 279, in <listcomp>
    return [self._convert(x) for x in ret]

  File "/home/moss/.local/lib/python3.10/site-packages/tensorflow/python/ops/script_ops.py", line 237, in _convert
    return result.astype(np.bytes_)

UnicodeEncodeError: 'ascii' codec can't encode character '\xae' in position 401: ordinal not in range(128)


     [[{{node PyFunc}}]]
     [[IteratorGetNext]]
0 successful operations.
0 derived errors ignored. [Op:__inference_adapt_step_195]

解决方法 #

该问题源于TensorFlow处理字符串时，系统默认编码被设置为ASCII，可通过以下方式解决：

1. 强制设置UTF-8环境编码 #

在运行脚本前设置环境变量：

export PYTHONIOENCODING=utf-8
export LC_ALL=C.UTF-8
export LANG=C.UTF-8

或者在Python代码开头添加：

import os
os.environ['PYTHONIOENCODING'] = 'utf-8'
os.environ['LC_ALL'] = 'C.UTF-8'
os.environ['LANG'] = 'C.UTF-8'

2. 自定义标准化函数过滤特殊字符 #

如果不想修改环境变量，可自定义标准化逻辑，过滤或替换无法用ASCII编码的特殊字符：

import tensorflow as tf

def custom_standardize(input_data):
    # 转换为小写
    lowercase = tf.strings.lower(input_data)
    # 保留字母、数字、空格及常见标点，移除其他特殊Unicode字符
    cleaned = tf.strings.regex_replace(lowercase, r'[^a-zA-Z0-9\s.,!?]', '')
    return cleaned

encoder = tf.keras.layers.TextVectorization(
    max_tokens=VOCAB_SIZE,
    standardize=custom_standardize,
)

encoder.adapt(train.map(lambda doc, label: doc))

3. 升级TensorFlow版本 #

该编码问题在TensorFlow 2.11及以上版本中已被修复，直接升级到最新稳定版即可解决：

pip install --upgrade tensorflow

内容的提问来源于stack exchange，提问作者Moss Richardson