排查cudf.tokenize()使用非空格分隔符时的「长度不匹配」错误

正常表现（空格作为分隔符） #

调用tokenize(' ')时得到预期结果：

0                     Due
0                      to
0                   being
0                      on
0                    FMLA
               ...       
23956              copies
23956                  of
23956                 the
23956            original
23956    message.\r\n\r\n
Name: description, Length: 3520739, dtype: object

使用分隔符'b'时的错误 #

当使用出现频率低于空格的字符'b'调用tokenize('b')，触发如下报错：

ValueError                                Traceback (most recent call last)
Cell In[6], line 57
     52 desc = cudf.Series.from_pandas(df.description)
     56 desc.str.normalize_characters()
---> 57 desc.str.tokenize('b')
     58 #sr = sr.str.normalize_characters()
     59 #sr.str.tokenize(res)

File ~/base-env/envs/workenvs/envs/workdir/lib/python3.8/site-packages/cudf/core/column/string.py:4592, in StringMethods.tokenize(self, delimiter)
   4587     raise TypeError(
   4588         f"Expected a Scalar or Column\
   4589         for delimiters, but got {type(delimiter)}"
   4590     )
   4591 if isinstance(self._parent, cudf.Series):
-> 4592     result.index = self._parent.index.repeat(  # type: ignore
   4593         self.token_count()
   4594     )
   4595 return result

File ~/base-env/envs/workenvs/envs/workdir/lib/python3.8/site-packages/cudf/core/indexed_frame.py:528, in IndexedFrame.index(self, value)
    526 # A DataFrame with 0 columns can have an index of arbitrary length.
    527 if len(self._data) > 0 and new_length != old_length:
--> 528     raise ValueError(
    529         f"Length mismatch: Expected axis has {old_length} elements, "
    530         f"new values have {len(value)} elements"
    531     )
    532 self._index = Index(value)

ValueError: Length mismatch: Expected axis has 292122 elements, new values have 3520739 elements

使用分隔符'a'时的错误 #

当使用出现频率高于空格的字符'a'调用tokenize('a')，同样触发长度不匹配错误：

ValueError                                Traceback (most recent call last)
Cell In[5], line 57
     52 desc = cudf.Series.from_pandas(df.description)
     56 desc.str.normalize_characters()
---> 57 desc.str.tokenize('a')
     58 #sr = sr.str.normalize_characters()
     59 #sr.str.tokenize(res)

File ~/base-env/envs/workenvs/envs/workdir/lib/python3.8/site-packages/cudf/core/column/string.py:4592, in StringMethods.tokenize(self, delimiter)
   4587     raise TypeError(
   4588         f"Expected a Scalar or Column\
   4589         for delimiters, but got {type(delimiter)}"
   4590     )
   4591 if isinstance(self._parent, cudf.Series):
-> 4592     result.index = self._parent.index.repeat(  # type: ignore
   4593         self.token_count()
   4594     )
   4595 return result

File ~/base-env/envs/workenvs/envs/workdir/lib/python3.8/site-packages/cudf/core/indexed_frame.py:528, in IndexedFrame.index(self, value)
    526 # A DataFrame with 0 columns can have an index of arbitrary length.
    527 if len(self._data) > 0 and new_length != old_length:
--> 528     raise ValueError(
    529         f"Length mismatch: Expected axis has {old_length} elements, "
    530         f"new values have {len(value)} elements"
    531     )
    532 self._index = Index(value)

ValueError: Length mismatch: Expected axis has 1991581 elements, new values have 3520739 elements

问题推测 #

该函数可能基于空格预计算元素大小，导致使用非空格分隔符时出现长度不匹配错误。

内容的提问来源于stack exchange，提问作者Using_System