fast.ai加载数据集触发TypeError：无法比较'L'与'int'实例

问题背景 #

参与农田边界检测竞赛，参考多光谱卫星图像分割的FastAI实现流程，在加载数据集阶段出现上述错误，此前步骤均正常。

相关代码 #

img_pipe = Pipeline([get_filenames, open_ms_tif])
mask_pipe = Pipeline([label_func, partial(open_tif, cls=TensorMask)])

db = DataBlock(blocks=(TransformBlock(img_pipe), 
                       TransformBlock(mask_pipe)),
               splitter=RandomSplitter(valid_pct=0.2, seed=42)
              )

ds = db.datasets(source=train_files)
dl = db.dataloaders(source=train_files, bs=4)

train_files示例（Path列表）：

[Path('nasa_rwanda_field_boundary_competition/nasa_rwanda_field_boundary_competition_source_train/nasa_rwanda_field_boundary_competition_source_train_09_2021_08/B01.tif'),
 Path('nasa_rwanda_field_boundary_competition/nasa_rwanda_field_boundary_competition_source_train/nasa_rwanda_field_boundary_competition_source_train_39_2021_04/B01.tif'),
 Path('nasa_rwanda_field_boundary_competition/nasa_rwanda_field_boundary_competition_source_train/nasa_rwanda_field_boundary_competition_source_train_12_2021_11/B01.tif'),
 Path('nasa_rwanda_field_boundary_competition/nasa_rwanda_field_boundary_competition_source_train/nasa_rwanda_field_boundary_competition_source_train_06_2021_10/B01.tif'),
 Path('nasa_rwanda_field_boundary_competition/nasa_rwanda_field_boundary_competition_source_train/nasa_rwanda_field_boundary_competition_source_train_08_2021_08/B01.tif')]

完整错误堆栈 #

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [66], in <cell line: 10>()
      2 mask_pipe = Pipeline([label_func, partial(open_tif, cls=TensorMask)])
      4 db = DataBlock(blocks=(TransformBlock(img_pipe), 
      5                        TransformBlock(mask_pipe)),
      6                splitter=RandomSplitter(valid_pct=0.2, seed=42)
      7               )
---> 10 ds = db.datasets(source=train_files)
     11 dl = db.dataloaders(source=train_files, bs=4)

File /usr/local/lib/python3.9/dist-packages/fastai/data/block.py:147, in DataBlock.datasets(self, source, verbose)
    145 splits = (self.splitter or RandomSplitter())(items)
    146 pv(f"{len(splits)} datasets of sizes {','.join([str(len(s)) for s in splits])}", verbose)
---> 147 return Datasets(items, tfms=self._combine_type_tfms(), splits=splits, dl_type=self.dl_type, n_inp=self.n_inp, verbose=verbose)

File /usr/local/lib/python3.9/dist-packages/fastai/data/core.py:451, in Datasets.__init__(self, items, tfms, tls, n_inp, dl_type, **kwargs)
    442 def __init__(self, 
    443     items:list=None, # List of items to create `Datasets`
    444     tfms:list|Pipeline=None, # List of `Transform`(s) or `Pipeline` to apply
   (...)
    448     **kwargs
    449 ):
    450     super().__init__(dl_type=dl_type)
---> 451     self.tls = L(tls if tls else [TfmdLists(items, t, **kwargs) for t in L(ifnone(tfms,[None]))])
    452     self.n_inp = ifnone(n_inp, max(1, len(self.tls)-1))

File /usr/local/lib/python3.9/dist-packages/fastai/data/core.py:451, in <listcomp>(.0)
    442 def __init__(self, 
    443     items:list=None, # List of items to create `Datasets`
    444     tfms:list|Pipeline=None, # List of `Transform`(s) or `Pipeline` to apply
   (...)
    448     **kwargs
    449 ):
    450     super().__init__(dl_type=dl_type)
---> 451     self.tls = L(tls if tls else [TfmdLists(items, t, **kwargs) for t in L(ifnone(tfms,[None]))])
    452     self.n_inp = ifnone(n_inp, max(1, len(self.tls)-1))

File /usr/local/lib/python3.9/dist-packages/fastcore/foundation.py:98, in _L_Meta.__call__(cls, x, *args, **kwargs)
     96 def __call__(cls, x=None, *args, **kwargs):
     97     if not args and not kwargs and x is not None and isinstance(x,cls): return x
---> 98     return super().__call__(x, *args, **kwargs)

File /usr/local/lib/python3.9/dist-packages/fastai/data/core.py:361, in TfmdLists.__init__(self, items, tfms, use_list, do_setup, split_idx, train_setup, splits, types, verbose, dl_type)
    359 if isinstance(tfms,TfmdLists): tfms = tfms.tfms
    360 if isinstance(tfms,Pipeline): do_setup=False
---> 361 self.tfms = Pipeline(tfms, split_idx=split_idx)
    362 store_attr('types,split_idx')
    363 if do_setup:

File /usr/local/lib/python3.9/dist-packages/fastcore/transform.py:190, in Pipeline.__init__(self, funcs, split_idx)
    188 else:
    189     if isinstance(funcs, Transform): funcs = [funcs]
---> 190     self.fs = L(ifnone(funcs,[noop])).map(mk_transform).sorted(key='order')
    191 for f in self.fs:
    192     name = camel2snake(type(f).__name__)

File /usr/local/lib/python3.9/dist-packages/fastcore/foundation.py:136, in L.sorted(self, key, reverse)
---> 136 def sorted(self, key=None, reverse=False): return self._new(sorted_ex(self, key=key, reverse=reverse))

File /usr/local/lib/python3.9/dist-packages/fastcore/basics.py:619, in sorted_ex(iterable, key, reverse)
    617 elif isinstance(key,int): k=itemgetter(key)
    618 else: k=key
---> 619 return sorted(iterable, key=k, reverse=reverse)

TypeError: '<' not supported between instances of 'L' and 'int'

解决方案 #

错误根源是TransformBlock的参数传递格式问题：直接将Pipeline实例传给TransformBlock时，FastAI内部会再次尝试将其包装为Pipeline，形成嵌套结构，最终在排序transform的order属性时触发类型不匹配错误。

修正后的代码（两种可行方式） #

# 方式1：将已创建的Pipeline放入列表中传入TransformBlock
img_pipe = Pipeline([get_filenames, open_ms_tif])
mask_pipe = Pipeline([label_func, partial(open_tif, cls=TensorMask)])

db = DataBlock(blocks=(TransformBlock(tfms=[img_pipe]), 
                       TransformBlock(tfms=[mask_pipe])),
               splitter=RandomSplitter(valid_pct=0.2, seed=42)
              )

# 方式2：直接传递transform函数列表给TransformBlock，无需提前创建Pipeline
db = DataBlock(blocks=(TransformBlock(tfms=[get_filenames, open_ms_tif]), 
                       TransformBlock(tfms=[label_func, partial(open_tif, cls=TensorMask)])),
               splitter=RandomSplitter(valid_pct=0.2, seed=42)
              )

ds = db.datasets(source=train_files)
dl = db.dataloaders(source=train_files, bs=4)

额外检查点 #

• 确认get_filenames、open_ms_tif、label_func的返回值符合FastAI要求，比如open_ms_tif需返回模型可接收的张量格式。
• 若get_filenames用于获取同一场景的多波段文件，需确保返回的是L或列表格式的路径，且open_ms_tif能正确处理多波段数据。

内容的提问来源于stack exchange，提问作者Steven Lohrenz