文件存在时rasterio报invalid path or file及权限拒绝错误

• 编写HuBMAP数据集加载代码时，使用pandas读取sample_submission.csv样本文件，通过glob递归匹配test_images目录下所有.tiff格式影像路径，自定义HuBMAPDataset类在初始化方法中调用rasterio.open读取影像。运行代码时，即使目标tiff文件真实存在，程序仍抛出TypeError: invalid path or file错误
• 使用root用户身份直接在终端执行对应tiff文件路径，返回Permission denied权限拒绝提示

import rasterio

sample = pd.read_csv(os.path.join(config.BASE_PATH, "sample_submission.csv"))
test_images = glob.glob(os.path.join(config.BASE_PATH + "test_images", "**", "*.tiff"), recursive=True)

class HuBMAPDataset:
    def __init__(self, idx, sz=sz, reduce=reduce):
        self.data = rasterio.open(test_images, transform = identity, num_threads='all_cpus')    

for idx,row in tqdm(sample.iterrows(),total=len(sample)):
    idx = str(row['id'])
    ds = HuBMAPDataset(idx)

Python运行报错栈 #

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [89], in <cell line: 2>()
      2 for idx,row in tqdm(sample.iterrows(),total=len(sample)):
      3     idx = str(row['id'])
----> 4     ds = HuBMAPDataset(idx)
      5     #rasterio cannot be used with multiple workers
      6     dl = DataLoader(ds,bs,num_workers=0,shuffle=False,pin_memory=True)

Input In [85], in HuBMAPDataset.__init__(self, idx, sz, reduce)
     14 def __init__(self, idx, sz=sz, reduce=reduce):
     15     #self.data = rasterio.open(os.path.join(config.BASE_PATH, test_images,idx+'.tiff'), transform = identity, num_threads='all_cpus')      
---&gt; 16     self.data = rasterio.open(test_images, transform = identity, num_threads='all_cpus')     
     18     # some images have issues with their format 
     19     # and must be saved correctly before reading with rasterio
     20     if self.data.count != 3:

File ~/anaconda3/lib/python3.9/site-packages/rasterio/env.py:442, in ensure_env_with_credentials.<locals>.wrapper(*args, **kwds)
    439     session = DummySession()
    441 with env_ctor(session=session):
--&gt; 442     return f(*args, **kwds)

File ~/anaconda3/lib/python3.9/site-packages/rasterio/__init__.py:189, in open(fp, mode, driver, width, height, count, crs, transform, dtype, nodata, sharing, **kwargs)
    183 if not isinstance(fp, str):
    184     if not (
    185         hasattr(fp, "read")
    186         or hasattr(fp, "write")
    187         or isinstance(fp, (os.PathLike, MemoryFile, FilePath))
    188     ):
--&gt; 189         raise TypeError("invalid path or file: {0!r}".format(fp))
    190 if mode and not isinstance(mode, str):
    191     raise TypeError("invalid mode: {0!r}".format(mode))

TypeError: invalid path or file: ['./input/hubmap-organ-segmentation/test_images/10078.tiff']

终端执行返回结果 #

!./input/hubmap-organ-segmentation/test_images/10078.tiff
/bin/bash: ./input/hubmap-organ-segmentation/test_images/10078.tiff: Permission denied

1. 传参类型错误：glob.glob()返回值是所有匹配路径组成的列表，rasterio.open()仅接受单个字符串路径、类路径对象或可读写的文件句柄，不支持传入列表。从报错信息可以看到传入的参数带方括号['./input/xxx.tiff']，是典型的列表格式，直接触发类型错误。
2. 路径拼接错误：glob匹配时写的config.BASE_PATH + "test_images"没有加路径分隔符，若config.BASE_PATH末尾不带/，会拼成错误的连字路径。
3. 终端操作逻辑错误：直接在终端输入tiff文件路径执行，本质是尝试把tiff二进制影像当成可执行脚本运行，tiff本身没有可执行权限，无论用什么用户身份执行都会报Permission denied，和文件读权限无关。

• 修正路径拼接逻辑，统一用os.path.join()处理路径拼接，避免手动拼接出现分隔符缺失问题
• 提前把glob匹配到的所有tiff路径构建为影像id: 完整路径的字典，Dataset初始化时根据传入的id取对应的单个字符串路径传给rasterio.open()
• 验证tiff文件是否可访问时，不要直接执行文件路径，改用ls -l 目标文件路径查看权限、file 目标文件路径确认文件格式即可。

修复后的可运行代码：

import os
import glob
import rasterio
import pandas as pd
from tqdm import tqdm
from torch.utils.data import DataLoader

# 读取样本csv
sample = pd.read_csv(os.path.join(config.BASE_PATH, "sample_submission.csv"))
# 修正路径拼接，递归匹配所有tiff文件
all_tiff_paths = glob.glob(
    os.path.join(config.BASE_PATH, "test_images", "**", "*.tiff"),
    recursive=True
)
# 构建id到文件路径的映射
id2path = {
    os.path.splitext(os.path.basename(p))[0]: p 
    for p in all_tiff_paths
}

class HuBMAPDataset:
    def __init__(self, idx, sz=sz, reduce=reduce):
        # 传入对应id的单个路径
        self.data = rasterio.open(
            id2path[idx],
            transform=identity,
            num_threads='all_cpus'
        )
        # 原有通道数异常处理逻辑保留
        if self.data.count != 3:
            pass

for idx, row in tqdm(sample.iterrows(), total=len(sample)):
    img_id = str(row['id'])
    ds = HuBMAPDataset(img_id)
    # rasterio不支持多worker加载，num_workers固定为0
    dl = DataLoader(ds, bs, num_workers=0, shuffle=False, pin_memory=True)

内容的提问来源于stack exchange，提问作者melolilili