使用datasets库的load_dataset()加载WikiSQL数据集时出现PermissionError权限错误

Problem Context #

I tried loading the WikiSQL validation set using this code snippet:

from datasets import load_dataset
import random, warnings
warnings.filterwarnings("ignore")
valid_dataset = load_dataset('wikisql', split='validation')
valid_dataset[0]

But it threw a permission error:

PermissionError: [WinError 5] Access is denied: 'C:\Users\XXXXXX\.cache\huggingface\datasets\wiki_sql\default\0.1.0\2e98053891fd8f9b2c4348bba609ce40cc0a4d7f621191cebcd7cb558b5f8a70.incomplete' -> 'C:\Users\XXXXXX\.cache\huggingface\datasets\wiki_sql\default\0.1.0\2e98053891fd8f9b2c4348bba609ce40cc0a4d7f621191cebcd7cb558b5f8a70'

I'm not an admin, and this error only happens with some datasets. I suspected a C盘 permission issue with the default cache directory, so I tried specifying a custom cache path:

dataset = load_dataset('wikisql', split='validation', cache_dir="PATH/TO/MY/CACHE/DIR")

But the error still pointed to the original C盘 path. I'm using datasets version 1.4.1, and need a working solution.

Workable Solutions #

Here are targeted fixes sorted by practicality:

1. Set a Permanent Cache Directory via Environment Variables #

Older datasets versions (like 1.4.1) sometimes have bugs with the cache_dir parameter. A more reliable fix is to set a global cache path via environment variables:

• Windows Steps:
  1. Right-click "This PC" → Properties → Advanced System Settings → Environment Variables
  2. Under "User Variables", click "New":
     • Variable name: HUGGINGFACE_HUB_CACHE
     • Variable value: Path to a folder you have read/write access to (e.g., D:\huggingface_cache)
  3. Restart your Python editor/terminal to apply the changes
• This will make all future load_dataset calls use your custom cache path automatically, no need to pass cache_dir every time.

2. Clean Up Incomplete Cache Files #

The .incomplete suffix in the error means a previous download was interrupted, leaving locked or permission-residual files. Do this:

• Navigate to the original cache path: C:\Users\XXXXXX\.cache\huggingface\datasets\wiki_sql\default\0.1.0\
• Delete all files with the .incomplete extension, plus the corresponding folder
• Retry loading the dataset with your custom cache_dir parameter

3. Upgrade the Datasets Library #

Version 1.4.1 is quite outdated (current stable versions are in the 2.x range), and the cache_dir handling bug might have been fixed in newer releases. Upgrade via pip:

pip install --upgrade datasets

After upgrading, try specifying cache_dir again—it should correctly point to your desired path.

4. Manually Download the Dataset #

If all else fails, manually fetch the dataset files:

1. Locate the WikiSQL dataset files on the Hugging Face Hub
2. Download the validation set files and place them in your custom cache directory, matching the folder structure of the original cache path
3. Call load_dataset with your cache_dir parameter—it will read the local files instead of trying to download to the C盘 path

Verification Test #

After applying any fix, run this test code to confirm it works:

from datasets import load_dataset
import warnings
warnings.filterwarnings("ignore")
# Replace with your actual cache path
valid_dataset = load_dataset('wikisql', split='validation', cache_dir="D:/huggingface_cache")
print(valid_dataset[0])

If it outputs the dataset sample without errors, the issue is resolved.

内容的提问来源于stack exchange，提问作者Gihan Gamage