虚假新闻检测训练报错:已建logs目录仍触发FailedPreconditionError
虚假新闻检测训练报错:
./logs is not a directory 问题排查 问题描述
开发虚假新闻检测项目,完成数据预处理后执行模型训练时出现如下错误,且已在项目目录下创建logs文件夹但问题仍存在:
Traceback (most recent call last): File "c:\Users\asus\Masaüstü\BitirmeProjesi\training.py", line 123, in trainer.train() File "C:\Users\asus\AppData\Local\Programs\Python\Python311\Lib\site-packages\transformers\trainer.py", line 1591, in train return inner_training_loop( ^^^^^^^^^^^^^^^^^^^^ File "C:\Users\asus\AppData\Local\Programs\Python\Python311\Lib\site-packages\transformers\trainer.py", line 1826, in _inner_training_loop self.control = self.callback_handler.on_train_begin(args, self.state, self.control) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\asus\AppData\Local\Programs\Python\Python311\Lib\site-packages\transformers\trainer_callback.py", line 362, in on_train_begin return self.call_event("on_train_begin", args, state, control) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\asus\AppData\Local\Programs\Python\Python311\Lib\site-packages\transformers\trainer_callback.py", line 406, in call_event result = getattr(callback, event)( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\asus\AppData\Local\Programs\Python\Python311\Lib\site-packages\transformers\integrations\integration_utils.py", line 628, in on_train_begin self._init_summary_writer(args, log_dir) File "C:\Users\asus\AppData\Local\Programs\Python\Python311\Lib\site-packages\transformers\integrations\integration_utils.py", line 614, in _init_summary_writer self.tb_writer = self._SummaryWriter(log_dir=log_dir) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\asus\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\utils\tensorboard\writer.py", line 243, in __init__ self._get_file_writer() File "C:\Users\asus\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\utils\tensorboard\writer.py", line 273, in _get_file_writer self.file_writer = FileWriter( ^^^^^^^^^^^ File "C:\Users\asus\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\utils\tensorboard\writer.py", line 72, in __init__ self.event_writer = EventFileWriter( ^^^^^^^^^^^^^^^^ File "C:\Users\asus\AppData\Local\Programs\Python\Python311\Lib\site-packages\tensorboard\summary\writer\event_file_writer.py", line 72, in __init__ tf.io.gfile.makedirs(logdir) File "C:\Users\asus\AppData\Local\Programs\Python\Python311\Lib\site-packages\tensorflow\python\lib\io\file_io.py", line 513, in recursive_create_dir_v2 _pywrap_file_io.RecursivelyCreateDir(compat.path_to_bytes(path)) tensorflow.python.framework.errors_impl.FailedPreconditionError: ./logs is not a directory
训练代码
import torch import pandas as pd from transformers.file_utils import is_tf_available, is_torch_available, is_torch_tpu_available from transformers import DistilBertForSequenceClassification, DistilBertTokenizerFast from transformers import Trainer, TrainingArguments import numpy as np from sklearn.model_selection import train_test_split from sklearn.metrics import accuracy_score import random news_d=pd.read_csv('C:\\Users\\asus\\Masaüstü\\BitirmeProjesi\\preprocessed_data.csv', encoding='utf-8') def set_seed(seed: int): random.seed(seed) np.random.seed(seed) if is_torch_available(): torch.manual_seed(seed) torch.cuda.manual_seed_all(seed) # ^^ safe to call this function even if cuda is not available if is_tf_available(): import tensorflow as tf tf.random.set_seed(seed) set_seed(1) model_name = "distilbert-base-uncased" # max sequence length for each document/sentence sample max_length = 300 # load the tokenizer tokenizer = DistilBertTokenizerFast.from_pretrained(model_name, do_lower_case=True) news_df = news_d[news_d['text'].notna()] news_df = news_df[news_df["title"].notna()] def prepare_data(df, test_size=0.2, include_title=True): texts = [] labels = [] for i in range(len(df)): text = df["text"].iloc[i] label = df["label"].iloc[i] if include_title: text = df["title"].iloc[i] + " - " + text if text and label in [0, 1]: texts.append(text) labels.append(label) return train_test_split(texts, labels, test_size=test_size) train_texts, valid_texts, train_labels, valid_labels = prepare_data(news_df) print(len(train_texts), len(train_labels)) print(len(valid_texts), len(valid_labels)) # tokenize the dataset, truncate when passed `max_length`, # and pad with 0's when less than `max_length` train_encodings = tokenizer(train_texts, truncation=True, padding=True, max_length=max_length) valid_encodings = tokenizer(valid_texts, truncation=True, padding=True, max_length=max_length) class NewsGroupsDataset(torch.utils.data.Dataset): def __init__(self, encodings, labels): self.encodings = encodings self.labels = labels def __getitem__(self, idx): item = {k: torch.tensor(v[idx]) for k, v in self.encodings.items()} item["labels"] = torch.nn.functional.one_hot(torch.tensor([self.labels[idx]]), num_classes=2).squeeze(0) return item def __len__(self): return len(self.labels) # convert our tokenized data into a torch Dataset train_dataset = NewsGroupsDataset(train_encodings, train_labels) valid_dataset = NewsGroupsDataset(valid_encodings, valid_labels) # load the model model = DistilBertForSequenceClassification.from_pretrained(model_name, num_labels=2) def compute_metrics(pred): labels = pred.label_ids preds = pred.predictions.argmax(-1) # calculate accuracy using sklearn's function acc = accuracy_score(labels, preds) return { 'accuracy': acc, } # Training configuration training_args = TrainingArguments( output_dir='./results', num_train_epochs=1, per_device_train_batch_size=10, per_device_eval_batch_size=20, warmup_steps=100, logging_dir='./logs', load_best_model_at_end=False, logging_steps=200, save_steps=200, evaluation_strategy="steps", ) trainer = Trainer( model=model, # the instantiated Transformers model to be trained args=training_args, # training arguments, defined above train_dataset=train_dataset, # training dataset eval_dataset=valid_dataset, # evaluation dataset compute_metrics=compute_metrics, # the callback that computes metrics of interest ) # train the model trainer.train()
原因分析与解决方案
可能原因
- 工作目录不匹配:代码中使用相对路径
./logs,但运行脚本时的终端工作目录并非项目根目录,导致程序找不到你创建的logs文件夹 - 权限不足:虽然创建了
logs文件夹,但Python进程没有对该文件夹的读写权限 - 同名文件存在:项目目录下可能存在名为
logs的文件(而非文件夹),导致TensorFlow无法将其识别为目录 - TensorFlow路径解析问题:Windows系统下,TensorFlow的
tf.io.gfile对相对路径的解析逻辑和Python原生存在差异,或路径存在编码问题
对应解决方案
验证并修正工作目录
在代码开头添加以下代码,打印当前工作目录:import os print("当前工作目录:", os.getcwd())如果输出不是
C:\Users\asus\Masaüstü\BitirmeProjesi,可以:- 切换终端到项目根目录后再运行脚本
- 或修改
TrainingArguments中的logging_dir为绝对路径:logging_dir='C:\\Users\\asus\\Masaüstü\\BitirmeProjesi\\logs'
检查文件夹权限
右键项目目录下的logs文件夹,选择「属性」→「安全」,确认当前用户拥有「读取」「写入」权限,若没有则添加对应权限。检查是否存在同名文件
打开项目目录,确认logs是文件夹而非文件。如果是文件,删除后重新创建logs文件夹。临时禁用TensorBoard日志(可选)
如果不需要使用TensorBoard查看训练日志,可以在TrainingArguments中添加report_to='none',跳过日志文件的创建:training_args = TrainingArguments( # 其他参数保持不变 report_to='none' )
内容的提问来源于stack exchange,提问作者Hilal Derya Eryılmaz
相关产品推荐
相关产品推荐

