R语言批量转换遥测TXT数据为CSV文件代码运行报错求解

问题背景 #

• 存储为系列TXT文件的无线电遥测数据，配套有2018年编写的批量遍历TXT、转换为CSV的R代码，2021年之前代码可正常运行，近期运行至lapply循环段时持续报错。

初始报错信息 #

Error in setnames(x, value) :
Can't assign 1 names to a 4 column data.table

初始运行的完整代码如下：

# set the working directory to the folder that contain this script, must run in RStudio 
setwd(dirname(rstudioapi::callFun("getActiveDocumentContext")$path)) 

# get the path to the master data folder
path_to_data <- paste(getwd(), "data", sep = "/", collapse = NULL)

# extract .TXT file
files <- list.files(path=path_to_data, pattern="*.TXT", full.names=TRUE, recursive=TRUE)

# regular expression of the record we want
regex <- "^\\d*\\/\\d*\\/\\d*\\s*\\d*:\\d*:\\d*\\s*\\d*\\s*\\d*\\s*\\d*\\s*\\d*"

# vector of column names, no whitespace
columns <- c("Date", "Time", "Channel", "TagID", "Antenna", "Power")

# loop through all .TXT files, extract valid records and save to .csv files
lapply(files, function(x){
  df <- read_table(file) # read the .TXT file to a DataFrame
  dt <- data.table(df) # convert the dataframe to a more efficient data structure 
  colnames(dt) <- c("columns") # modify the column name
  valid <- dt %>% filter(str_detect(col, regex)) # filter based on regular expression
  valid <- separate(valid, col, into = columns, sep = "\\s+") # split into columns
  towner_name <- str_sub(basename(file), start = 1 , end = 2) # extract tower name
  valid$Tower <- rep(towner_name, nrow(valid)) # add Tower column
  file_path <- file.path(dirname(file), paste(str_sub(basename(file), end = -5), ".csv", sep=""))
  write.csv(valid, file = file_path, row.names = FALSE, quote = FALSE) # save to .csv
})

尝试在循环中加入setnames(skip_absent=TRUE)后，第一个报错消失，但触发新报错：

Error in is.data.frame(x) : argument "x" is missing, with no default

修改后的循环代码如下：

lapply(files, function(file){
  df <- read_table(file) # read the .TXT file to a DataFrame
  dt <- data.table(df) # convert the dataframe to a more efficient data structure 
  setnames(skip_absent = TRUE)
  colnames(dt) <- c("col") # modify the column name
  valid <- dt %>% filter(str_detect(col, regex)) # filter based on regular expression
  valid <- separate(valid, col, into = columns, sep = "\\s+") # split into columns
  towner_name <- str_sub(basename(file), start = 1 , end = 2) # extract tower name
  valid$Tower <- rep(towner_name, nrow(valid)) # add Tower column
  file_path <- file.path(dirname(file), paste(str_sub(basename(file), end = -5), ".csv", sep=""))
  write.csv(valid, file = file_path, row.names = FALSE, quote = FALSE) # save to .csv
})

报错根因 #

代码失效核心是R依赖包版本迭代后，旧代码里的隐性笔误、不规范写法不再被容错，具体问题点：

• 匿名函数参数不匹配：第一版循环定义的传入参数是x，但函数内部读文件、处理路径时用的变量是file，参数完全没传入逻辑。旧版本运行时如果全局环境恰好存在同名file变量就不会报错，环境清理后就会触发异常。
• 列名赋值逻辑错误：原始TXT本应按整行读入、存为1列数据，旧代码写colnames(dt) <- c("columns")既没指定正确的列名，也没限制读文件时的列拆分规则。如果TXT里存在空行、多空格分隔的不规则内容，read_table会自动猜测文件结构、把内容拆成多列，此时强行给多列data.table赋1个列名，就会触发第一个列数不匹配的报错。
• setnames调用方式错误：该函数是data.table包提供的列名修改函数，必须传入要操作的data.table对象作为第一个参数，仅传skip_absent=TRUE缺少核心操作对象，就会触发参数缺失的报错。
• 读文件未做类型约束：调用read_table时没有指定列名、列类型，函数自动推断结构的逻辑遇到格式稍不规整的遥测TXT，就会拆分出多余列，导致后续所有处理逻辑错位。

修复后可运行代码 #

提前安装好依赖包：rstudioapi、readr、data.table、dplyr、tidyr、stringr，运行以下代码即可：

# 加载依赖包
library(rstudioapi)
library(readr)
library(data.table)
library(dplyr)
library(tidyr)
library(stringr)

# 设置工作目录为脚本所在文件夹，需在RStudio环境运行
setwd(dirname(rstudioapi::callFun("getActiveDocumentContext")$path)) 

# 定位数据存储文件夹
path_to_data <- paste(getwd(), "data", sep = "/")

# 递归提取所有后缀为.TXT的文件路径
files <- list.files(path=path_to_data, pattern="*.TXT", full.names=TRUE, recursive=TRUE)

# 匹配有效遥测记录的正则规则
regex <- "^\\d*\\/\\d*\\/\\d*\\s*\\d*:\\d*:\\d*\\s*\\d*\\s*\\d*\\s*\\d*\\s*\\d*"

# 定义拆分后的字段列名
columns <- c("Date", "Time", "Channel", "TagID", "Antenna", "Power")

# 批量遍历处理所有TXT文件
lapply(files, function(file_path){
  # 按行读入TXT，强制所有内容存为1列字符型数据，列名设为col，避免自动拆分多列
  df <- read_table(file_path, col_names = "col", col_types = cols(col = col_character()))
  dt <- as.data.table(df)
  # 过滤匹配正则的有效记录
  valid <- dt %>% filter(str_detect(col, regex))
  # 按连续空白符拆分单列为指定字段
  valid <- separate(valid, col, into = columns, sep = "\\s+")
  # 提取文件名前2位作为基站编号
  tower_name <- str_sub(basename(file_path), start = 1 , end = 2)
  valid$Tower <- rep(tower_name, nrow(valid))
  # 生成对应CSV输出路径
  output_path <- file.path(dirname(file_path), paste0(str_sub(basename(file_path), end = -5), ".csv"))
  # 写出CSV文件
  write.csv(valid, file = output_path, row.names = FALSE, quote = FALSE)
})

内容的提问来源于stack exchange，提问作者Jallen4