使用RSelenium进行网页爬取时构建for循环并保存为DataFrame的异常问题

1. 冗余列产生的原因 #

核心问题出在你循环里的lapply(Position, ScrapeDF)这一行。这里的Position是一个webElement对象，而不是你以为的需要遍历的列表/向量。当你用lapply去处理这个对象时，它会遍历webElement的所有内部属性（比如checkStatus、statusClass、nativeEvents这些你看到的冗余字段），每一个属性都会被传入ScrapeDF函数。

更糟的是，你的ScrapeDF函数根本没用到传入的link_element参数——函数里每次都是直接重新查找页面元素生成数据，所以每遍历一个webElement属性，就会生成一个结构相同的DataFrame，最后bind_rows把这些重复的DataFrame合并时，就会把所有webElement的属性名作为列名保留下来，导致出现几百列冗余内容。

另外，函数里直接使用了全局的Text变量，这也是代码不规范的地方，但不是冗余列的直接原因。

2. 正确绑定数据的修复方案 #

我们需要改掉错误的lapply调用，直接调用ScrapeDF生成单行数据，再合并到总DataFrame里，同时优化函数的参数传递：

第一步：修改ScrapeDF函数 #

让函数接收位置文本作为参数，避免依赖全局变量，代码更健壮：

ScrapeDF <- function(position_text){
  # General Stats
  link_element <- remDr$findElement(using = "css selector",".statappearances")
  Appearance <- as.character(link_element$getElementText())
  link_element <- remDr$findElement(using = "css selector",".statwins")
  Wins <- as.character(link_element$getElementText())
  link_element <- remDr$findElement(using = "css selector",".statlosses")
  Losses <- as.character(link_element$getElementText())
  
  # Defence Stats（中间所有爬取代码保持不变，此处省略重复部分）
  link_element <- remDr$findElement(using = "css selector",".statclean_sheet")
  CleanSheet <- as.character(link_element$getElementText())
  # ... 其他防御/团队/纪律/进攻数据爬取代码 ...
  
  # 最后生成DataFrame时使用传入的position_text
  DF_Compiled <- data.frame(
    "Position" = position_text, 
    "Appearance" = Appearance, 
    "Wins" = Wins, 
    "Losses" = Losses, 
    "Goals" = Goals, 
    "HeadedGoals" = HeadedGoal, 
    "RightFootGoal" = RightFootGoal, 
    "LeftFootGoal" = LeftFootGoal, 
    "Woodwork" = Woodwork, 
    "YellowCard" = YelCard, 
    "RedCard" = RedCard, 
    "Fouls" = Fouls, 
    "Offside" = Offside, 
    "Assist" = Assists, 
    "Passes" = Passes, 
    "PassperMatch" = PassperMatch, 
    "BigChanceCreated" = BigChanceCreated, 
    "Crosses" = Crosses, 
    "CrossAcc" = CrossAcc, 
    "ThroughBall" = ThroughBall, 
    "AccLongBall" = LongBall, 
    "CleanSheet" = CleanSheet, 
    "Conceded" = Conceded, 
    "Tackles" = Tackles, 
    "SuccessTackle" = SuccessfulTackle, 
    "LastManTackle" = LastManTackle, 
    "BlockedShots" = BlockedShots, 
    "Interceptions" = Interceptions, 
    "Clearances" = Clearance, 
    "HeadedClearance" = HeadedClearance, 
    "OffLineClearance" = ClearanceOffLine,
    "Recoveries" = Recovery, 
    "DuelsWon" = DuelsWon, 
    "DuelsLost" = DuelsLost, 
    "Successful50_50" = Successful5050, 
    "AerialWon" = AerialWon, 
    "AerialLost" = AerialLost, 
    "OwnGoal" = OwnGoal, 
    "ErrorsToGoal" = ErrorsToGoal,
    stringsAsFactors = FALSE # 避免自动转换为因子，推荐添加
  )
  return(DF_Compiled)
}

第二步：修改循环逻辑 #

去掉错误的lapply，直接调用ScrapeDF生成单行数据，再合并到总DataFrame：

CompletePlayerData <- data.frame(matrix(nrow = 0,ncol = 0))
# 循环爬取每个玩家的URL
for (url in URL_list) {
  remDr$navigate(url)
  Sys.sleep(4) # 等待页面加载，时间可以根据实际情况调整
  
  # 获取球员位置
  Position <- remDr$findElement(using = "css selector",".info")
  Text <- as.character(Position$getElementText())
  player_df <- NULL
  
  # 判断是否为后卫，是则爬取数据
  if(Text == "Defender"){
    player_df <- ScrapeDF(Text)
  } else {
    # 尝试获取第二个.info元素的位置
    Position <- remDr$findElement(using = "css selector",".info~ .info")
    Text <- as.character(Position$getElementText())
    if(Text == "Defender"){
      player_df <- ScrapeDF(Text)
    }
  }
  
  # 如果成功获取到数据，合并到总DataFrame
  if(!is.null(player_df)){
    CompletePlayerData <- bind_rows(CompletePlayerData, player_df)
  }
}

关键修改点说明 #

• 移除了lapply(Position, ScrapeDF)：Position是单个webElement，我们只需要调用一次ScrapeDF就能得到当前球员的单行数据，不需要遍历它的内部属性。
• 给ScrapeDF添加了position_text参数：避免依赖全局变量，让函数的输入输出更清晰。
• 添加了player_df的空值判断：防止没有符合条件的球员时，合并空数据导致异常。

内容的提问来源于stack exchange，提问作者evanbourne220