将含15万条记录的复杂JSON转Dataframe遇1:nrow(test)错误求助

Hey there, let's break down what's going wrong and fix your code step by step.

The Root Cause #

That error pops up because your test object is a list (not a data frame)! Looking at your JSON structure, it's an array of objects—when you convert that to R (say with jsonlite::fromJSON()), you get a list, not a 2D data frame. nrow() only works on data frames/matrices, so calling nrow(test) returns NULL, which makes 1:nrow(test) throw that "argument of length 0" error.

Fixed Code & Explanations #

Here's the adjusted code, with changes explained so you understand what's different:

Step 1: Correctly Iterate Over List Elements #

Instead of 1:nrow(test), use seq_along(test) to loop through each element in your list. This works for both lists and data frames, so it's safer.

Step 2: Simplify Empty Response Checks #

Your original code had odd indexing (test[[1]][[j]]) that wasn't targeting the right elements. Instead, we can directly check if each element in the test list is empty.

Step 3: Clean Up Data Frame Construction #

We'll directly extract the nested daily data from valid responses and attach the corresponding reviewid cleanly.

# test <- Your JSON conversion result (e.g., from jsonlite::fromJSON())
reviewid <- c(98338143, 58929813, 65945346)

# Filter valid (non-empty) responses and null responses
valid <- which(sapply(test, function(x) length(x) > 0))
NullResponses <- which(sapply(test, function(x) length(x) == 0))

# Build list of data frames from valid responses
dflist <- lapply(valid, function(j) {
  # Extract the nested daily data from the j-th response
  daily_data <- test[[j]]$daily[[1]]
  # Create data frame with reviewid attached
  df <- data.frame(reviewid = reviewid[j], daily_data, stringsAsFactors = FALSE)
  df
})

# Combine all data frames into one
answer <- dplyr::bind_rows(dflist)

Pro Tips for Handling Large (150k Row) Data #

Since you're working with 150k observations, here's a more efficient approach using purrr (avoids pre-filtering valid and is faster for big datasets):

library(purrr)

answer <- map_dfr(seq_along(test), function(j) {
  # Skip empty responses entirely
  if (length(test[[j]]) == 0) return(NULL)
  # Extract and format data
  daily_data <- test[[j]]$daily[[1]]
  data.frame(reviewid = reviewid[j], daily_data, stringsAsFactors = FALSE)
})

Also, when converting your JSON, use jsonlite::fromJSON(simplifyVector = FALSE) to keep the raw list structure—this avoids unexpected auto-conversions that can mess up your data. Running str(test) will help you visualize the structure if you get stuck later!

内容的提问来源于stack exchange，提问作者Evelien van der Waal