通过SDMX(XML)在R中获取OECD REGION_ECONOM数据集遇400错误求助

Hey there, let's figure out why you're hitting that 400 Bad Request error and how to fix it—this is a common issue with large OECD dataset requests, so you're not alone!

Why the Error Happens #

There are two main culprits here:

1. URL Length Limits #

When you use ALL for regions and include a huge range of years (1990-2018), the generated SDMX URL gets extremely long. Most web servers and HTTP clients have a limit on URL length (typically around 2000-4000 characters). When your URL exceeds this limit, the server rejects the request with a 400 error. Your partial-region request works because the URL is short enough to stay under this threshold.

2. OECD API Request Constraints #

The OECD API has built-in limits on how much data you can request in a single call. Datasets like REGION_ECONOM have hundreds of regional codes, so combining all regions with multiple indicators (GDP + POP_AVG) and a long time range creates a request that's too large for the server to process efficiently. Other datasets like FTPTC_D have far fewer regions, so their full requests don't trigger this restriction.

Fixes to Try #

1. Split Requests into Batches #

The most reliable solution is to break your request into smaller chunks. You can split by regions or time periods, then combine the results. Here's how to do it with the OECD package:

if (!require(OECD)) {
  install.packages('OECD')
  library(OECD)
}

# First, get all valid region codes for REGION_ECONOM
data_structure <- get_data_structure("REGION_ECONOM")
all_regions <- data_structure$REGION$id

# Split regions into smaller groups (e.g., 20 regions per batch)
region_batches <- split(all_regions, ceiling(seq_along(all_regions)/20))

# Loop through each batch to fetch data
data_list <- lapply(region_batches, function(batch) {
  # Format the filter string for this batch
  region_filter <- paste(batch, collapse = "+")
  full_filter <- paste0(region_filter, ".GDP+POP_AVG")
  
  # Fetch data for the batch
  get_dataset("REGION_ECONOM",
              filter = full_filter,
              start_time = 2008,
              end_time = 2009,
              pre_formatted = TRUE)
})

# Combine all batches into one data frame
final_data <- do.call(rbind, data_list)
head(final_data)

2. Shorten the Time Range #

If you don't need every year from 1990-2018, split the time range into smaller chunks (e.g., 1990-2000, 2001-2010, 2011-2018). Fetch each chunk separately, then merge them together. This reduces the URL length and the total data per request.

3. Verify SDMX URL Format #

Double-check that your SDMX URL follows the correct dimension order for the REGION_ECONOM dataset. You can use get_data_structure("REGION_ECONOM") to view the official dimension sequence—if your URL mixes up the order (e.g., putting indicators before regions), it will trigger a 400 error. For example, confirm that the URL structure matches the dataset's defined dimension hierarchy.

Quick Validation Tip #

Before running a full batch, test with a small subset of regions and years to confirm your request format works. Once that's successful, scale up to larger batches.

内容的提问来源于stack exchange，提问作者Luna