使用rvest爬取FiveThirtyEight球员5年市场价值返回NA的问题

Hey there! Let's break down why your rvest code is returning NA and fix it up.

The Root Problem #

The issue here is that the market value data on that FiveThirtyEight page is dynamically rendered with JavaScript. When you use read_html() directly, rvest only grabs the initial static HTML sent by the server—this doesn't include the content that loads after the page's JavaScript runs. The .market-value element you're targeting doesn't exist in that static HTML, hence the NA result.

Solution 1: Use RSelenium to Capture Dynamically Rendered Content #

RSelenium lets you control a real web browser, which will execute the page's JavaScript and load all the dynamic content. Here's how to use it:

First, make sure you have ChromeDriver installed (it needs to match your Chrome browser version). Then run this code:

# Install required packages if you haven't already
install.packages(c("RSelenium", "rvest"))

library(RSelenium)
library(rvest)

# Start a Chrome browser session
driver <- rsDriver(browser = "chrome", port = 4567L)
remote_driver <- driver[["client"]]

# Navigate to the page
remote_driver$navigate("https://projects.fivethirtyeight.com/carmelo/kyrie-irving/")

# Wait 3 seconds to give JavaScript time to load the data
Sys.sleep(3)

# Grab the fully rendered page source
page_source <- remote_driver$getPageSource()[[1]]
kyrie_irving <- read_html(page_source)

# Extract the market value
market_value <- kyrie_irving %>% 
  html_node(".market-value") %>% 
  html_text() %>% 
  as.numeric()

# Print the result
print(market_value)

# Clean up: close the browser session
remote_driver$close()
driver$server$stop()

Solution 2: Scrape the Direct API Endpoint (More Efficient) #

Many dynamic sites pull data from a backend API. You can skip rendering the whole page by calling this API directly:

1. Open your browser's Developer Tools (F12), go to the Network tab, and reload the page.
2. Look for XHR/fetch requests—you'll likely find a JSON endpoint that returns player data (including market value).
3. Use httr and jsonlite to fetch and parse that data:

install.packages(c("httr", "jsonlite"))

library(httr)
library(jsonlite)

# Replace this with the actual API endpoint you find in Developer Tools
api_response <- GET("https://projects.fivethirtyeight.com/carmelo/api/players/kyrie-irving.json")
player_data <- fromJSON(content(api_response, "text"))

# Extract the 5-year market value (adjust the field name based on the JSON structure)
five_year_value <- player_data$market_value$five_year
print(five_year_value)

This method is faster and more reliable than using a browser, since you're pulling the raw data directly.

内容的提问来源于stack exchange，提问作者dsfreddo