使用R进行网页抓取：CSS与XPath使用难题

Hey Ricardo, sorry to hear you're hitting a wall with rvest—let's work through this together to get those title and body contents extracted successfully.

First, Rule Out Static vs. Dynamic Content #

The most common roadblock with rvest is dynamic JavaScript-rendered content. rvest only pulls static HTML (what you see when you right-click > "View Page Source" in your browser). If your target title or body doesn't show up in that source, the page loads content after the initial HTML loads, and you'll need a browser-simulating tool like RSelenium or playwrightR instead of plain rvest.

If the content is in the static source, let's walk through the standard workflow:

Step 1: Load rvest & Fetch the Page #

Start with the basics to ensure you're pulling the page correctly:

library(rvest)

# Replace with your example page URL
target_url <- "https://your-example-page.com"
page <- read_html(target_url)

Step 2: Pinpoint the Right Selectors (CSS or XPath) #

Use your browser's DevTools (F12 > "Elements" tab) to inspect where the title and body live in the HTML. Here's how to translate that into rvest code:

Extracting the Title

Most pages use an <h1> for the main page title, or a <title> tag for the browser tab title. Try these approaches:

# Option 1: CSS selector (adjust to match your page's actual tag/class)
main_title <- page %>% 
  html_element("h1.article-headline") %>% 
  html_text()

# Option 2: XPath selector (useful if CSS is tricky)
main_title <- page %>% 
  html_element(xpath = "//h1[@class='article-headline']") %>% 
  html_text()

# For the browser tab title instead:
tab_title <- page %>% html_element("title") %>% html_text()

Extracting the Body Content

Body content is usually in a container <div> with a class like article-body, post-content, or similar. Use html_text2() instead of html_text() to handle line breaks and spacing cleanly:

# CSS selector for body content
body_content <- page %>% 
  html_element("div.post-content") %>%  # Adjust to your page's container
  html_text2()

# XPath alternative (use contains() if the class has multiple values)
body_content <- page %>% 
  html_element(xpath = "//div[contains(@class, 'post-content')]") %>% 
  html_text2()

Common Pitfalls to Fix #

• Dynamic class names: Some sites use auto-generated class names (like css-789abc) that change. Instead, target parent elements or stable attributes (like id, which is unique) to locate content.
• Multiple matching elements: If html_element() returns nothing, try html_elements() (plural) to see all matches—you might need to pick the right index (e.g., html_elements("h1")[[1]]).
• Anti-scraping blocks: Some sites reject non-browser requests. Add a user-agent header to mimic a real browser:

  page <- read_html(target_url, user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36")
  

If you're still stuck, sharing the example page URL or a snippet of the HTML structure for the title/body sections would help narrow down the issue even more.

内容的提问来源于stack exchange，提问作者Ricardo Stary Rodrigues