Hey there! Let's tackle your two questions about scraping JavaScript-rendered pages in Python without relying on Selenium, and how tools like BeautifulSoup or lxml fit into the picture.

1. 不用Selenium，Python能爬取JS渲染的页面吗？ #

Absolutely! You don’t need Selenium for every JS-rendered page—there are several workarounds depending on how the website loads its dynamic content:

• Target backend APIs directly: Most modern sites load dynamic data via XHR/Fetch requests to their APIs. Instead of scraping rendered HTML, you can inspect your browser’s network tab to find these API endpoints, then use requests to fetch raw JSON data. This is often faster and cleaner than dealing with rendered HTML.
• Use headless browser alternatives: Tools like requests-html (bundles a lightweight Chromium instance) or pyppeteer (Python’s port of Puppeteer) can render JavaScript just like a real browser, without Selenium’s heavy setup. They return fully rendered HTML for you to parse.
• Extract embedded JSON from script tags: Many single-page apps (React/Vue sites) embed initial state data directly in a <script> tag (e.g., window.__INITIAL_STATE__). Use a parser like BeautifulSoup to grab this script content, then extract and parse the JSON with Python’s built-in json module—no JS execution needed.

2. 用BeautifulSoup或lxml，不用Selenium能爬JS渲染网页吗？ #

Short answer: BeautifulSoup and lxml can’t execute JavaScript on their own—they only parse static HTML. If you just use requests to fetch raw page source and pass it to these tools, you’ll only get pre-rendered HTML (before JS runs), missing dynamic content.

But you can pair them with the methods above to avoid Selenium:

• Pair with API data: If you fetch JSON from an API, you might not even need BeautifulSoup/lxml (unless the API returns HTML snippets). But if you do, you can parse those snippets with either tool.
• Pair with headless tools: Use requests-html or pyppeteer to get fully rendered HTML, then feed that HTML into BeautifulSoup or lxml for extraction. This combines JS rendering power with the ease of parsing these tools offer.
• Extract embedded JSON: Use BeautifulSoup to locate the script tag with embedded data, extract the text, parse it as JSON, and work directly with structured data.

Here’s a quick example using requests-html + BeautifulSoup:

from requests_html import HTMLSession
from bs4 import BeautifulSoup

# Initialize a JS-rendering session
session = HTMLSession()
resp = session.get("https://your-js-rendered-site.com")
resp.html.render()  # Executes the page's JavaScript

# Pass rendered HTML to BeautifulSoup for parsing
soup = BeautifulSoup(resp.html.html, "lxml")
dynamic_content = soup.find("div", class_="dynamic-content").get_text(strip=True)
print(dynamic_content)

A quick note: If the site has heavy anti-scraping measures or requires complex user interactions (clicking buttons, filling forms), Selenium might still be the easier option. But for most standard dynamic content, the above methods work great without it.

内容的提问来源于stack exchange，提问作者Kemal Ebubekir Atabey