lxml.html.parse无法读取部分URL，如何将requests结果转为DOM对象

一、为lxml设置自定义User-Agent #

lxml默认的HTTP请求头可能被目标网站反爬机制拦截，导致加载失败。可以通过urllib构建带自定义User-Agent的请求对象，再传给lxml解析：

from lxml import html
from urllib.request import build_opener, Request

url = "http://akademos-garden.com/homeschooling-tips-work-home-parents/"
# 自定义符合浏览器特征的User-Agent
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'}
req = Request(url, headers=headers)
opener = build_opener()
response = opener.open(req)
# 解析响应生成DOM树
tree = html.parse(response)

二、将requests响应转为lxml DOM对象 #

如果用requests.get能正常获取内容，只需把响应的二进制内容（response.content）传给lxml的解析方法即可：

import requests
from lxml import html

url = "http://akademos-garden.com/homeschooling-tips-work-home-parents/"
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'}
# 发送请求（即使不设UA返回200，建议带上避免后续被拦截）
response = requests.get(url, headers=headers)
# 生成lxml DOM树
tree = html.fromstring(response.content)
# 若需要保留完整文档结构（如DOCTYPE、根节点），改用document_fromstring
# tree = html.document_fromstring(response.content)

内容的提问来源于stack exchange，提问作者NealWalters