Scrapy结合Splash无法加载动态内容求助
Scrapy + Splash 无法加载页面动态渲染内容
我正在使用Scrapy搭配Splash加载页面中的动态渲染内容,但未达到预期效果,以下是我的配置与代码:
settings.py 配置
SPIDER_MIDDLEWARES = { 'scrapy_splash.SplashDeduplicateArgsMiddleware': 100, } DOWNLOADER_MIDDLEWARES = { 'scrapy_splash.SplashCookiesMiddleware': 723, 'scrapy_splash.SplashMiddleware': 725, } SPLASH_URL="http://localhost:8050" DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter' HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage' SPLASH_COOKIES_DEBUG = False
爬虫核心代码
def start_requests(self): urls = [ "https://callmeduy.com/san-pham/" ] for url in urls: yield SplashRequest(url=url, # endpoint='render.html', callback=self.parse, args={ 'wait': 5 }) def parse(self, response): print(response.xpath("//body").get()) f = open('res.html', 'w+') f.write(response.xpath("//body").get()) f.close()
当前问题
目前页面的动态内容仍未加载,响应内容如下:
恳请各位技术人士帮忙解决。
内容的提问来源于stack exchange,提问作者mhtuan
相关产品推荐
相关产品推荐

