如何让Jsoup仅加载所需资源以提升网页爬取效率？

First off, a key point to clarify: Jsoup doesn’t automatically load external resources like images, CSS files, or external links (e.g., Facebook) by default. When you use Jsoup.connect(url).get(), it only fetches the raw HTML content of the target page—those extra resources aren’t downloaded at all. So if you’re seeing slow load times, it’s likely due to the target website’s server response speed, or you can optimize your request to speed things up further.

That said, you can tweak your single Jsoup.connect line with a few configurations to make the request even more efficient and ensure no accidental external resource requests (though again, Jsoup doesn’t do this by default):

• Set a strict timeout: Prevents your request from hanging indefinitely if the server is slow
• Disable unnecessary redirects: Stops Jsoup from following redirects to external sites
• Use a realistic User-Agent: Some websites throttle requests from non-browser user agents, slowing down responses

Here’s how your updated line would look:

Document doc = Jsoup.connect("YOUR_TARGET_URL")
                    .userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36")
                    .timeout(3000) // 3-second timeout, adjust as needed
                    .followRedirects(false)
                    .get();

Quick breakdown of each addition: #

• userAgent(...): Makes your request look like it’s coming from a standard desktop browser, which can help avoid rate-limiting or slow responses from the server.
• timeout(3000): Ensures you don’t wait longer than 3 seconds for a response—tweak the millisecond value based on your tolerance for wait times.
• followRedirects(false): If the target page tries to redirect you to an external site (like Facebook), Jsoup won’t follow that redirect, saving you time and unnecessary requests.

If you’re still experiencing slow load times after these tweaks, it’s probably an issue with the target website’s server performance, not Jsoup loading extra resources.

内容的提问来源于stack exchange，提问作者user9549355