使用Jsoup编写爬虫时遭遇HTTP错误的技术求助

Hey Anna, let's work through this 404 error your Jsoup crawler is hitting. The error message gives us all the clues we need—let's break it down step by step:

1. Spot the Malformed URL #

First, look at the problematic URL from the error:

https://www.mkyong.com/spring-boot/spring-boot-hibernate-search-example/%E2%80%9Chttp:/wildfly.org/downloads/

This URL is completely invalid—it’s accidentally stitching together two separate URLs (a Spring Boot tutorial page and a WildFly download link) plus an encoded left quote (%E2%80%9C, which maps to the character “). No server will recognize this as a valid address, hence the 404.

2. Common Root Causes to Check #

• Messy Link Extraction: If you’re scraping links from a web page, you might be pulling extra text along with the actual URL. For example, maybe you’re grabbing element text instead of the href attribute, or the page’s HTML has quotes embedded in the link value that you’re not stripping out.
• Buggy String Concatenation: Double-check your code where you build or modify URLs. It’s easy to accidentally append a second URL or stray characters when combining strings (like copying a test URL without cleaning it up first).

3. Fixes to Implement #

• Validate Target URLs Manually: First, confirm the correct address you want to crawl. Open either https://www.mkyong.com/spring-boot/spring-boot-hibernate-search-example/ or https://wildfly.org/downloads/ (note the https for WildFly—most modern sites use HTTPS now) in your browser to ensure they load properly.
• Clean Up Extracted Links: When pulling links from a page, always target the href attribute directly (e.g., element.attr("href") instead of element.text()). Then add cleanup steps:
  • Use url.trim() to remove leading/trailing whitespace.
  • Strip unwanted quotes with url.replaceAll("[\"']", "") to eliminate literal or encoded quote characters.
• Validate URLs Before Requesting: Add a quick check in your code to catch invalid URLs early. Use Java’s built-in URL class to parse the string before sending the Jsoup request:

  try {
      new URL(yourTargetUrl);
      // Proceed with Jsoup connection
      Document doc = Jsoup.connect(yourTargetUrl).get();
  } catch (MalformedURLException e) {
      // Log or handle the invalid URL
      System.out.println("Skipping invalid URL: " + yourTargetUrl);
  }
  

内容的提问来源于stack exchange，提问作者Anna Noukou