如何用Spring Boot获取网页HTML代码并存储为字符串？求示例

Hey there! I totally get the frustration of hunting for a clean, working example to grab HTML from a website and store it in a string variable with Spring Boot. Let’s break down two solid approaches you can use right away.

Approach 1: Using RestTemplate (Traditional Synchronous) #

RestTemplate is the tried-and-true synchronous HTTP client for Spring apps. Here’s how to use it:

Step 1: Add Dependencies #

Make sure you have the spring-boot-starter-web dependency in your pom.xml (Maven) or build.gradle (Gradle)—it includes RestTemplate by default.

Step 2: Configure RestTemplate Bean #

Since Spring Boot doesn’t auto-configure RestTemplate anymore, define it as a bean in your configuration class:

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.client.RestTemplate;

@Configuration
public class WebConfig {

    @Bean
    public RestTemplate restTemplate() {
        return new RestTemplate();
    }
}

Step 3: Fetch HTML and Store as String #

Inject the RestTemplate into a service or controller, then make the GET request:

import org.springframework.stereotype.Service;
import org.springframework.web.client.RestTemplate;
import org.springframework.web.client.HttpClientErrorException;
import org.springframework.web.client.ResourceAccessException;

@Service
public class HtmlFetcherService {

    private final RestTemplate restTemplate;

    // Constructor injection (preferred over @Autowired)
    public HtmlFetcherService(RestTemplate restTemplate) {
        this.restTemplate = restTemplate;
    }

    public String fetchHtmlAsString(String url) {
        try {
            // getForObject directly returns the response body as a String
            return restTemplate.getForObject(url, String.class);
        } catch (HttpClientErrorException e) {
            // Handle 4xx errors like 404 Not Found
            throw new RuntimeException("Failed to fetch HTML: Client error - " + e.getStatusCode(), e);
        } catch (ResourceAccessException e) {
            // Handle connection issues, timeouts, etc.
            throw new RuntimeException("Failed to connect to URL: " + url, e);
        }
    }
}

Approach 2: Using WebClient (Reactive, Spring 5+) #

WebClient is the modern, non-blocking alternative to RestTemplate, recommended for new Spring applications.

Step 1: Add Dependencies #

Include spring-boot-starter-webflux in your build file:

<!-- Maven -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-webflux</artifactId>
</dependency>

Step 2: Fetch HTML (Synchronous or Reactive) #

You can use WebClient in a synchronous way (if you need to block for the result) or fully reactive. Here’s both options:

Synchronous Usage (Block for Result)

import org.springframework.stereotype.Service;
import org.springframework.web.reactive.function.client.WebClient;
import org.springframework.web.reactive.function.client.WebClientResponseException;

@Service
public class ReactiveHtmlFetcherService {

    private final WebClient webClient;

    public ReactiveHtmlFetcherService(WebClient.Builder webClientBuilder) {
        this.webClient = webClientBuilder.build();
    }

    public String fetchHtmlSync(String url) {
        try {
            // Block to get the string result synchronously
            return webClient.get()
                    .uri(url)
                    .retrieve()
                    .bodyToMono(String.class)
                    .block(); // Only use block() if you absolutely need synchronous behavior
        } catch (WebClientResponseException e) {
            throw new RuntimeException("Failed to fetch HTML: Status code " + e.getStatusCode(), e);
        }
    }

    // Reactive usage (no blocking)
    public Mono<String> fetchHtmlReactive(String url) {
        return webClient.get()
                .uri(url)
                .retrieve()
                .bodyToMono(String.class)
                .onErrorResume(WebClientResponseException.class, e -> {
                    // Handle errors reactively
                    return Mono.error(new RuntimeException("Reactive fetch failed: " + e.getStatusCode(), e));
                });
    }
}

Quick Usage Example #

In a controller or main class, you can call these services like so:

import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;

@SpringBootApplication
public class HtmlFetcherApplication {

    public static void main(String[] args) {
        SpringApplication.run(HtmlFetcherApplication.class, args);
    }

    @Bean
    CommandLineRunner run(HtmlFetcherService htmlFetcherService) {
        return args -> {
            String url = "https://example.com";
            String htmlContent = htmlFetcherService.fetchHtmlAsString(url);
            // Now htmlContent holds the full HTML of the website
            System.out.println("Fetched HTML length: " + htmlContent.length());
        };
    }
}

Key Notes #

• Always handle exceptions properly—network calls can fail due to timeouts, invalid URLs, or server errors.
• For production use, add timeouts to both RestTemplate and WebClient to prevent hanging requests.
• If the website requires custom headers (like a User-Agent), add them to the request (e.g., restTemplate.getForObject(url, String.class, Map.of("User-Agent", "Spring Boot HTML Fetcher")) or via WebClient's header() method).

内容的提问来源于stack exchange，提问作者AJITHAN