Htmlunit请求无HTTP头的第三方页面报错，求助解决方案

数据老陈

2026-5-14

Hey there! Let's figure out how to get HtmlUnit working with that tricky third-party server that's sending invalid HTTP headers. Browsers are pretty forgiving when it comes to non-compliant responses, but HtmlUnit (built on Apache HttpClient) is stricter by default—so we need to tweak its settings to match browser behavior.

Here are a few actionable fixes you can try:

1. Mimic a real browser's User-Agent

First, some servers tailor their responses based on the User-Agent string. If you're using HtmlUnit's default UA, the server might be sending a stripped-down or broken response. Let's set it to match a modern Chrome browser:

webClient.getOptions().setUserAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36");

This alone might fix the issue, as many servers behave better when they think they're talking to a mainstream browser.

2. Add a response interceptor to fix missing headers

If the server is skipping critical headers (like a status line or Content-Type), we can add an interceptor to inject those missing bits before HtmlUnit parses the response:

import org.apache.http.HttpResponse;
import org.apache.http.HttpResponseInterceptor;
import org.apache.http.HttpContext;
import org.apache.http.HttpException;
import org.apache.http.message.BasicStatusLine;
import org.apache.http.HttpVersion;
import org.apache.http.HttpStatus;

// Get the underlying HttpClient from HtmlUnit
CloseableHttpClient httpClient = (CloseableHttpClient) webClient.getWebConnection().getHttpClient();

// Add an interceptor to repair broken responses
httpClient.addInterceptorFirst(new HttpResponseInterceptor() {
    @Override
    public void process(HttpResponse response, HttpContext context) throws HttpException, IOException {
        // If no status line exists, add a default 200 OK
        if (response.getStatusLine() == null) {
            response.setStatusLine(new BasicStatusLine(HttpVersion.HTTP_1_1, HttpStatus.SC_OK, "OK"));
        }
        // Inject Content-Type if missing (adjust charset as needed)
        if (response.getFirstHeader("Content-Type") == null) {
            response.addHeader("Content-Type", "text/html; charset=utf-8");
        }
    }
});

This interceptor runs right after the response is received, fixing gaps before the parser throws an error.

3. Use a lenient HTTP response parser

The core error is coming from Apache HttpClient's strict response parsing. We can replace the default parser with a custom one that handles invalid protocol versions gracefully:

First, create the lenient parser class:

import org.apache.http.impl.conn.DefaultHttpResponseParser;
import org.apache.http.io.SessionInputBuffer;
import org.apache.http.message.LineParser;
import org.apache.http.message.BasicLineParser;
import org.apache.http.ProtocolVersion;
import org.apache.http.HttpException;

public class LenientHttpResponseParser extends DefaultHttpResponseParser {
    private static final LineParser FORGIVING_LINE_PARSER = new BasicLineParser() {
        @Override
        public ProtocolVersion parseProtocolVersion(final String line, final LineParser parser) throws HttpException {
            try {
                return super.parseProtocolVersion(line, parser);
            } catch (HttpException e) {
                // Fall back to HTTP/1.1 if parsing fails
                return new ProtocolVersion("HTTP", 1, 1);
            }
        }
    };

    public LenientHttpResponseParser(SessionInputBuffer buffer) {
        super(buffer, FORGIVING_LINE_PARSER, null, null);
    }
}

Then configure HtmlUnit's HttpClient to use this parser:

import org.apache.http.conn.ManagedHttpClientConnectionFactory;
import org.apache.http.conn.HttpConnectionCreationContext;
import org.apache.http.impl.conn.DefaultManagedHttpClientConnection;
import org.apache.http.conn.PoolingHttpClientConnectionManager;
import org.apache.http.conn.HttpClientConnectionManager;

HttpClientConnectionManager connManager = httpClient.getConnectionManager();
if (connManager instanceof PoolingHttpClientConnectionManager) {
    PoolingHttpClientConnectionManager poolingConnManager = (PoolingHttpClientConnectionManager) connManager;
    poolingConnManager.setConnectionFactory(new ManagedHttpClientConnectionFactory() {
        @Override
        public DefaultManagedHttpClientConnection create(HttpConnectionCreationContext context) throws IOException {
            return new DefaultManagedHttpClientConnection() {
                @Override
                protected DefaultHttpResponseParser createResponseParser(SessionInputBuffer buffer, 
                                                                         org.apache.http.io.MessageParserFactory<HttpResponse> responseParserFactory, 
                                                                         org.apache.http.params.HttpParams params) {
                    return new LenientHttpResponseParser(buffer);
                }
            };
        }
    });
}

This parser will ignore invalid protocol version strings and use a safe default instead of throwing an exception.

4. Upgrade HtmlUnit to the latest version

Newer versions of HtmlUnit often include improvements to handle non-compliant servers. Check if you're using the latest release (as of now, 2.70.0) and update your dependencies if needed—this might resolve the issue without custom code.

Putting it all together

Here's how your modified code would look with all these fixes:

import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
import com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.HttpResponseInterceptor;
import org.apache.http.HttpResponse;
import org.apache.http.HttpContext;
import org.apache.http.HttpException;
import org.apache.http.message.BasicStatusLine;
import org.apache.http.HttpVersion;
import org.apache.http.HttpStatus;
import org.apache.http.conn.PoolingHttpClientConnectionManager;
import org.apache.http.conn.HttpClientConnectionManager;
import org.apache.http.conn.ManagedHttpClientConnectionFactory;
import org.apache.http.conn.HttpConnectionCreationContext;
import org.apache.http.impl.conn.DefaultManagedHttpClientConnection;
import org.apache.http.impl.conn.DefaultHttpResponseParser;
import org.apache.http.io.SessionInputBuffer;
import org.apache.http.message.LineParser;
import org.apache.http.message.BasicLineParser;
import org.apache.http.ProtocolVersion;

import java.io.IOException;

public class YourLoginClass {
    public void login() throws IOException {
        WebClient webClient = new WebClient();
        webClient.getOptions().setUseInsecureSSL(true);
        webClient.getOptions().setThrowExceptionOnScriptError(false);
        webClient.getOptions().setJavaScriptEnabled(true);
        webClient.getOptions().setCssEnabled(false);
        webClient.getOptions().setRedirectEnabled(true);
        webClient.getOptions().setPopupBlockerEnabled(false);
        webClient.setAjaxController(new NicelyResynchronizingAjaxController());
        // Mimic Chrome's User-Agent
        webClient.getOptions().setUserAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36");

        // Get HttpClient and add response interceptor
        CloseableHttpClient httpClient = (CloseableHttpClient) webClient.getWebConnection().getHttpClient();
        httpClient.addInterceptorFirst(new HttpResponseInterceptor() {
            @Override
            public void process(HttpResponse response, HttpContext context) throws HttpException, IOException {
                if (response.getStatusLine() == null) {
                    response.setStatusLine(new BasicStatusLine(HttpVersion.HTTP_1_1, HttpStatus.SC_OK, "OK"));
                }
                if (response.getFirstHeader("Content-Type") == null) {
                    response.addHeader("Content-Type", "text/html; charset=utf-8");
                }
            }
        });

        // Configure lenient response parser
        HttpClientConnectionManager connManager = httpClient.getConnectionManager();
        if (connManager instanceof PoolingHttpClientConnectionManager) {
            PoolingHttpClientConnectionManager poolingConnManager = (PoolingHttpClientConnectionManager) connManager;
            poolingConnManager.setConnectionFactory(new ManagedHttpClientConnectionFactory() {
                @Override
                public DefaultManagedHttpClientConnection create(HttpConnectionCreationContext context) throws IOException {
                    return new DefaultManagedHttpClientConnection() {
                        @Override
                        protected DefaultHttpResponseParser createResponseParser(SessionInputBuffer buffer, 
                                                                                 org.apache.http.io.MessageParserFactory<HttpResponse> responseParserFactory, 
                                                                                 org.apache.http.params.HttpParams params) {
                            return new LenientHttpResponseParser(buffer);
                        }
                    };
                }
            });
        }

        // Try loading the page again
        HtmlPage page = webClient.getPage("http://applicationtoanalize.com/login.cgi");
        // Now you can use XPath queries like this:
        // List<HtmlElement> loginElements = page.getByXPath("//form[@id='login-form']");
        // ... your processing logic ...

        webClient.close();
    }
}

// Custom lenient parser class
class LenientHttpResponseParser extends DefaultHttpResponseParser {
    private static final LineParser FORGIVING_LINE_PARSER = new BasicLineParser() {
        @Override
        public ProtocolVersion parseProtocolVersion(final String line, final LineParser parser) throws HttpException {
            try {
                return super.parseProtocolVersion(line, parser);
            } catch (HttpException e) {
                return new ProtocolVersion("HTTP", 1, 1);
            }
        }
    };

    public LenientHttpResponseParser(SessionInputBuffer buffer) {
        super(buffer, FORGIVING_LINE_PARSER, null, null);
    }
}

These changes should make HtmlUnit as forgiving as a regular browser, allowing you to load the page and run your XPath queries successfully.

内容的提问来源于stack exchange，提问作者Marcius