如何使用crawler4J爬取网站图片及配置登录凭证实现需登录站点爬取

Got it, let's tackle how to add login authentication to your crawler4J image crawler. Since crawler4J uses Apache HttpClient under the hood, we can leverage HttpClient's cookie handling to maintain a logged-in session across all crawler requests. Here's a step-by-step solution:

First, we need to simulate the login request to the target site, capture the session cookies, and make them available to the crawler. Most sites use form-based authentication, so we'll construct a POST request with your credentials. If the site uses CSRF tokens (common for secure logins), we'll first fetch the login page to extract that token.

import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.BasicCookieStore;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

public class LoginUtils {
    public static BasicCookieStore performLogin(String loginUrl, String username, String password) throws Exception {
        BasicCookieStore cookieStore = new BasicCookieStore();
        HttpClient httpClient = HttpClients.custom().setDefaultCookieStore(cookieStore).build();
        HttpClientContext context = HttpClientContext.create();
        
        // Step 1: Fetch login page to extract CSRF token (adjust if your site doesn't use this)
        HttpGet getLoginPage = new HttpGet(loginUrl);
        HttpResponse loginPageResponse = httpClient.execute(getLoginPage, context);
        String loginPageContent = EntityUtils.toString(loginPageResponse.getEntity());
        
        // Parse CSRF token using Jsoup (update the selector to match your site's HTML)
        Document doc = Jsoup.parse(loginPageContent);
        String csrfToken = doc.select("input[name=_csrf]").val();
        
        // Step 2: Build and send login POST request
        HttpPost loginPost = new HttpPost(loginUrl);
        // Match form parameter names to your site's login form (check dev tools for exact keys)
        String formData = String.format("username=%s&password=%s&_csrf=%s", 
            username, password, csrfToken);
        loginPost.setEntity(new StringEntity(formData));
        loginPost.setHeader("Content-Type", "application/x-www-form-urlencoded");
        
        HttpResponse loginResponse = httpClient.execute(loginPost, context);
        EntityUtils.consume(loginResponse.getEntity()); // Ensure we complete the request
        
        // Return cookie store with authenticated session cookies
        return cookieStore;
    }
}

Note: Use your browser's dev tools (Network tab) to inspect the actual login request parameters—you’ll need to adjust form field names (like username/password) and the CSRF token selector to match the target site’s HTML.

Next, we’ll pass the authenticated cookie store to crawler4J’s HttpClient setup. Modify the controller class from the image crawler example to integrate this:

import edu.uci.ics.crawler4j.crawler.CrawlConfig;
import edu.uci.ics.crawler4j.crawler.CrawlController;
import edu.uci.ics.crawler4j.fetcher.PageFetcher;
import edu.uci.ics.crawler4j.robotstxt.RobotstxtConfig;
import edu.uci.ics.crawler4j.robotstxt.RobotstxtServer;
import org.apache.http.impl.client.BasicCookieStore;

public class ImageCrawlController {
    public static void main(String[] args) throws Exception {
        String crawlStorageFolder = "/tmp/crawler4j/image-storage";
        int numberOfCrawlers = 5;

        CrawlConfig config = new CrawlConfig();
        config.setCrawlStorageFolder(crawlStorageFolder);
        config.setIncludeBinaryContentInCrawling(true); // Required to download images

        // Perform login and get authenticated cookies
        String loginUrl = "https://your-target-site.com/login";
        String username = "your-actual-username";
        String password = "your-actual-password";
        BasicCookieStore cookieStore = LoginUtils.performLogin(loginUrl, username, password);

        // Override PageFetcher to use our authenticated cookie store
        PageFetcher pageFetcher = new PageFetcher(config) {
            @Override
            protected org.apache.http.client.HttpClient createHttpClient() {
                return HttpClients.custom()
                    .setDefaultCookieStore(cookieStore)
                    .build();
            }
        };

        RobotstxtConfig robotstxtConfig = new RobotstxtConfig();
        RobotstxtServer robotstxtServer = new RobotstxtServer(robotstxtConfig, pageFetcher);

        CrawlController controller = new CrawlController(config, pageFetcher, robotstxtServer);

        // Add the protected page URL you want to crawl images from
        controller.addSeed("https://your-target-site.com/protected-image-gallery");

        // Start the crawler
        controller.start(ImageCrawler.class, numberOfCrawlers);
    }
}

Keep using the ImageCrawler class from the crawler4J example—it will automatically inherit the authenticated session, allowing it to access protected pages and download images.

Quick Tips:

• Always check the target site’s robots.txt and terms of service before crawling.
• If sessions time out, add logic to re-authenticate and refresh cookies mid-crawl.
• Match all request headers (like User-Agent) to your browser’s request to avoid being blocked.

内容的提问来源于stack exchange，提问作者José R. Sánchez