如何使用Java获取网站HTML文件并提取指定标签间数据及学生表格详情

Hey there! Let's tackle your two Java-related web scraping tasks—they’re super practical for your parent portal desktop client project. I’ll walk you through each with clear, actionable code examples.

The easiest way to do this in Java is using a combination of an HTTP client to grab the HTML and an HTML parser to extract the data. Jsoup is my go-to for parsing—it’s intuitive and handles messy HTML way better than native DOM APIs.

Step 1: Set Up Dependencies #

If you’re using Maven, add this to your pom.xml to include Jsoup:

<dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.17.2</version> <!-- Use the latest stable version -->
</dependency>

Step 2: Fetch HTML and Extract Data #

Here’s a simple example that fetches a webpage, then extracts text from specific tags:

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;

public class HtmlScraper {
    public static void main(String[] args) {
        String url = "https://your-target-website.com"; // Replace with your URL
        
        try {
            // Fetch the full HTML document
            Document doc = Jsoup.connect(url).get();
            
            // Example 1: Extract all text inside <h2> tags
            Elements subheadings = doc.select("h2");
            for (Element heading : subheadings) {
                System.out.println("Subheading: " + heading.text());
            }
            
            // Example 2: Extract text from a specific div with ID "student-info"
            Element infoDiv = doc.selectFirst("div#student-info");
            if (infoDiv != null) {
                System.out.println("Student Info: " + infoDiv.text());
            }
            
        } catch (IOException e) {
            e.printStackTrace();
            System.err.println("Failed to fetch or parse the webpage.");
        }
    }
}

The select() method uses CSS selector syntax—super flexible for targeting tags, classes, IDs, or even attributes like [data-id="123"].

This adds a critical layer: maintaining a session (you need to stay logged in after submitting credentials). Jsoup handles this automatically by storing cookies between requests.

Step 1: Simulate Login & Maintain Session #

First, inspect your school’s login form with your browser’s dev tools to find:

• The POST URL the form submits to
• The exact names of the username/password fields (e.g., parent-email instead of username)

Here’s the code to handle login and fetch the student table:

import org.jsoup.Connection;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;
import java.util.Map;

public class ParentPortalClient {
    public static void main(String[] args) {
        String loginUrl = "https://your-school-portal.com/login-submit"; // Form's POST URL
        String detailsUrl = "https://your-school-portal.com/student-table"; // Page with the table
        String username = "your-parent-username";
        String password = "your-parent-password";
        
        try {
            // 1. Send login request to establish a session
            Connection.Response loginResponse = Jsoup.connect(loginUrl)
                    .data("parent-email", username) // Replace with actual field name
                    .data("parent-pass", password) // Replace with actual field name
                    .method(Connection.Method.POST)
                    .execute();
            
            // Save session cookies to stay logged in
            Map<String, String> sessionCookies = loginResponse.cookies();
            
            // 2. Fetch the student details page using the session
            Document detailsDoc = Jsoup.connect(detailsUrl)
                    .cookies(sessionCookies)
                    .get();
            
            // 3. Parse the student table
            Element studentTable = detailsDoc.selectFirst("table#student-details"); // Target your table
            if (studentTable != null) {
                // Skip header row (start at index 1) and iterate through data rows
                Elements rows = studentTable.select("tr");
                for (int i = 1; i < rows.size(); i++) {
                    Element row = rows.get(i);
                    Elements cells = row.select("td");
                    
                    // Extract data based on your table's column order
                    String studentName = cells.get(0).text();
                    String studentID = cells.get(1).text();
                    String gradeLevel = cells.get(2).text();
                    String attendance = cells.get(3).text();
                    
                    System.out.printf("Name: %s | ID: %s | Grade: %s | Attendance: %s%n", 
                                      studentName, studentID, gradeLevel, attendance);
                }
            } else {
                System.err.println("Could not locate the student details table.");
            }
            
        } catch (IOException e) {
            e.printStackTrace();
            System.err.println("Login failed or unable to fetch student data.");
        }
    }
}

Key Tips for This Project: #

• Handle CSRF Tokens: If the login form has a hidden _token field, first fetch the login page, extract the token value, and include it in your POST data.
• Wrap in a UI: For the desktop client, use JavaFX or Swing to add a login form, a "Fetch Data" button, and a table to display results—just call the scraping logic when the button is clicked.
• Error Handling: Add checks for invalid credentials, network issues, or changes to the portal’s HTML structure (schools sometimes update their portals!).

内容的提问来源于stack exchange，提问作者BBloggsbott