基于PHP和Simple HTML DOM实现带表单输入的网页搜索抓取

Got it, let's walk through how to hook up your existing web scraper to a user-facing form, so when someone types in a search term and hits submit, you pull results from the Colby Digital Commons Special Collections site. Here's a complete, actionable implementation:

First, create a simple HTML form where users can enter their search terms. We'll use a POST request to send the input to your backend (avoids exposing the search term in the URL):

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Colby Special Collections Search</title>
    <style>
        body { max-width: 800px; margin: 2rem auto; padding: 0 1rem; font-family: Arial, sans-serif; }
        .result-item { margin: 1.5rem 0; padding-bottom: 1rem; border-bottom: 1px solid #eee; }
    </style>
</head>
<body>
    <h1>Search Colby Special Collections</h1>
    <form method="POST" action="/run-scrape">
        <input type="text" name="searchString" placeholder="e.g., 'civil war letters'" required>
        <button type="submit">Get Results</button>
    </form>

    {% if error %}
        <p style="color: red;">{{ error }}</p>
    {% endif %}

    {% if results %}
        <h2>Found {{ results|length }} Results:</h2>
        <div class="results-list">
            {% for item in results %}
                <div class="result-item">
                    <h3><a href="{{ item.url }}" target="_blank">{{ item.title }}</a></h3>
                    <p>{{ item.description }}</p>
                </div>
            {% endfor %}
        </div>
    {% endif %}
</body>
</html>

We'll use Python + Flask for this example (super common for web scraping projects), but the same logic applies to Node.js/Express or other frameworks.

First, install dependencies:

pip install flask requests beautifulsoup4

Then create your Flask app, which will:

• Serve the form page
• Accept the search term from the form
• Pass it to your modified scraper function
• Return the results to render on the page

from flask import Flask, render_template, request
import requests
from bs4 import BeautifulSoup

app = Flask(__name__)

# Modified scraper function that accepts a search term as input
def scrape_colby_collections(search_term):
    # Target site's search URL (we found this by testing a search on their site)
    search_url = "https://digitalcommons.colby.edu/special_collections/search"
    # The site uses 'q' as the search parameter (check the URL after searching manually)
    params = {"q": search_term}

    try:
        # Send request to the site's search endpoint
        response = requests.get(search_url, params=params)
        response.raise_for_status()  # Catch HTTP errors (404, 500, etc.)
        soup = BeautifulSoup(response.text, "html.parser")

        # Parse results (update these selectors to match the site's actual HTML!)
        # Inspect the search results page to find the right classes/tags
        results = []
        for result_card in soup.select(".artifact-title"):
            title_link = result_card.find("a")
            if not title_link:
                continue
            
            title = title_link.get_text(strip=True)
            url = f"https://digitalcommons.colby.edu{title_link['href']}"
            # Grab the description if available
            description = result_card.find_next_sibling(".abstract").get_text(strip=True) if result_card.find_next_sibling(".abstract") else "No description provided"
            
            results.append({"title": title, "url": url, "description": description})
        
        return results
    except Exception as e:
        print(f"Scraping error: {str(e)}")
        return []

# Serve the form on the home page
@app.route("/", methods=["GET"])
def home():
    return render_template("index.html")

# Handle form submission and run the scraper
@app.route("/run-scrape", methods=["POST"])
def run_scrape():
    search_string = request.form.get("searchString")
    
    # Validate input
    if not search_string.strip():
        return render_template("index.html", error="Please enter a valid search term.")
    
    # Run the scraper with the user's input
    results = scrape_colby_collections(search_string)
    return render_template("index.html", results=results)

if __name__ == "__main__":
    app.run(debug=True)

• Update Selectors: The soup.select and find calls are examples—you need to inspect the actual search results page on digitalcommons.colby.edu to find the correct HTML classes/tags for result titles and descriptions. Right-click a result and select "Inspect" to get the right selectors.
• Respect Site Rules: Check the site's robots.txt to make sure scraping is allowed. Add delays between requests if needed to avoid getting blocked.
• Error Handling: The example includes basic error handling, but you can expand it to handle things like network timeouts or empty result sets more gracefully.
• Alternative Stack: If you prefer JavaScript, you can use Express.js with axios (for requests) and cheerio (for parsing HTML)—the core logic (accept form input, fetch search results, parse, return) is identical.

内容的提问来源于stack exchange，提问作者Mike Jandreau