Flask API请求次数限制及爬虫/抓取工具防范方案咨询

Hey there! Let's break down how to solve this problem—you’re totally right to worry about VPN-powered crawlers bypassing simple IP-based limits, since that’s a super common loophole. Here are some layered, practical approaches to enforce your 3-request limit and block persistent bots effectively, tailored to your Flask + Python setup:

1. Use a Combined User Fingerprint (Not Just IP) #

Instead of relying solely on request.remote_addr, create a unique fingerprint that combines multiple identifiers to make it harder for bots to switch identities:

• IP address
• User Agent string (request.user_agent.string)
• A custom session cookie (we’ll set this for unregistered users)

This way, even if a bot switches IP via VPN, if it keeps the same browser session (or we force a persistent cookie), we can still track it.

2. Implement Persistent Counting & Permanent Blacklisting #

For production, don’t use Flask’s in-memory storage—it’ll reset when your app restarts or scales to multiple instances. Instead, use a persistent store like Redis (fast, ideal for caching/key-value data) to track request counts and blacklist users.

Here’s a working code example with Redis:
First, install the Redis client:

pip install redis

Then update your Flask app:

from flask import Flask, request, jsonify, make_response
import redis
import uuid
from datetime import timedelta

app = Flask(__name__)
# Connect to Redis (configure host/port/password for production)
r = redis.Redis(host='localhost', port=6379, db=0, decode_responses=True)

def get_user_fingerprint():
    # Generate a unique fingerprint for the user
    ip = request.remote_addr
    user_agent = request.user_agent.string
    # Check for our custom session cookie
    session_id = request.cookies.get('api_session_id')
    
    # If no session exists, create a new one
    if not session_id:
        session_id = str(uuid.uuid4())
    
    # Combine identifiers into a single fingerprint
    return f"{session_id}_{ip}_{user_agent}", session_id

@app.route('/api/givedata', methods=['GET'])
def givedata():
    fingerprint, session_id = get_user_fingerprint()
    
    # First, check if the user is already blacklisted
    if r.sismember('api_blacklist', fingerprint):
        return jsonify({'error': '请求次数超限'}), 403
    
    # Get current request count
    count_key = f"req_count:{fingerprint}"
    current_count = int(r.get(count_key)) if r.get(count_key) else 0
    
    # Enforce the 3-request limit
    if current_count >= 3:
        # Add to permanent blacklist
        r.sadd('api_blacklist', fingerprint)
        return jsonify({'error': '请求次数超限'}), 403
    
    # Increment the count (persists until user registers)
    r.incr(count_key)
    
    # Set the session cookie if it doesn't exist (expires in 30 days)
    response = make_response(jsonify({'data': 'Your requested content here'}), 200)
    if not request.cookies.get('api_session_id'):
        response.set_cookie(
            'api_session_id', 
            session_id, 
            max_age=timedelta(days=30).total_seconds(),
            httponly=True  # Prevents client-side JS from accessing the cookie (more secure)
        )
    return response

# Example registration endpoint to lift restrictions
@app.route('/api/register', methods=['POST'])
def register():
    # Handle your registration logic here (validate user input, save to DB, etc.)
    fingerprint, _ = get_user_fingerprint()
    
    # Remove from blacklist and reset request count
    r.srem('api_blacklist', fingerprint)
    r.delete(f"req_count:{fingerprint}")
    
    return jsonify({'message': '注册成功，请求限制已解除'}), 200

if __name__ == '__main__':
    app.run(debug=True)

3. Add Lightweight Bot Mitigation #

To block even more sophisticated bots, add a simple human check before the 3rd request (or on the first request if you want):

• Hidden form field: Add a hidden input in your HTML that bots will fill out, but humans won’t (e.g., <input type="hidden" name="bot_check" value="">). Reject requests where this field isn’t empty.
• Simple math challenge: Have your frontend generate a random math problem (like 2 + 5 = ?), store the answer in a session, and require the user to submit the answer with their AJAX request. The backend validates it before processing.

4. Extra Tips for Production #

• Rate limiting alongside count limits: Even if a user hasn’t hit the 3-request cap, add rate limiting (e.g., 1 request per second) to prevent rapid-fire bot attacks. You can use flask-limiter with Redis for this.
• Log suspicious activity: Track requests with unusual User Agents, frequent IP switches, or failed bot checks—this helps you refine your fingerprinting and blacklisting rules over time.
• Redis persistence: Make sure your Redis instance is configured to persist data (RDB/AOF) so blacklists and counts don’t disappear on restart.

内容的提问来源于stack exchange，提问作者Anonymous