Python菜谱搜索引擎代码优化求助：单查询耗时需降至0.1秒内

Great question! Let's break down exactly why your current code is slow and fix it using only native Python tools—no extra third-party modules required. The main bottlenecks here are repeated O(n) lookups with list.count() and redundant string processing. Here's how to tackle this:

1. Precompute Character Mappings & Optimize Tokenization #

Your countTokens function recreates the translation table every time it's called (for every field of every recipe), which adds unnecessary overhead. We'll precompute this once, and switch from returning a list of tokens to a frequency dictionary (so we can look up token counts in O(1) time instead of O(n)).

First, import the necessary standard library modules (these are native to Python, so no extra installs):

import string
from collections import Counter

Then, predefine the translation table outside your functions:

# Precompute once, not every time countTokens is called
TRANSLATION_TABLE = str.maketrans(string.digits + string.punctuation, " " * len(string.digits + string.punctuation))

Refactor countTokens to return a Counter (or a plain dict) of token frequencies:

def count_tokens(token):
    # Convert to string and lowercase once
    token_str = str(token).lower()
    # Replace digits/punctuation with spaces in one pass
    cleaned = token_str.translate(TRANSLATION_TABLE)
    # Split into tokens and count frequencies (O(n) once, not repeated)
    return Counter(cleaned.split())

Note: If you want to avoid collections.Counter for strict "pure dict" usage, you can build the frequency dict manually—Counter is just a convenient wrapper for this.

2. Optimize the Core Scoring Loop #

The biggest slowdown in normalOrder is calling list.count(q) multiple times per query per recipe. Each count scans the entire list, which adds up fast with large recipe datasets. Instead, we'll compute token frequencies once per field, then look up counts in O(1) time.

We'll also:

• Preprocess query terms to lowercase once (instead of handling case in the loop)
• Use dict.get() for cleaner, faster field checks
• Minimize redundant dictionary key lookups

Here's the refactored normal_order function:

def normal_order(recipes, queries):
    # Preprocess queries to lowercase once (matches our tokenization)
    lower_queries = [q.lower() for q in queries]
    
    for r in recipes:
        # Compute frequency counters for each relevant field (only once per recipe)
        title_counts = count_tokens(r["title"])
        cat_counts = count_tokens(r.get("categories", ""))
        ing_counts = count_tokens(r.get("ingredients", ""))
        dir_counts = count_tokens(r.get("directions", ""))
        
        total_score = 0
        for q in lower_queries:
            # Look up counts in O(1) time using the frequency dicts
            total_score += (8 * title_counts.get(q, 0) +
                            4 * cat_counts.get(q, 0) +
                            2 * ing_counts.get(q, 0) +
                            1 * dir_counts.get(q, 0))
        
        # Add rating if present
        total_score += r.get("rating", 0)
        r["score"] = total_score
    
    return recipes

3. Additional Quick Wins #

If you still need more speed, try these:

• Pre-tokenize all recipes upfront: If your recipe dataset doesn't change often, compute the frequency counters for all fields once when loading the data, not during each query. This cuts tokenization time entirely during search.
• Avoid string conversions where possible: If you know certain fields (like title) are always strings, skip the str(token) call in count_tokens for those fields.
• Use generator expressions or list comprehensions: Where applicable, replace explicit loops with faster native iterators (though the above changes already cover most of this).

Why This Works #

• We reduced repeated O(n) operations (like list.count()) to O(1) lookups using frequency dictionaries.
• Precomputing the translation table eliminates redundant work across every tokenization call.
• Preprocessing queries once avoids repeated lowercase conversions during the scoring loop.

These changes should easily bring your query time down to under 0.1 seconds—even with large recipe datasets.

内容的提问来源于stack exchange，提问作者Maria Isabel Lopez