Flask API构建求助：URL文本提取与智能标签生成方案

问题分析 #

当前API存在两个核心痛点：

• NLTK、TF-IDF仅依赖词频提取关键词，无法生成文本中未直接出现的语义标签（比如从咖喱鸡食谱中生成"Indian Cuisine"这类隐含标签）
• GPT-2的提示设计和生成参数不合理，导致输出标签不符合预期

优化方案 #

1. 优化GPT-2的提示与生成逻辑 #

GPT-2的输出质量高度依赖提示清晰度和参数设置，调整方向如下：

• 明确提示格式：要求模型输出逗号分隔的纯文本标签，避免歧义
• 截断过长文本：适配GPT-2的上下文长度限制（gpt2-medium为1024 tokens），防止核心内容被截断
• 调整生成参数：降低temperature减少随机性，设置合理的max_length确保只生成标签内容

修改后的generate_tags函数：

def generate_tags(text, max_length=150, num_return_sequences=1):
    try:
        # 明确提示格式，截断文本适配模型长度限制
        prompt = f"Generate 3-5 relevant semantic tags (comma-separated, no hashtags) for this text: {text[:1000]}"

        # 为GPT-2添加pad token（默认无，用eos_token替代）
        tokenizer.pad_token = tokenizer.eos_token
        inputs = tokenizer(
            prompt,
            return_tensors='pt',
            truncation=True,
            max_length=1024,
            padding=True
        )

        # 调整生成参数，控制输出稳定性
        output = model.generate(
            **inputs,
            max_length=max_length + inputs['input_ids'].shape[1],
            num_return_sequences=num_return_sequences,
            temperature=0.7,
            num_beams=3,
            no_repeat_ngram_size=2,
            early_stopping=True,
            pad_token_id=tokenizer.eos_token_id
        )

        # 解析输出，提取标签并去重
        decoded_seq = tokenizer.decode(output[0], skip_special_tokens=True).strip()
        tags_part = decoded_seq.split("for this text:")[-1].strip()
        tags = [tag.strip() for tag in tags_part.split(",") if tag.strip()]
        return list(set(tags))
    
    except Exception as e:
        print(f"Error generating tags: {str(e)}")
        return None

2. 改用KeyBERT生成语义标签 #

KeyBERT结合BERT词嵌入与TF-IDF，能生成贴合文本语义的标签，支持1-2词组合的标签生成，无需依赖文本中出现的词汇：

先安装依赖：

pip install keybert

替换generate_tags函数为KeyBERT实现：

from keybert import KeyBERT

# 初始化轻量级KeyBERT模型
kw_model = KeyBERT(model='all-MiniLM-L6-v2')

def generate_tags(text, top_n=5):
    try:
        tags = kw_model.extract_keywords(
            text,
            keyphrase_ngram_range=(1, 2),  # 允许生成1-2词的标签
            stop_words='english',
            top_n=top_n
        )
        # 仅提取标签文本，忽略置信度分数
        return [tag[0] for tag in tags]
    
    except Exception as e:
        print(f"Error generating tags: {str(e)}")
        return None

3. 网页文本预处理优化 #

原代码提取所有<p>标签文本，可能包含广告、导航等冗余内容，优化提取逻辑保留核心正文：

def fetch_text_from_url(url):
    try:
        response = requests.get(url)
        response.raise_for_status()
        soup = BeautifulSoup(response.content, 'html.parser')
        
        # 移除冗余元素
        for element in soup(['script', 'style', 'nav', 'aside', 'footer']):
            element.decompose()
        
        # 优先提取article标签内的正文，无则提取所有p标签
        article = soup.find('article')
        paragraphs = article.find_all('p') if article else soup.find_all('p')
        
        text = ' '.join([para.get_text(strip=True) for para in paragraphs])
        # 清理多余空格
        text = ' '.join(text.split())
        return text if text else None
    except requests.exceptions.RequestException as e:
        print(f"Error fetching content from {url}: {str(e)}")
        return None
    except Exception as e:
        print(f"Error parsing content from {url}: {str(e)}")
        return None

完整优化后代码示例（KeyBERT版本） #

import requests
from bs4 import BeautifulSoup
from keybert import KeyBERT
from flask import Flask, request, jsonify

app = Flask(__name__)

# 初始化KeyBERT模型
kw_model = KeyBERT(model='all-MiniLM-L6-v2')

def fetch_text_from_url(url):
    try:
        response = requests.get(url)
        response.raise_for_status()
        soup = BeautifulSoup(response.content, 'html.parser')
        
        # 移除冗余元素
        for element in soup(['script', 'style', 'nav', 'aside', 'footer']):
            element.decompose()
        
        # 优先提取article内的正文
        article = soup.find('article')
        paragraphs = article.find_all('p') if article else soup.find_all('p')
        
        text = ' '.join([para.get_text(strip=True) for para in paragraphs])
        text = ' '.join(text.split())
        return text if text else None
    except requests.exceptions.RequestException as e:
        print(f"Error fetching content from {url}: {str(e)}")
        return None
    except Exception as e:
        print(f"Error parsing content from {url}: {str(e)}")
        return None

def generate_tags(text, top_n=5):
    try:
        tags = kw_model.extract_keywords(
            text,
            keyphrase_ngram_range=(1, 2),
            stop_words='english',
            top_n=top_n
        )
        return [tag[0] for tag in tags]
    except Exception as e:
        print(f"Error generating tags: {str(e)}")
        return None

@app.route('/generate_tags', methods=['POST'])
def generate_tags_api():
    data = request.get_json()
    url = data.get('url')
    if not url:
        return jsonify({'error': 'URL is required'}), 400

    text = fetch_text_from_url(url)
    if not text:
        return jsonify({'error': 'Failed to fetch content from URL'}), 500
    
    tags = generate_tags(text)
    if tags:
        return jsonify({'tags': tags})
    else:
        return jsonify({'error': 'Failed to generate tags from URL'}), 500

if __name__ == "__main__":
    app.run(port=8000, debug=True)

内容的提问来源于stack exchange，提问作者Noobmaster69