如何在Django的views.py中集成Scrapy邮件爬虫并实现前端触发？

1. 改造Scrapy爬虫，支持动态传入URL #

修改你的EmailExtractor爬虫，让它可以接收外部传入的目标URL，替代固定的start_urls：

# scrapy_project/spiders/email_spider.py
import scrapy
from scrapy.spiders import Spider

class EmailExtractorSpider(Spider):
    name = 'email_extractor'
    allowed_domains = []  # 留空允许爬取任意域名，按需调整

    def __init__(self, start_url=None, *args, **kwargs):
        super().__init__(*args, **kwargs)
        if not start_url:
            raise ValueError("必须传入start_url参数")
        self.start_urls = [start_url]
        self.emails = []  # 内存存储爬取到的邮箱

    def parse(self, response):
        # 邮箱提取正则，可根据目标网站结构调整
        email_pattern = r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}'
        extracted_emails = response.xpath('//text()').re(email_pattern)
        # 去重后存入列表
        unique_emails = list(set(extracted_emails))
        self.emails.extend(unique_emails)

        # 如需跟进页面内链接爬取，取消下方注释
        # for href in response.css('a::attr(href)').getall():
        #     yield response.follow(href, self.parse)

2. 编写Django视图，实现触发爬虫+返回文件 #

在views.py中用Scrapy的API同步调用爬虫，收集结果后生成文本文件返回给前端：

# django_project/app/views.py
from django.http import HttpResponse
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
from your_scrapy_project.spiders.email_spider import EmailExtractorSpider
import io
import os

# 确保Django能找到Scrapy配置
os.environ.setdefault('SCRAPY_SETTINGS_MODULE', 'your_scrapy_project.settings')

def extract_emails(request):
    if request.method == 'POST':
        target_url = request.POST.get('url')
        if not target_url:
            return HttpResponse("请输入目标URL", status=400)

        # 初始化Scrapy进程
        process = CrawlerProcess(get_project_settings())
        # 创建爬虫实例并传入目标URL
        spider = EmailExtractorSpider(start_url=target_url)
        # 启动爬虫，阻塞直到爬取完成
        process.crawl(spider)
        process.start()

        # 生成邮箱文本内容
        email_content = '\n'.join(spider.emails) if spider.emails else "未提取到任何邮箱地址"
        # 构造下载响应
        response = HttpResponse(io.BytesIO(email_content.encode('utf-8')), content_type='text/plain')
        response['Content-Disposition'] = f'attachment; filename="extracted_emails.txt"'
        return response

    # GET请求返回表单页面
    return HttpResponse('''
        <form method="post">
            {% csrf_token %}
            <input type="url" name="url" placeholder="输入目标网站URL" required>
            <button type="submit">提取邮箱</button>
        </form>
    ''')

3. 关键优化与注意事项 #

• 异步处理避免阻塞：如果爬取大型网站，同步调用会导致请求超时，可改用CrawlerRunner结合Django异步视图（Python 3.7+、Django 3.1+）：

  from django.http import HttpResponse
  from scrapy.crawler import CrawlerRunner
  from scrapy.utils.log import configure_logging
  from twisted.internet import reactor
  import asyncio
  import io
  
  configure_logging()
  runner = CrawlerRunner(get_project_settings())
  
  async def extract_emails_async(request):
      if request.method == 'POST':
          target_url = request.POST.get('url')
          if not target_url:
              return HttpResponse("请输入目标URL", status=400)
  
          spider = EmailExtractorSpider(start_url=target_url)
          # 异步运行爬虫
          await runner.crawl(spider)
          reactor.stop()
          await asyncio.sleep(0.5)  # 等待reactor完全停止
  
          email_content = '\n'.join(spider.emails) if spider.emails else "未提取到任何邮箱地址"
          response = HttpResponse(io.BytesIO(email_content.encode('utf-8')), content_type='text/plain')
          response['Content-Disposition'] = f'attachment; filename="extracted_emails.txt"'
          return response
  
      # 返回表单
      return HttpResponse('''
          <form method="post">
              {% csrf_token %}
              <input type="url" name="url" placeholder="输入目标网站URL" required>
              <button type="submit">提取邮箱</button>
          </form>
      ''')
  

• CSRF保护：前端表单必须包含{% csrf_token %}，否则Django会拒绝POST请求。
• 爬取范围控制：合理设置allowed_domains，避免爬取无关网站浪费资源。

4. 配置URL路由 #

在Django项目的urls.py中添加视图路由：

from django.urls import path
from .views import extract_emails  # 或extract_emails_async

urlpatterns = [
    path('extract-emails/', extract_emails, name='extract_emails'),
]

内容的提问来源于stack exchange，提问作者Faizan