如何将Django爬虫结果存入Model对应MySQL表并查看？

我在Django项目中创建了对应MySQL books表的BookList模型，同时编写了爬虫函数用于爬取书籍数据。目前已在函数文件中导入BookList模型，希望将爬虫结果存入MySQL表，并且能查看函数执行结果，该如何操作？

现有代码 #

爬虫函数代码 #

from ast import Str
from base64 import decode
import requests
from bs4 import BeautifulSoup
import pandas as pd
import csv
from time import sleep
from random import randint
import numpy as np
from bookList.models import BookList

headers = dict()
headers[
    "User-Agent"
] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"

def sendBooksToMysql():
      bkmArt()
def bkmArt():
      pages = range(1, 3, 1)
      for page in pages:
            url = "https://www.bkmkitap.com/sanat"
            results = requests.get(url, headers=headers)
            soup = BeautifulSoup(results.content, "html.parser")
            book_div = soup.find_all("div", class_="col col-12 drop-down hover lightBg")
            sleep(randint(2, 10))
            for bookSection in book_div:
                  img_url = bookSection.find("img", class_="lazy stImage").get('data-src')
                  name = bookSection.find("a",class_="fl col-12 text-description detailLink").get('title')
                  author = bookSection.find("a", class_="fl col-12 text-title").get('title').replace('Model:', '')
                  publisher = bookSection.find("a", class_="col col-12 text-title mt").get('title').replace(name, '')
                  price = bookSection.find("div", class_="col col-12 currentPrice").text.replace('\n', '').replace('TL', ' TL')
                  b_link = bookSection.find("a").get('href')
                  
                  bookList = BookList.objects.using("bookList").update_or_create(
                  site_name="Bkmkitap",
                  site_id="1",
                  image=img_url,
                  #book=name,
                  #author=author,
                  publisher=publisher,
                  price=price,
                  link=b_link,
                  category_name="Sanat",
                  category_id="1",
                  defaults={             
                        "site_name":"Bkmkitap",
                        "site_id":"1",
                        "image":img_url,
                        "book":name,
                        "author":author,
                        "publisher":publisher,
                        "price":price,
                        "link":b_link,
                        "category_name":"Sanat",
                        "category_id":"1",
                  }
            )

Model代码（models.py） #

from django.db import models

# Create your models here.
class BookList(models.Model):
    site_name = models.TextField()
    site_id = models.TextField()
    image = models.TextField()
    book = models.TextField()
    author = models.TextField()
    publisher = models.TextField()
    price = models.TextField()
    link = models.TextField()
    category_name = models.TextField()
    category_id = models.TextField()
    class Meta:                                                                                                                                                        
        managed = True
        db_table = 'books'

一、修正爬虫代码问题 #

1. 修复分页逻辑：当前循环了2页，但请求URL未携带页码参数，每次都爬取第一页。需修改URL为带分页参数的格式：

   url = f"https://www.bkmkitap.com/sanat?page={page}"
   

2. 修正update_or_create参数：该方法需用唯一标识字段判断数据是否存在（比如书籍链接link），否则会导致重复创建或错误更新：

   bookList, created = BookList.objects.using("bookList").update_or_create(
       link=b_link,  # 用唯一链接作为判断条件
       defaults={             
           "site_name":"Bkmkitap",
           "site_id":"1",
           "image":img_url,
           "book":name,
           "author":author,
           "publisher":publisher,
           "price":price,
           "category_name":"Sanat",
           "category_id":"1",
       }
   )
   

   其中created为布尔值，True表示新增数据，False表示更新数据，可用于打印执行日志。
3. 添加异常处理：避免爬虫过程中因元素缺失、请求失败导致程序崩溃：

   for bookSection in book_div:
       try:
           # 原有爬取逻辑代码
           img_url = bookSection.find("img", class_="lazy stImage").get('data-src')
           name = bookSection.find("a",class_="fl col-12 text-description detailLink").get('title')
           # ...其余字段爬取代码...
   
           # 执行update_or_create
           bookList, created = BookList.objects.using("bookList").update_or_create(
               link=b_link,
               defaults={...}
           )
           # 打印执行日志
           print(f"{'新增' if created else '更新'}书籍：{name}")
       except Exception as e:
           print(f"处理书籍出错：{str(e)}")
           continue
   

二、运行爬虫函数 #

方式1：Django Shell运行 #

在项目根目录终端执行：

python manage.py shell

进入shell后导入函数并执行：

from 你的函数文件名 import sendBooksToMysql
sendBooksToMysql()

执行过程中会打印新增/更新日志，结束后即可查看数据库。

方式2：自定义Django命令运行 #

1. 在app目录下创建management/commands文件夹（需添加__init__.py文件），新建crawl_books.py：

   from django.core.management.base import BaseCommand
   from 你的函数文件名 import sendBooksToMysql
   
   class Command(BaseCommand):
       help = '爬取书籍数据并存入MySQL'
   
       def handle(self, *args, **options):
           self.stdout.write("开始爬取书籍数据...")
           sendBooksToMysql()
           self.stdout.write(self.style.SUCCESS("爬取完成！"))
   

2. 终端执行命令：

   python manage.py crawl_books
   

三、查看数据库结果 #

方式1：Django Admin后台查看 #

1. 在admin.py中注册模型：

   from django.contrib import admin
   from .models import BookList
   
   admin.site.register(BookList)
   

2. 创建超级用户（若未创建）：

   python manage.py createsuperuser
   

3. 启动服务并访问后台：

   python manage.py runserver
   

   打开http://127.0.0.1:8000/admin/，登录后即可查看BookList中的所有数据。

方式2：直接查询MySQL数据库 #

用MySQL客户端（如Navicat、MySQL Workbench）连接数据库，执行SQL查询：

SELECT * FROM books;

方式3：Django Shell查询 #

进入shell后执行：

from bookList.models import BookList
# 查询总数量
print(f"总书籍数：{BookList.objects.count()}")
# 查看前5条数据
for book in BookList.objects.all()[:5]:
    print(f"书名：{book.book}，作者：{book.author}")

内容的提问来源于stack exchange，提问作者océane