咒术回战漫画章节URL异常重定向导致爬虫脚本失效求解决方案
问题解决方法
- 修正URL拼写错误
你当前generate_comics_urls函数中的URL模板存在拼写错误,章节前缀的jujustu应为正确的jujutsu,错误拼写的URL会直接触发站点的重定向逻辑跳转到首页。 - 新增请求头配置
该站点存在基础反爬校验,会拦截无正常浏览器标识的请求,请求时需要携带包含User-Agent的请求头,模拟浏览器访问行为。 - 优化重定向处理逻辑
可关闭requests默认的自动重定向开关,校验响应状态码,如果出现3xx重定向可增加重试机制,避免单次请求失败直接抛出异常。 - 增加请求间隔
高频访问容易触发站点的频率限制,可在每次请求后增加1-2秒的随机等待时间,降低被封IP的概率。
修正后代码示例
import requests import os import shutil import time import random from bs4 import BeautifulSoup from tqdm import tqdm class CustomError(Exception): pass # 全局请求头配置 HEADERS = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36", "Referer": "https://jujustukaisen.com/" } # 最大重试次数 MAX_RETRY = 3 def generate_comics_urls(): comics_sites_url = [] # 修正URL拼写错误 a = "https://jujustukaisen.com/manga/jujutsu-kaisen-chapter-{0}/" for i in range(1,154): comics_sites_url.append(a.format(i)) return comics_sites_url comics_urls = generate_comics_urls() for chapter, comics_url in enumerate(tqdm(comics_urls), start=1): if not os.path.exists(str(chapter)): os.mkdir(str(chapter)) # 增加重试逻辑 retry_count = 0 comics_page = None while retry_count < MAX_RETRY: try: comics_page = requests.get(comics_url, headers=HEADERS, timeout=10) # 校验是否被重定向到首页 if comics_page.url == "https://jujustukaisen.com/": retry_count +=1 time.sleep(random.uniform(1,2)) continue if comics_page.status_code == 200: break except Exception as e: retry_count +=1 time.sleep(random.uniform(1,2)) if not comics_page or comics_page.status_code != 200 or comics_page.url == "https://jujustukaisen.com/": raise CustomError(f"INVALID URL - {comics_url}") soup = BeautifulSoup(comics_page.content, "html.parser") picture_order = 1 while True: picture_url = soup.find("img",alt="Jujutsu Kaisen, Chapter {0} image {1:0>3}".format(chapter,picture_order)) if picture_url: picture_url = picture_url["src"] # 图片请求也增加重试和头配置 pic_retry = 0 picture = None while pic_retry < MAX_RETRY: try: picture = requests.get(picture_url, headers=HEADERS, stream=True, timeout=10) if picture.status_code == 200: break except: pic_retry +=1 time.sleep(random.uniform(0.5,1)) if not picture or picture.status_code !=200: raise CustomError(f"IVALID URL - {picture_url} - {comics_url}") picture.raw.decode_content = True fh = open("{0}/{1}.jpg".format(str(chapter),str(picture_order)),"wb") shutil.copyfileobj(picture.raw, fh) fh.close() picture_order += 1 # 图片请求间隔 time.sleep(random.uniform(0.2,0.5)) else: break # 章节请求间隔 time.sleep(random.uniform(1,2))
内容的提问来源于stack exchange,提问作者I need HELP
相关产品推荐
相关产品推荐

