如何从请求外部调用Request函数并正确初始化爬虫相关变量？

Hey there! Let's tackle your two crawler issues one by one—they're common pitfalls, but easy to fix with the right approach.

问题1：外部调用Request函数 #

First off, making external requests depends on whether you're using a third-party library like requests or a custom Request function you built. Here's how to ensure it works smoothly:

• If using a third-party library (e.g., requests):
  Make sure you've installed the library first (pip install requests), then import it and call its methods directly in your crawler code. For example:

  import requests
  
  # 外部调用请求函数
  def fetch_page(url):
      # 可以添加headers、timeout等参数适配目标网站
      response = requests.get(url, headers={"User-Agent": "Educational Crawler/1.0"})
      response.raise_for_status()  # 捕获请求错误
      return response
  

• If using a custom Request function:
  Ensure the function is accessible in your current code scope. If it's defined in another module (like utils.py), import it explicitly:

  from utils import Request  # 假设你的Request函数在utils.py中
  
  def crawl():
      url = "https://your-education-site.com"
      response = Request(url, method="GET")
      # 传递给回调函数处理
      request_callback(response)
  

问题2：让PDF_LISTS和PDF_LINKS适配75-78行的循环（全局变量失效的解决） #

The global variable issue usually happens because Python treats assignments inside functions as local variables by default—even if you have a global variable with the same name. Here are two solid fixes:

方案1：用global关键字明确声明 #

If you want to stick with global variables, add a global statement at the start of your callback function to tell Python you're referencing the global variables, not creating new local ones:

# 全局初始化
PDF_LISTS = []
PDF_LINKS = []

def request_callback(response):
    # 声明要操作全局变量
    global PDF_LISTS, PDF_LINKS
    # 解析响应，提取PDF数据（这里替换成你的解析逻辑）
    parsed_pdfs = extract_pdf_info(response.text)
    # 更新全局列表
    PDF_LISTS.extend(parsed_pdfs)
    PDF_LINKS.extend([pdf["url"] for pdf in parsed_pdfs])

# 75-78行的循环示例（现在能正常遍历全局变量了）
for pdf_link in PDF_LINKS:
    print(f"Downloading PDF: {pdf_link}")
    # 你的PDF处理逻辑

方案2：封装为类属性（更推荐，避免全局变量混乱） #

For better code structure (especially for larger crawlers), wrap your crawler logic in a class. This way, you can use instance attributes instead of global variables, which avoids scope issues entirely:

class EducationalPDFCrawler:
    def __init__(self):
        # 初始化实例属性，替代全局变量
        self.pdf_lists = []
        self.pdf_links = []

    def request_callback(self, response):
        # 直接访问实例属性，无需全局声明
        parsed_pdfs = extract_pdf_info(response.text)
        self.pdf_lists.extend(parsed_pdfs)
        self.pdf_links.extend([pdf["url"] for pdf in parsed_pdfs])

    def run_crawl(self):
        # 外部调用请求函数
        import requests
        url = "https://your-education-site.com"
        response = requests.get(url)
        self.request_callback(response)
        
        # 75-78行的循环，直接使用实例属性
        for pdf_link in self.pdf_links:
            print(f"Processing PDF: {pdf_link}")
            # 你的PDF处理代码

# 使用爬虫
crawler = EducationalPDFCrawler()
crawler.run_crawl()

为什么你的全局变量没生效？ #

If you tried assigning PDF_LISTS = [] inside the callback function without global, Python creates a new local variable with that name instead of modifying the global one. Using global tells Python to use the existing global variable, while mutable objects (like lists) let you modify their contents (e.g., append, extend) without needing a global statement—though declaring global still helps if you need to reassign the entire list.

内容的提问来源于stack exchange，提问作者jewstin