如何让基于BeautifulSoup的异步教区爬虫代码在普通Colab运行？

1. Fix the ModuleNotFoundError for httpx #

The error occurs because Google Colab doesn’t include httpx or beautifulsoup4 by default. Install them first using pip—no Colab Pro account required:

!pip install httpx beautifulsoup4

2. Simplified Working Async Scraper Code #

Here’s a streamlined version of your scraper that runs reliably in regular Colab. It extracts diocese name, bishop details, and contact info from your target page:

import asyncio
import httpx
from bs4 import BeautifulSoup

async def scrape_diocese(url):
    async with httpx.AsyncClient() as client:
        # Fetch the page
        response = await client.get(url)
        response.raise_for_status()  # Throw error for HTTP issues
        
        soup = BeautifulSoup(response.text, 'html.parser')
        
        # Extract diocese name
        diocese_name = soup.find('h1').text.strip() if soup.find('h1') else 'N/A'
        
        # Extract bishop data
        bishop_list = []
        bishop_table = soup.select_one('table[summary="Bishop"]')
        if bishop_table:
            for row in bishop_table.find_all('tr')[1:]:  # Skip header row
                cols = row.find_all('td')
                if len(cols) >=3:
                    bishop_list.append({
                        "name": cols[0].text.strip(),
                        "title": cols[1].text.strip(),
                        "term": cols[2].text.strip()
                    })
        
        # Extract contact information
        contact_info = {}
        contact_section = soup.find('div', string=lambda text: 'Contact Information' in text)
        if contact_section:
            for line in contact_section.find_next('div').stripped_strings:
                if 'Phone:' in line:
                    contact_info['phone'] = line.replace('Phone:', '').strip()
                elif 'Fax:' in line:
                    contact_info['fax'] = line.replace('Fax:', '').strip()
                elif 'Email:' in line:
                    contact_info['email'] = line.replace('Email:', '').strip()
                elif 'Website:' in line:
                    contact_info['website'] = line.replace('Website:', '').strip()
        
        return {
            "diocese_name": diocese_name,
            "bishops": bishop_list,
            "contact": contact_info
        }

# Execute the scraper
async def main():
    target_url = "https://www.catholic-hierarchy.org/diocese/laa.html"
    result = await scrape_diocese(target_url)
    
    # Print formatted results
    print(f"Diocese: {result['diocese_name']}\n")
    print("Bishops:")
    for bishop in result['bishops']:
        print(f"- {bishop['name']} | {bishop['title']} | Term: {bishop['term']}")
    print("\nContact Details:")
    for key, val in result['contact'].items():
        print(f"{key.capitalize()}: {val}")

# Run in Colab's async environment
await main()

3. Colab Package & Environment FAQs #

• Package Installation: Always use !pip install [package-name] to install missing libraries in Colab. This works for almost all Python packages, and regular accounts have full access to this functionality.
• Conda Environment: Colab doesn’t include Conda by default. While you can install it manually (via !pip install condacolab followed by setup), it’s unnecessary for this scraper. Pip is simpler and sufficient for managing dependencies here.
• Colab Pro Requirement: You do not need Colab Pro. Regular accounts support async code execution, package installation, and web scraping without restrictions. The initial error was purely due to missing packages, not account limitations.

内容的提问来源于stack exchange，提问作者zero