如何从维基百科下载埃及阿拉伯语方言文章？

Hey Shereen, I totally get that you need to pull a bunch (or even all) of Egyptian Arabic dialect Wikipedia pages for your research—and since you're new to this, let me break down the most reliable, straightforward methods that are both respectful of Wikipedia's rules and perfect for academic work.

维基百科 regularly generates full database dumps specifically for research, mirroring, and bulk content access. This is the most efficient and compliant way to get large volumes of pages.

• Step 1: Locate the right dump file
  The Egyptian Arabic Wikipedia (arz.wikipedia.org) has dedicated dump packages. You'll want the latest pages-articles dump—it only includes main content pages (no talk pages, edit histories, or irrelevant metadata).
• Step 2: Filter for your target pages (if you don't need everything)
  If you only want pages related to Egyptian Arabic dialect, you can use command-line tools or Python libraries to filter the dump. For example, use grep to pull pages containing the dialect's Arabic keyword:

  grep -n "اللهجة المصرية" your-extracted-dump.xml > filtered-dialect-pages.txt
  

• Step 3: Extract readable content
  Dumps are in XML format, so use the wikiextractor tool to convert them into clean text/JSON files (perfect for analysis):

  python -m wikiextractor your-dump-file.xml --output extracted-content --json
  

  This will save each page as a separate JSON file with a clear title and plaintext content.

If dumps are too large, or you need real-time updates, a simple Python script works—but always follow Wikipedia's robots.txt rules to avoid overwhelming their servers.
Here's a basic, respectful example using requests and BeautifulSoup:

import requests
from bs4 import BeautifulSoup
import time

# Base URL for Egyptian Arabic Wikipedia
BASE_URL = "https://arz.wikipedia.org/wiki/"
# List of page titles you want (grab these from category pages if needed)
TARGET_TITLES = ["اللهجة_المصرية", "مصطلحات_لهجة_مصرية"]

for title in TARGET_TITLES:
    page_url = BASE_URL + title
    try:
        # Identify yourself to Wikipedia (replace with your research email)
        headers = {"User-Agent": "EgyptianDialectResearchBot/1.0 (your-research-email@example.com)"}
        response = requests.get(page_url, headers=headers)
        response.raise_for_status()
        
        # Parse and extract the main content
        soup = BeautifulSoup(response.text, "html.parser")
        main_content = soup.find("div", class_="mw-content-text").get_text(strip=True)
        
        # Save to a UTF-8 encoded file (critical for Arabic text)
        with open(f"{title}.txt", "w", encoding="utf-8") as file:
            file.write(main_content)
        
        # Wait 2 seconds between requests to respect rate limits
        time.sleep(2)
        print(f"Saved page: {title}")
    except Exception as e:
        print(f"Failed to fetch {title}: {str(e)}")

To scale this, first scrape category pages (like Category:اللهجة_المصرية) to get all related page titles automatically.

• Copyright Compliance: Wikipedia content uses the CC BY-SA 4.0 license. Always cite the original source in your research, and share modified content under the same license.
• Server Respect: Dumps are always preferred over scraping—they reduce load on Wikipedia's servers. If you scrape, stick to slow request intervals (1-2 seconds minimum) and avoid parallel requests.
• Encoding: Egyptian Arabic uses UTF-8, so always specify encoding="utf-8" when saving files to prevent garbled text.

内容的提问来源于stack exchange，提问作者Shereen