First off, the short answer is yes, WikiCorpus does support Arabic Wikipedia dumps—but there are critical caveats that are almost certainly causing your three-day hang. Let’s break down why your code is stuck, and how to fix it:

1. The Default Tokenizer Isn’t Built for Arabic #

The biggest issue here is that WikiCorpus uses an English/European-language tokenizer out of the box. Arabic has unique linguistic features (like diacritics/tashkeel, elongation/tatweel, and root-based word structure) that the default tool can’t parse efficiently. This leads to glacial processing speeds, or even infinite loops as it struggles to make sense of text it wasn’t designed for.

Fix: Use a Custom Arabic Tokenizer #

You’ll need to replace the default tokenizer with one tailored for Arabic. Libraries like pyarabic or NLTK’s Arabic tools work perfectly. Here’s a practical example:

from gensim.corpora.wikicorpus import WikiCorpus
import pyarabic.araby as araby

def arabic_tokenizer(text):
    # Clean Arabic text: remove diacritics and unnecessary elongation
    cleaned_text = araby.strip_tashkeel(araby.strip_tatweel(text))
    # Tokenize the cleaned content
    tokens = araby.tokenize(cleaned_text)
    # Filter out short noise words (optional but recommended for efficiency)
    tokens = [token for token in tokens if len(token) > 2]
    return tokens

# Initialize WikiCorpus with your custom Arabic tokenizer
wiki = WikiCorpus(self.in_f, tokenizer_func=arabic_tokenizer)

2. Windows Single-Threaded Bottleneck #

The warning you saw (detected Windows; aliasing chunkize to chunkize_serial) is harmless, but it reveals a key limitation: on Windows, WikiCorpus runs in single-threaded mode. Processing a nearly 1GB dump with one thread (especially with a mismatched tokenizer) will take an absurdly long time—three days without progress is totally expected here.

Fix: Switch to a Linux/Mac Environment (If Possible) #

WikiCorpus uses multi-threaded processing on Unix-like systems, which will drastically cut down your processing time. If switching environments isn’t an option, focus on optimizing other parameters (see below) to mitigate the slowdown.

3. Disable Resource-Heavy Default Features #

WikiCorpus has default settings that work great for English but are overkill (and slow) for Arabic:

• lemmatize=True: By default, it tries to lemmatize text using WordNet, which doesn’t support Arabic. This wastes tons of processing power for no benefit.
• Unnecessary namespaces: It may be parsing non-article pages (like talk pages) that add unnecessary load.

Fix: Optimize Initialization Parameters #

Update your WikiCorpus call to disable unused features:

wiki = WikiCorpus(
    self.in_f,
    tokenizer_func=arabic_tokenizer,
    lemmatize=False,  # Disable useless Arabic lemmatization
    filter_namespaces=('0',)  # Only process main namespace (actual articles)
)

4. Verify Dump File Integrity #

It’s also worth checking if your arwiki-20200201-pages-articles.xml.bz2 file is intact. A corrupted dump can cause WikiCorpus to hang mid-parsing. Cross-check the file’s MD5 hash against the one listed on the Wikipedia dump download page to confirm it’s not damaged.

内容的提问来源于stack exchange，提问作者Islam Kh