用PyMongo执行20万+记录集合聚合查询时遇报错，如何解决？

Hey there, let's sort out that OperationFailure error you're facing when aggregating your large collection. The root cause here is that starting with MongoDB 3.6, the aggregate command requires the cursor option unless you're using the explain argument—your current db.command() call is missing this, which triggers the error.

Here are two straightforward fixes, with the second being the more recommended approach for most cases:

方法1：手动给db.command()添加cursor参数 #

If you want to stick with using the command() method explicitly, just add the cursor parameter to your call. You can even set a batchSize to control how many documents are fetched per round trip, which is great for large datasets to avoid memory overload:

pipeline = [...]  # 你的聚合管道定义

# 调用command时加入cursor选项
result = db.command(
    'aggregate',
    'statCollection',
    pipeline=pipeline,
    cursor={'batchSize': 1000}  # 按需调整批次大小，比如1000或5000
)

# 从返回结果中获取cursor并遍历数据
agg_cursor = result['cursor']
# 先处理第一批数据
for doc in agg_cursor['firstBatch']:
    # 在这里处理每条记录
    print(doc)
# 如果有更多批次，PyMongo的Cursor其实会自动处理后续的getMore请求，但如果需要手动处理也可以调用对应的方法

方法2：使用PyMongo的Collection.aggregate()方法（推荐） #

This is the cleaner and more Pythonic way to handle aggregations in PyMongo. The aggregate() method automatically handles the cursor option for you, and returns an iterable Cursor object that fetches data in batches—perfect for your 200k+ record collection.

from pymongo import MongoClient

# 连接数据库
client = MongoClient()
db = client.your_database_name  # 替换成你的数据库名
collection = db.statCollection

pipeline = [...]  # 你的聚合管道定义

# 执行聚合，推荐加上allowDiskUse=True处理大数据量
agg_cursor = collection.aggregate(
    pipeline,
    allowDiskUse=True  # 当聚合操作需要处理大量数据时，允许MongoDB使用磁盘存储中间结果
)

# 遍历结果，逐批获取数据，不会一次性加载所有记录到内存
for doc in agg_cursor:
    # 处理每条数据的逻辑
    print(doc)

额外小贴士 #

• For large datasets like yours, always include allowDiskUse=True—this prevents MongoDB from hitting memory limits during the aggregation process by letting it use temporary disk storage.
• Using the Cursor object returned by aggregate() is memory-efficient because it only fetches batches of data as you iterate, instead of loading all 200k+ records into your local memory at once.

内容的提问来源于stack exchange，提问作者Lasith Niroshan