Node.js中Redis scanIterator无限循环问题排查

我尝试从Redis实例每次获取N个键，设置batchSize = 100。但代码陷入无限循环，输出反复显示“got a batch of 719 logs”。测试数据库共有719个键，理论上应调用8次retrieveLogs，其中7次返回100个键，1次返回19个。找不到足够的使用文档，附上完整代码，请问是否存在明显错误？

/* eslint-disable no-await-in-loop */
/* eslint-disable no-restricted-syntax */
import { createClient } from 'redis';

const redisClient = createClient({ url: process.env.REDIS_URL });

redisClient.on('error', (err) => {
    console.log('Redis Client Error', err);
    throw new Error('Redis Client Error');
});

async function retrieveLogs(batchSize) {
    const logs = [];

    for await (const cursor of redisClient.scanIterator({
        MATCH: 'log:*',
        COUNT: batchSize,
    })) {
        const log = await redisClient.get(cursor);
        if (log) {
            logs.push(JSON.parse(log));
        }
        if (cursor === '0') {
            console.log('breaking');
            break;
        }
    }

    return logs;
}

async function main() {
    redisClient.on('error', (err) => {
        console.log('Redis Client Error', err);
        throw new Error('Redis Client Error');
    });

    await redisClient.connect();

    // Set the number of logs to retrieve and delete at a time
    const batchSize = 100;
    // Set a flag to indicate whether there are more logs to process
    let moreLogs = true;

    while (moreLogs) {
        // Retrieve a batch of logs from Redis
        // eslint-disable-next-line no-await-in-loop
        const batch = await retrieveLogs(batchSize);

        if (batch.length === 0) {
            // If there are no logs, set the flag to false to exit the loop
            moreLogs = false;
            break;
        }
        console.log(`got a batch of ${batch.length} logs`);
    }

    await redisClient.disconnect();
}

main();

错误原因分析 #

1. scanIterator使用逻辑错误：scanIterator会自动遍历所有匹配的键直到游标归0，你的retrieveLogs函数会一次性把全部719个键都取出来，而非每次取batchSize个。这导致每次调用retrieveLogs都返回全量数据，batch.length永远不为0，外层while循环无限执行。
2. 游标判断完全错误：for await...of循环里的cursor变量是遍历到的键名，不是SCAN命令返回的游标值。你的键是log:*格式，不可能等于'0'，所以break永远不会触发，直到scanIterator自动遍历完所有键。

修复方案 #

要实现分批获取，需手动控制SCAN的游标流转，而非依赖scanIterator自动遍历全部。修改代码如下：

/* eslint-disable no-await-in-loop */
/* eslint-disable no-restricted-syntax */
import { createClient } from 'redis';

const redisClient = createClient({ url: process.env.REDIS_URL });

redisClient.on('error', (err) => {
    console.log('Redis Client Error', err);
    throw new Error('Redis Client Error');
});

// 修改为接收当前游标，返回批次数据和下一个游标
async function retrieveLogs(cursor, batchSize) {
    const logs = [];
    const scanResult = await redisClient.scan(cursor, {
        MATCH: 'log:*',
        COUNT: batchSize,
    });
    
    for (const key of scanResult.keys) {
        const log = await redisClient.get(key);
        if (log) {
            logs.push(JSON.parse(log));
        }
    }
    
    return { logs, nextCursor: scanResult.cursor };
}

async function main() {
    redisClient.on('error', (err) => {
        console.log('Redis Client Error', err);
        throw new Error('Redis Client Error');
    });

    await redisClient.connect();

    const batchSize = 100;
    let currentCursor = '0';

    do {
        const { logs: batch, nextCursor } = await retrieveLogs(currentCursor, batchSize);
        console.log(`got a batch of ${batch.length} logs`);
        currentCursor = nextCursor;
        // 游标回到0时，说明所有键已遍历完成
    } while (currentCursor !== '0');

    await redisClient.disconnect();
}

main();

关键修改点 #

• 改用手动调用redisClient.scan，控制游标流转，每次仅获取指定COUNT的键
• retrieveLogs返回当前批次数据和下一个游标，外层循环根据游标判断是否继续
• 外层使用do-while循环，确保至少执行一次，直到游标归0终止循环

内容的提问来源于stack exchange，提问作者randombits