Node.js net Socket收发Buffer性能异常：Buffer.concat致大文件速度暴跌

问题描述 #

我通过node:net的Server和Socket发送Node.js Buffer，逻辑运行正常，但接收端使用Buffer.concat()重建Buffer时速度极慢。预期Buffer大小翻倍时耗时大致翻倍、字节/秒速度保持稳定；但实际观测：5MB Buffer耗时0.1秒，50MB Buffer耗时10秒，求优化方案。

测试代码（socketSpeed.js） #

const { Socket, Server } = require('node:net');

const server = process.argv[2] === 'server'
const client = process.argv[2] === 'client';
const address = process.argv[3];
const port = process.argv[4];
const mbToSend = process.argv[5];

if (server) {
    const serverOpts = {
        keepAlive: true,
        keepAliveInitialDelay: 3000,
    };
    socketServer = new Server(serverOpts);
    socketServer.on('connection', (socket) => {
        console.log('Server connected'); 
        socket.on('data', async (data) => {
            console.log('Server input: ' + data.toString());
        });
        socket.on('error', async (error) => {
            console.log('Server error: ' + error);
        });
        socket.on('end', async () => {
            console.log('Server end reached: ');
        });

        // 发送数据
        console.log(`Sending ${mbToSend} MB.`);
        socket.write(Buffer.from('x'.repeat( mbToSend * 1024 * 1024)));
        socket.end();
        socketServer.close();
    });
    const options = {
        port: port,
        host: address,
    };
    socketServer.listen(options);


} else if (client) {
    const socketClient = new Socket();
    const options = {
        port: port,
        host: address,
    };

    socketClient.on('connection', (socket) => {
        console.log('Client connected');        
    });
    socketClient.connect(options);

    let dataReceived = 0;
    let allData = undefined;

    const start = Date.now();
    socketClient.on('data', async (data) => {
        dataReceived += data.length;
        // 此处Buffer.concat()导致性能瓶颈
        if (allData === undefined) allData = data;
        else allData = Buffer.concat([allData, data]);
    });
    socketClient.on('error', async (error) => {
        console.log('Client error: ' + error);
    });
    socketClient.on('end', async () => {
        const end = Date.now();
        const time = (end - start) / 1000;
        
        console.log(`Client end reached, got ${dataReceived / 1024/1024} MB in ${time} seconds.`);
        console.log(` speed: ${dataReceived/1024/1024 / time} MB/s`);
    });

} else {
    console.log('在两个终端分别执行：');
    console.log('    node socketSpeed.js server localhost 34567 10');
    console.log('    node socketSpeed.js client localhost 34567');
}

运行方式 #

• 服务端：node socketSpeed.js server localhost 34567 10（参数10表示发送10MB Buffer）
• 客户端：node socketSpeed.js client localhost 34567

核心问题原因 #

每次调用Buffer.concat([allData, data])都会创建新Buffer，并将旧数据和新数据全部拷贝。当数据量增大时，拷贝次数和单次拷贝量呈指数级增长，时间复杂度变为O(n²)，这是大文件接收速度暴跌的根本原因。

优化方案 #

方案1：数组暂存小Buffer，最后一次性拼接 #

无需修改服务端，仅调整客户端逻辑：将每次收到的小Buffer存入数组，等接收完成后再调用一次Buffer.concat()，只做一次全量拷贝，时间复杂度降为O(n)。

修改客户端代码片段：

// 客户端修改部分
let dataReceived = 0;
const chunks = []; // 用数组暂存所有小Buffer

const start = Date.now();
socketClient.on('data', (data) => {
    dataReceived += data.length;
    chunks.push(data); // 直接存入数组，不实时拼接
});
socketClient.on('end', () => {
    const end = Date.now();
    const time = (end - start) / 1000;
    // 接收完成后一次性拼接
    const allData = Buffer.concat(chunks);
    
    console.log(`Client end reached, got ${dataReceived / 1024/1024} MB in ${time} seconds.`);
    console.log(` speed: ${dataReceived/1024/1024 / time} MB/s`);
});

方案2：预先分配Buffer，直接写入数据 #

如果能提前获取数据总长度（需服务端配合发送长度信息），可直接创建对应大小的Buffer，用copy()方法将数据写入对应位置，完全避免拼接操作。

服务端修改（先发送数据长度）：

// 服务端发送数据前先写入长度
const totalLength = mbToSend * 1024 * 1024;
socket.write(Buffer.alloc(4).writeUInt32BE(totalLength)); // 用4字节存储长度
socket.write(Buffer.from('x'.repeat(totalLength)));

客户端修改：

// 客户端修改部分
let dataReceived = 0;
let allData = null;

const start = Date.now();
socketClient.on('data', (data) => {
    if (!allData) {
        // 第一个数据包读取总长度
        const totalLength = data.readUInt32BE();
        allData = Buffer.alloc(totalLength);
        // 拷贝长度字段之后的内容
        const contentLen = data.length - 4;
        data.copy(allData, 0, 4);
        dataReceived += contentLen;
    } else {
        // 直接写入到对应偏移位置
        data.copy(allData, dataReceived);
        dataReceived += data.length;
    }
});

方案3：调大Socket的highWaterMark参数 #

默认net.Socket的highWaterMark为64KB，调大该参数可减少data事件触发次数，间接降低数组存储的Buffer数量，可与方案1配合使用进一步优化：

// 创建客户端Socket时指定参数
const socketClient = new Socket({ highWaterMark: 1024 * 1024 }); // 设置为1MB

内容的提问来源于stack exchange，提问作者GerritP