使用LZ4压缩C++服务端客户端通信数据时出现corrupted size vs. prev_size错误

问题背景 #

使用C++实现客户端与服务端JSON通信，为提升性能引入LZ4压缩后，客户端持续触发"corrupted size vs. prev_size"错误，服务端无此问题。未压缩前程序运行正常，共用通信代码如下：

void serializeInt(std::array<uint8_t,8>& buf, int i, uint32_t val) { //Assumes that val is a 32-bit number (almost always true). Serializes in little endian in endian-agnostic way
    buf[i+0] = (val) & 0xFF;
    buf[i+1] = (val >> 8) & 0xFF;
    buf[i+2] = (val >> 16) & 0xFF;
    buf[i+3] = (val >> 24) & 0xFF;
}

uint32_t deserializeInt(std::array<uint8_t,8>& buf, int i){ //Deserialzes from little endian in endian-agnostic way
    return buf[i+0] | (buf[i+1] << 8) | (buf[i+2] << 16) | (buf[i+3] << 24);
}

boost::json::object readFromConn(){
    auto curr=currStruct();
    asio::error_code ec;
    
    asio::read(*(curr->conn), asio::buffer(curr->size_buf, 8), asio::transfer_exactly(8), ec);
    if (ec){
        throw RWError(ec);
    }
    
    auto compressed_size=deserializeInt(curr->size_buf, 0);
    auto input_size=deserializeInt(curr->size_buf, 4);
    

    auto compressed_data=(char*)malloc(compressed_size);
    auto input=(char*)malloc(input_size);
    
    asio::read(*(curr->conn), asio::buffer(compressed_data, compressed_size), asio::transfer_exactly(compressed_size), ec);
    
    if (ec){
        throw RWError(ec);
    }
    
    auto line=std::string_view(input,input_size);
    LZ4_decompress_safe(compressed_data, input, compressed_size, input_size);

    boost::json::object json=boost::json::parse(line,{}, {.max_depth=180,.allow_invalid_utf8=true,.allow_infinity_and_nan=true}).get_object();
    
    free(input);
    free(compressed_data);
    
    return json;
}

void writeToConn(boost::json::object& json){
    auto curr=currStruct();
    
    json["uuid"]=uuid;
    
    auto line=boost::json::serialize(json,boost::json::serialize_options{.allow_infinity_and_nan=true});

    auto input_size=line.size();
    auto input=line.c_str();
    auto max_compressed_size=LZ4_compressBound(input_size);
    auto compressed_data=(char*)malloc(max_compressed_size);
    
    auto compressed_size=LZ4_compress_default(input, compressed_data, input_size, max_compressed_size);
    
    serializeInt(curr->size_buf, 0, compressed_size);
    serializeInt(curr->size_buf, 4, input_size);
    asio::error_code ec;
    asio::write(*(curr->conn), asio::buffer(curr->size_buf,8), ec);
    
    if (ec){
        throw RWError(ec);
    }
    
    asio::write(*(curr->conn), asio::buffer(compressed_data,compressed_size), ec);
    
    if (ec){
        throw RWError(ec);
    }
    
    free(compressed_data);
}

问题根源与修复方案 #

1. std::string_view提前绑定未初始化内存 #

在readFromConn中，先创建std::string_view line(input, input_size)时，input是刚malloc出的未初始化内存，string_view会绑定到垃圾数据上。后续解压覆盖input后，string_view的底层指针已固定，内存检测工具会识别这种异常行为并触发报错。

修复：先执行解压操作，再创建std::string_view：

// 原代码顺序颠倒，修改后：
auto decompressed_bytes = LZ4_decompress_safe(compressed_data, input, compressed_size, input_size);
if (decompressed_bytes != input_size) {
    free(input);
    free(compressed_data);
    throw std::runtime_error("LZ4 decompression failed: size mismatch");
}
auto line = std::string_view(input, input_size);

2. 未检查malloc返回值 #

当input_size或compressed_size过大时，malloc可能返回nullptr，后续访问空指针会引发内存错误，触发检测工具报警。

修复：添加内存分配检查：

auto compressed_data=(char*)malloc(compressed_size);
if (!compressed_data) {
    throw std::bad_alloc();
}
auto input=(char*)malloc(input_size);
if (!input) {
    free(compressed_data);
    throw std::bad_alloc();
}

3. 忽略LZ4_decompress_safe返回值校验 #

LZ4_decompress_safe返回实际解压字节数，若返回值与input_size不匹配，说明压缩数据损坏或不完整，继续解析JSON会访问非法内存。

修复：如上述代码所示，校验返回值，不匹配则抛出错误。

4. 字节序一致性验证（潜在风险） #

虽然代码实现了小端序序列化/反序列化，但需确保客户端与服务端的uint32_t宽度一致。若两端字节序处理逻辑不一致，会导致解析出的compressed_size或input_size错误，进而引发内存越界。

验证方式：在客户端和服务端分别打印序列化前后的compressed_size和input_size，确认数值完全一致。

修改后的完整readFromConn函数 #

boost::json::object readFromConn(){
    auto curr=currStruct();
    asio::error_code ec;
    
    asio::read(*(curr->conn), asio::buffer(curr->size_buf, 8), asio::transfer_exactly(8), ec);
    if (ec){
        throw RWError(ec);
    }
    
    auto compressed_size=deserializeInt(curr->size_buf, 0);
    auto input_size=deserializeInt(curr->size_buf, 4);
    
    auto compressed_data=(char*)malloc(compressed_size);
    if (!compressed_data) {
        throw std::bad_alloc();
    }
    auto input=(char*)malloc(input_size);
    if (!input) {
        free(compressed_data);
        throw std::bad_alloc();
    }
    
    asio::read(*(curr->conn), asio::buffer(compressed_data, compressed_size), asio::transfer_exactly(compressed_size), ec);
    
    if (ec){
        free(input);
        free(compressed_data);
        throw RWError(ec);
    }
    
    auto decompressed_bytes = LZ4_decompress_safe(compressed_data, input, compressed_size, input_size);
    if (decompressed_bytes != input_size) {
        free(input);
        free(compressed_data);
        throw std::runtime_error("LZ4 decompression failed: expected " + std::to_string(input_size) + " bytes, got " + std::to_string(decompressed_bytes));
    }
    
    auto line = std::string_view(input, input_size);
    boost::json::object json=boost::json::parse(line,{}, {.max_depth=180,.allow_invalid_utf8=true,.allow_infinity_and_nan=true}).get_object();
    
    free(input);
    free(compressed_data);
    
    return json;
}

内容的提问来源于stack exchange，提问作者DarkRise