如何用ZSTD Simple API正确压缩C++ std::vector？解压不一致排查

问题描述 #

作为C++新手，尝试用ZSTD压缩库压缩std::vector<int>，调用ZSTD Simple API的ZSTD_compress和ZSTD_decompress接口后，解压得到的vector与原vector不一致，已查阅ZSTD官方文档未找到解决办法，请求帮助。

原始代码片段 #

//Initialize a vector
vector<int> NumToCompress ;
NumToCompress.resize(10000);
for(int i = 0; i < 10000; i++)
{
    NumToCompress[i] = rand()% 255;
}
//compress
int* com_ptr = NULL;
size_t NumSize = NumToCompress.size();
size_t Boundsize = ZSTD_compressBound(NumSize);
com_ptr =(int*) malloc(Boundsize);
size_t ComSize;
ComSize = ZSTD_compress(com_ptr,Boundsize,NumToCompress.data(),NumToCompress.size(),ZSTD_fast);
//decompress
int* decom_ptr = NULL;
unsigned long long decom_Boundsize;
decom_Boundsize = ZSTD_getFrameContentSize(com_ptr, ComSize);
decom_ptr = (int*)malloc(decom_Boundsize);
size_t  DecomSize;
DecomSize = ZSTD_decompress(decom_ptr, decom_Boundsize, com_ptr, ComSize);
vector<int> NumAfterDecompress(decom_ptr,decom_ptr+DecomSize);
//check if two vectors are same
if(NumToCompress == NumAfterDecompress)
{
    cout << "Two vectors are same" << endl;
}else
{cout << "Two vectors are insame" << endl;}
free(com_ptr);
free(decom_ptr);

测试结果 #

Two vectors are insame

Original vector:
163 151 162 85 83 190 241 252 249 121 107 82 20 19 233 226 45 81 142 31 86 8 87 39 167 5 212 208 82 130 119 117 27 153 74 237 88 61 106 82 54 213 36 74 104 142 173 149 95 60 53 181 196 140 221 108 17 50 61 226 180 180 89 207 206 35 61 39 223 167 249 150 252 30 224 102 44 14 123 140 202 48 66 143 188 159 123 206 209 184 177 135 236 138 214 187 46 21 99 14

Decompressed vector:
163 151 162 85 83 190 241 252 249 121 107 82 20 19 233 226 45 81 142 31 86 8 87 39 167 0 417 0551929248 21916 551935408 21916 551933352 21916 551939512 21916 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

疑问解答 #

Case 1: ZSTD能否直接压缩std::vector？ #

可以。ZSTD压缩的是连续内存块，std::vector底层存储为连续数组，通过data()方法可获取内存起始地址，压缩逻辑与普通C数组完全一致。

Case 2: 如何用ZSTD正确压缩vector？ #

代码核心错误是混淆元素个数与字节数，ZSTD所有API均基于字节数计算，而非元素数量。具体错误点与修正方案如下：

错误点分析

1. 压缩阶段：

   • ZSTD_compressBound参数应为输入数据总字节数，而非元素数，正确值为NumToCompress.size() * sizeof(int)
   • ZSTD_compress第四个参数是输入数据字节数，同样需传入NumToCompress.size() * sizeof(int)

2. 解压阶段：

   • ZSTD_getFrameContentSize返回解压后的总字节数，不是元素数
   • 构造NumAfterDecompress时，需将DecomSize（字节数）除以sizeof(int)得到元素数量

修正后的代码

#include <vector>
#include <iostream>
#include <cstdlib>
#include <zstd.h>

int main() {
    // Initialize a vector
    std::vector<int> NumToCompress;
    NumToCompress.resize(10000);
    for(int i = 0; i < 10000; i++)
    {
        NumToCompress[i] = rand() % 255;
    }

    // Compress
    const size_t inputByteSize = NumToCompress.size() * sizeof(int);
    const size_t compressBound = ZSTD_compressBound(inputByteSize);
    void* com_ptr = malloc(compressBound);
    const size_t ComSize = ZSTD_compress(com_ptr, compressBound, NumToCompress.data(), inputByteSize, ZSTD_fast);

    // Check compression error
    if (ZSTD_isError(ComSize)) {
        std::cerr << "Compression error: " << ZSTD_getErrorName(ComSize) << std::endl;
        free(com_ptr);
        return 1;
    }

    // Decompress
    unsigned long long decom_byteSize = ZSTD_getFrameContentSize(com_ptr, ComSize);
    if (decom_byteSize == ZSTD_CONTENTSIZE_ERROR || decom_byteSize == ZSTD_CONTENTSIZE_UNKNOWN) {
        std::cerr << "Failed to get decompressed size" << std::endl;
        free(com_ptr);
        return 1;
    }

    int* decom_ptr = (int*)malloc(decom_byteSize);
    const size_t DecomSize = ZSTD_decompress(decom_ptr, decom_byteSize, com_ptr, ComSize);

    // Check decompression error
    if (ZSTD_isError(DecomSize)) {
        std::cerr << "Decompression error: " << ZSTD_getErrorName(DecomSize) << std::endl;
        free(com_ptr);
        free(decom_ptr);
        return 1;
    }

    // Construct decompressed vector (convert byte count to element count)
    const size_t decom_elementCount = DecomSize / sizeof(int);
    std::vector<int> NumAfterDecompress(decom_ptr, decom_ptr + decom_elementCount);

    // Check if two vectors are same
    if(NumToCompress == NumAfterDecompress)
    {
        std::cout << "Two vectors are same" << std::endl;
    } else {
        std::cout << "Two vectors are insame" << std::endl;
    }

    free(com_ptr);
    free(decom_ptr);
    return 0;
}

额外提示：添加ZSTD错误检查可避免压缩/解压失败导致的未知问题。

内容的提问来源于stack exchange，提问作者zwq