OpenSSL RC4加解密输出末尾出现额外冗余乱码问题排查求助

问题现象 #

使用OpenSSL提供的RC4函数对字符串做加解密操作时，解密输出的字符串末尾被追加大量额外乱码字符。
初始实现代码如下：

RC4_KEY rc4_key;
QString key= "612c207468652066";

QByteArray keyData=key.QString::toUtf8();
unsigned char * mykey=reinterpret_cast<unsigned char *>(keyData.data());

QString input = "7447806cc997966f6b8b59a57c01bfe6fe8381d4fed5628b531fbeb1c4629151722ee712fa906dfea1c68b7015243bfcdd42d3e990c1bd6daa56df620a9d1e441f4ba308da5584e032f06fe925ee9df328fadab0ad2a5869fdea366189a397f72b320a4f11b0cc41cbfaa3b6923f20cc8c4c80d9c9b69eb5e2a6cc2fb9ee2e72ca79f08617b3b5812eddb3f58b68";
qDebug()<< "input"<<input << input.size();
QByteArray inputData=input.QString::toUtf8();
unsigned char * ciphertext1= (unsigned char*)malloc(input.size());
memset(ciphertext1, 0, input.size());
ciphertext1=reinterpret_cast<unsigned char *>(QByteArray::fromHex(inputData).data());
printf("ciphertext1: %s\n", ciphertext1);

RC4_set_key(&rc4_key,keyData.size() , mykey);

unsigned char * firsthashJson= (unsigned char*)malloc(input.size());
firsthashJson[input.size()-1]=0x00;
memset(firsthashJson, 0, input.size());
RC4(&rc4_key,input.length(),ciphertext1, firsthashJson);
printf("Decrypted: %s\n", firsthashJson);

QString str1 = QString::fromUtf8((char*)firsthashJson);

运行代码实际输出：

b036fe3eae7881fe135562335f9a##5265705250##b2a475f7a99cf400d34da436d522d9bc5566d8b7a015c7a16ee2d5e730199acf1f61fe2f3d6b9525e1c45acb36383253b4fc< Ú«Í`ã¿rßÖQÕ§ìj‗¤Ì&Öâ¤#>Xϸ0õ+▲®╩Ä=╚líUm"ód☻Ô1ı┘©ÜLÔ»âH÷/o┐öñ▄|▄ÐÄ┤N ☻§@ı2w5▒KTâUÆB¢ÏdÔxÚ⌂ê☺°¹╔«▓J)→8èê╔┤║ù@│k ↕▬↨[║GtUË(♂D

预期正确输出：

b036fe3eae7881fe135562335f9a##5265705250##b2a475f7a99cf400d34da436d522d9bc5566d8b7a015c7a16ee2d5e730199acf1f61fe2f3d6b9525e1c45acb36383253b4fc

按照给缓冲区追加结束符的方案调整代码后，问题仍然存在，调整后代码如下：

RC4_KEY rc4_key;
QString key= "612c207468652066";

QByteArray keyData=key.QString::toUtf8();
unsigned char * mykey=reinterpret_cast<unsigned char *>(keyData.data());

QString input = "7447806cc997966f6b8b59a57c01bfe6fe8381d4fed5628b531fbeb1c4629151722ee712fa906dfea1c68b7015243bfcdd42d3e990c1bd6daa56df620a9d1e441f4ba308da5584e032f06fe925ee9df328fadab0ad2a5869fdea366189a397f72b320a4f11b0cc41cbfaa3b6923f20cc8c4c80d9c9b69eb5e2a6cc2fb9ee2e72ca79f08617b3b5812eddb3f58b68";
qDebug()<< "input"<<input << input.size();
QByteArray inputData=input.QString::toUtf8();
unsigned char * ciphertext1= (unsigned char*)malloc(input.size());
memset(ciphertext1, 0, input.size());
ciphertext1=reinterpret_cast<unsigned char *>(QByteArray::fromHex(inputData).data());
printf("ciphertext1: %s\n", ciphertext1);

RC4_set_key(&rc4_key,keyData.size() , mykey);

unsigned char * firsthashJson= (unsigned char*)malloc(input.size()+1);

memset(firsthashJson, 0, input.size()+1);
firsthashJson[input.size()]=0;

RC4(&rc4_key,input.size(),ciphertext1, firsthashJson);
printf("Decrypted: %s\n", firsthashJson);

firsthashJson[input.size()]=0;
QString str1 = QString::fromUtf8((char*)firsthashJson);

qDebug()<<"final string "<<str1<< str1.size();

问题根因 #

乱码由两个代码错误共同导致：

• 长度计算错误：传入的input是十六进制编码的密文字符串，每2个十六进制字符对应1个实际密文字节，QByteArray::fromHex()解码后返回的密文数组实际长度为input.size()/2。代码中给RC4函数传入的解密长度是十六进制字符串的长度input.size()，是实际密文长度的2倍，RC4会把密文缓冲区后方的无效内存内容当作密文参与运算，解密结果自然会包含大量乱码。
• 悬空指针访问：QByteArray::fromHex(inputData)生成的是临时对象，当前代码行执行结束后临时对象会被自动销毁，直接取它的data()指针赋值给ciphertext1后，该指针指向的内存已经被释放，后续访问属于未定义行为，会读取到内存中的随机值，进一步导致解密结果异常。此外之前用malloc给ciphertext1分配的内存被直接覆盖指针地址，还会造成内存泄漏。

修正方案 #

修正后的代码如下：

RC4_KEY rc4_key;
QString key= "612c207468652066";

QByteArray keyData = key.toUtf8();
unsigned char * mykey = reinterpret_cast<unsigned char *>(keyData.data());

QString input = "7447806cc997966f6b8b59a57c01bfe6fe8381d4fed5628b531fbeb1c4629151722ee712fa906dfea1c68b7015243bfcdd42d3e990c1bd6daa56df620a9d1e441f4ba308da5584e032f06fe925ee9df328fadab0ad2a5869fdea366189a397f72b320a4f11b0cc41cbfaa3b6923f20cc8c4c80d9c9b69eb5e2a6cc2fb9ee2e72ca79f08617b3b5812eddb3f58b68";
qDebug()<< "input"<<input << input.size();

// 用局部变量持有十六进制解码后的密文数组，保证内存有效期覆盖整个解密流程
QByteArray cipherArray = QByteArray::fromHex(input.toUtf8());
int cipherLen = cipherArray.size();
unsigned char * ciphertext1 = reinterpret_cast<unsigned char *>(cipherArray.data());

RC4_set_key(&rc4_key, keyData.size(), mykey);

// 分配解密缓冲区，多预留1字节存放C风格字符串结束符
QByteArray plainArray(cipherLen + 1, 0);
unsigned char * firsthashJson = reinterpret_cast<unsigned char *>(plainArray.data());

// 传入实际密文长度做解密
RC4(&rc4_key, cipherLen, ciphertext1, firsthashJson);
printf("Decrypted: %s\n", firsthashJson);

QString str1 = QString::fromUtf8(plainArray);
qDebug()<<"final string "<<str1<< str1.size();

修正点说明：

• 用局部QByteArray变量存储十六进制解码后的密文，避免临时对象销毁产生悬空指针
• 解密长度使用解码后的实际密文字节长度，不再使用十六进制字符串的长度
• 用Qt的QByteArray自动管理缓冲区内存，避免手动malloc带来的内存泄漏、长度计算错误问题

内容的提问来源于stack exchange，提问作者yamilyana