难以排查的内存泄漏：分块流式数据解析方案求助

问题背景 #

我设计了一套解析连续分块流式数据的方案，数据包含多个可通过起始、终止ASCII序列识别的数据集，使用strcmp等比较函数进行解析：

• 接收分块数据时调用appendToBuffer(data, data_length);
• appendToBuffer调用处理函数，递归解析缓冲区中首个可用数据集，返回已解析字节数后从缓冲区头部移除这些字节
• 新分块到来时，通过safe_realloc扩容缓冲区，复制数据后重复解析流程

目前存在明显内存泄漏：每接收一块数据，堆内存都会显著下降。内存分析器指向safe_realloc中的malloc，但我认为是堆追踪仅覆盖appendToBuffer调用，期望函数返回时释放所有分配内存，而我的设计是让缓冲区在appendToBuffer返回后保持完整。需要解决内存泄漏问题，也接受基于C++标准库的全新实现方案。

现有代码 #

appendToBuffer函数 #

bool appendToBuffer(uint8_t* data, size_t len){

  // Either create a new buffer, or grow the existing one
  if(length == 0 || buffer == NULL){
    if(buffer != NULL)
      free(buffer);

    ESP_LOGD(TAG, "New buffer");
    buffer = (char*)malloc(len);
  }else{
    ESP_LOGD(TAG, "Realloc buffer");
    buffer = safe_realloc(buffer, length, length+len);
  }

  ESP_LOGD(TAG,"Free heap: %u", (unsigned int) esp_get_free_heap_size());

  assert(buffer != NULL);

  // Copy the new data into the buffer at the right spot, which is the length that is already in the buffer
  memcpy(buffer + length, data, len);

  // The buffer has just grown in size
  length += len;

  // Process the current buffer and know what to remove from the front of the buffer
  size_t bytesProcessed = processBuffer();

  // Remove these bytes from the front of the buffer
  size_t tempBufferLength = length-bytesProcessed;

  // Remove the first vCard in the buffer, as it has been processed
  if (bytesProcessed > 0 && tempBufferLength > 0){

    char* tempBuffer = (char*)malloc(tempBufferLength);

    // Copy the buffer, except the first vCard to a temp buffer
    memcpy(tempBuffer, buffer+bytesProcessed, tempBufferLength);

    free(buffer);

    // Shrink the buffer
    buffer = (char*)malloc(tempBufferLength);

    assert(buffer != NULL);

    // Copy the tempbuffer back to the buffer
    memcpy(buffer, tempBuffer, tempBufferLength);

    // Clean the tempbuffer
    free(tempBuffer);

    // Set the length
    length=tempBufferLength;
  }

  ESP_LOGD(TAG, "Remaining buffer: (len=%d) %s", length, buffer);

  // If all the data in the buffer has been parsed, free the buffer
  if(length == 0){
    ESP_LOGD(TAG, "Freeing buffer");
    free(buffer);
  }


  return true;
}

safe_realloc函数 #

static char* safe_realloc(char* original, size_t original_len, size_t new_len){
  if(original_len == new_len)
    return original;

  if(new_len == 0){
    free(original);
    return NULL;
  }

  char* buffer = (char*)malloc(new_len);
  assert(buffer != NULL);
  assert(original != NULL);

  if (new_len < original_len) {
      // Shrinks, only copy the first new_len bytes
      memcpy(buffer, original, new_len);
  } else {
      // Expands, copy the entire original buffer
      memcpy(buffer, original, original_len);
  }

  free(original);

  return buffer;
}

问题分析与C语言版本修复 #

内存泄漏根源 #

1. 野指针问题：当length == 0时调用free(buffer)但未将buffer置为NULL，后续进入分支时会再次free野指针，导致堆损坏，同时内存管理逻辑混乱。
2. 冗余内存分配：移除已处理数据时，先分配临时缓冲区，释放原缓冲区后又重新分配，额外的内存操作增加了泄漏和碎片风险。
3. 缓冲区收缩逻辑错误：未正确利用safe_realloc的特性，手动拷贝数据时容易出现边界错误。

修复后的代码 #

修复appendToBuffer函数

bool appendToBuffer(uint8_t* data, size_t len){

  // 初始化或扩容缓冲区
  if(length == 0 || buffer == NULL){
    if(buffer != NULL){
      free(buffer);
      buffer = NULL; // 释放后置空，避免野指针
    }
    ESP_LOGD(TAG, "New buffer");
    buffer = (char*)malloc(len);
  }else{
    ESP_LOGD(TAG, "Realloc buffer");
    buffer = safe_realloc(buffer, length, length + len);
  }

  ESP_LOGD(TAG,"Free heap: %u", (unsigned int) esp_get_free_heap_size());

  assert(buffer != NULL);

  // 复制新数据到缓冲区尾部
  memcpy(buffer + length, data, len);
  length += len;

  // 处理缓冲区中的数据
  size_t bytesProcessed = processBuffer();
  size_t tempBufferLength = length - bytesProcessed;

  // 移除已处理的数据
  if (bytesProcessed > 0) {
    if (tempBufferLength == 0) {
      // 所有数据都已处理，释放缓冲区并置空
      ESP_LOGD(TAG, "Freeing buffer");
      free(buffer);
      buffer = NULL;
      length = 0;
    } else {
      // 将有效数据移动到缓冲区头部，再收缩内存
      memmove(buffer, buffer + bytesProcessed, tempBufferLength);
      buffer = safe_realloc(buffer, length, tempBufferLength);
      assert(buffer != NULL);
      length = tempBufferLength;
    }
  }

  ESP_LOGD(TAG, "Remaining buffer: (len=%zu) %s", length, buffer);

  return true;
}

优化safe_realloc函数

static char* safe_realloc(char* original, size_t original_len, size_t new_len){
  if(original_len == new_len){
    return original;
  }

  if(new_len == 0){
    free(original);
    return NULL;
  }

  char* new_buffer = (char*)malloc(new_len);
  assert(new_buffer != NULL);
  assert(original != NULL);

  // 复制最小长度的数据，避免越界
  size_t copy_len = (new_len < original_len) ? new_len : original_len;
  memcpy(new_buffer, original, copy_len);

  free(original);
  return new_buffer;
}

C++标准库全新实现方案 #

利用std::string自带的内存管理能力，彻底避免手动内存操作，大幅简化代码：

#include <string>
#include <cstring>
#include "esp_log.h"

static std::string buffer;
static const char* TAG = "StreamParser";

bool appendToBuffer(uint8_t* data, size_t len){
  // 将新数据追加到缓冲区
  buffer.append(reinterpret_cast<char*>(data), len);

  ESP_LOGD(TAG,"Free heap: %u", (unsigned int) esp_get_free_heap_size());

  // 处理缓冲区数据，processBuffer需接收std::string&并返回已处理字节数
  size_t bytesProcessed = processBuffer(buffer);

  if (bytesProcessed > 0) {
    if (bytesProcessed >= buffer.size()) {
      // 所有数据处理完成，清空缓冲区
      buffer.clear();
      ESP_LOGD(TAG, "Buffer cleared");
    } else {
      // 移除已处理的前缀
      buffer = buffer.substr(bytesProcessed);
    }
  }

  ESP_LOGD(TAG, "Remaining buffer: (len=%zu) %s", buffer.size(), buffer.c_str());

  return true;
}

// 示例processBuffer实现（需根据实际数据集格式调整）
size_t processBuffer(std::string& buf){
  // 假设终止序列是"\r\nEND\r\n"
  const std::string end_seq = "\r\nEND\r\n";
  size_t end_pos = buf.find(end_seq);
  if(end_pos != std::string::npos){
    // 找到完整数据集，返回从开头到终止序列末尾的长度
    size_t data_len = end_pos + end_seq.size();
    ESP_LOGD(TAG, "Parsed dataset of length %zu", data_len);
    return data_len;
  }
  // 未找到完整数据集，返回0
  return 0;
}

C++方案优势 #

1. 完全消除手动内存管理，std::string自动处理分配、扩容和释放，从根源避免内存泄漏。
2. 代码简洁，substr操作直接移除已处理数据，无需手动拷贝和内存调整。
3. find方法高效查找起始/终止序列，替代手动strcmp循环，提升解析效率。

内容的提问来源于stack exchange，提问作者tim687