Linux下复用malloc管理特定内存区域的可行方案及替代分配器问询

Great question! Let's break this down clearly:

First: Can standard malloc()/free() handle custom memory regions? #

No, you can't directly repurpose the standard C library's malloc() and free() to manage custom memory regions like inter-process shared memory, memory-mapped files, or non-volatile memory.

The standard allocator is tightly coupled to the process's default heap—this is the memory it gets via brk()/sbrk() or anonymous mmap() calls. It doesn't expose any hooks or APIs to redirect its allocation logic to a user-controlled block of memory. So if you try to use free() on a pointer from shared memory, for example, it'll crash or cause undefined behavior because the allocator doesn't recognize that memory as part of its managed heap.

So what's the alternative? Use allocators with custom arena support #

Luckily, there are several well-known, battle-tested memory allocators designed explicitly to handle custom memory regions. Here are the most popular options:

• dlmalloc (Doug Lea's Malloc): The original high-performance allocator that many system malloc implementations (like glibc's ptmalloc) are based on. It supports creating custom memory arenas—you can initialize a malloc state using your own memory block (e.g., shared memory, mmap'd file) and then use allocator-specific functions (like dlmalloc()/dlfree() tied to that state) to manage memory within that region.
• jemalloc: Developed by Facebook for high concurrency and performance, jemalloc lets you create custom arenas backed by user-specified memory. You can configure an arena to use your shared, mmap'd, or NVM memory, then use jemalloc's arena-aware API to allocate and free within that space. It's also optimized for multi-threaded workloads.
• tcmalloc (Thread-Caching Malloc): Google's allocator, built for low-latency multi-threaded applications. It supports custom memory arenas, allowing you to tie an arena to your custom memory block and use arena-specific allocation calls instead of the default malloc().
• Hoard: An allocator focused on eliminating memory contention in multi-threaded systems. It lets you create private heaps that can be backed by your custom memory regions, making it a good fit for shared memory scenarios where multiple processes need to allocate from the same pool.

Quick example with dlmalloc #

Here's a rough idea of how you'd use dlmalloc with a shared memory region:

#include "dlmalloc.h"

// Assume shm_ptr is the pointer to your shared memory block, shm_size is its total size
malloc_state custom_arena;

// Initialize the custom arena state
malloc_init_state(&custom_arena);
// Attach our shared memory to the arena
dlmalloc_attach(&custom_arena, shm_ptr, shm_size);

// Allocate memory from the custom arena
void *my_ptr = dlmalloc(&custom_arena, 1024);
// ... use the allocated memory ...
// Free it back to the custom arena
dlfree(&custom_arena, my_ptr);

Note: Exact function names might vary slightly between dlmalloc versions, so check the documentation for your specific build.

Key considerations #

• Process/thread safety: If your custom region is shared across multiple processes, make sure the allocator you choose supports inter-process synchronization (or add your own locking around allocation calls).
• Non-volatile memory (NVM): Some allocators (like jemalloc with NVM extensions) or specialized NVM allocators (like PMDK's libpmemobj) are optimized for persistent memory, handling durability and consistency that standard allocators don't address.

内容的提问来源于stack exchange，提问作者68ejxfcj5669