如何在Python与C进程间无磁盘共享内存映射文件及相关实现疑问

Great question! Interprocess communication between Python and C using memory maps without hitting disk is totally doable, and let's break down each part of your query clearly.

To share data between Python and C without writing anything to disk, you'll want to use in-memory named shared regions—these live entirely in RAM and don't persist to disk. The implementation varies slightly between Unix-like systems and Windows:

Unix-like Systems (Linux/macOS) #

We use shared memory objects via /dev/shm (a tmpfs filesystem that stays in RAM). Here's how to set it up:

Python Code

import os
import mmap

# Define unique name and size for shared memory
SHM_NAME = "/my_unique_shared_memory"
SHM_SIZE = 1024  # Adjust based on your data needs

# Create/open the shared memory object
shm_fd = os.shm_open(SHM_NAME, os.O_CREAT | os.O_RDWR, 0o666)
os.ftruncate(shm_fd, SHM_SIZE)  # Set the size of the shared region

# Map the region into Python's address space
mm = mmap.mmap(shm_fd, SHM_SIZE, mmap.MAP_SHARED, mmap.PROT_READ | mmap.PROT_WRITE)
os.close(shm_fd)  # File descriptor can be closed after mapping

# Example: Write data to the shared region
mm.write(b"Hello from Python!")
mm.seek(0)
print(f"Python read: {mm.read(16)}")

C Code

#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

#define SHM_NAME "/my_unique_shared_memory"
#define SHM_SIZE 1024

int main() {
    // Open the shared memory object
    int shm_fd = shm_open(SHM_NAME, O_RDWR, 0666);
    if (shm_fd == -1) { perror("shm_open"); exit(EXIT_FAILURE); }

    // Map the region into C's address space
    char *ptr = mmap(NULL, SHM_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, shm_fd, 0);
    if (ptr == MAP_FAILED) { perror("mmap"); exit(EXIT_FAILURE); }

    // Example: Read and write data
    printf("C read: %s\n", ptr);
    snprintf(ptr, SHM_SIZE, "Hello from C!");

    // Cleanup
    munmap(ptr, SHM_SIZE);
    close(shm_fd);
    // Uncomment to delete the shared object when done
    // shm_unlink(SHM_NAME);
    return 0;
}

Windows Systems #

We use named shared memory via CreateFileMapping with INVALID_HANDLE_VALUE (no disk file attached):

Python Code

import mmap
import ctypes
from ctypes import wintypes

kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
FILE_MAP_ALL_ACCESS = 0x000F001F
INVALID_HANDLE_VALUE = wintypes.HANDLE(-1).value

# Create named shared memory
h_map = kernel32.CreateFileMappingW(
    INVALID_HANDLE_VALUE,
    None,
    wintypes.DWORD(0x04),  # PAGE_READWRITE
    0,
    1024,
    "Local\\MyUniqueSharedMemory"
)
if not h_map:
    raise ctypes.WinError(ctypes.get_last_error())

# Map the region into Python's address space
mm = mmap.mmap(
    fileno=None,
    length=1024,
    tagname="Local\\MyUniqueSharedMemory",
    access=mmap.ACCESS_WRITE
)

# Example: Write data
mm.write(b"Hello from Python on Windows!")
mm.seek(0)
print(f"Python read: {mm.read(30)}")

# Cleanup
mm.close()
kernel32.CloseHandle(h_map)

C Code

#include <stdio.h>
#include <windows.h>

#define SHM_NAME L"Local\\MyUniqueSharedMemory"
#define SHM_SIZE 1024

int main() {
    HANDLE h_map = OpenFileMappingW(FILE_MAP_ALL_ACCESS, FALSE, SHM_NAME);
    if (!h_map) { printf("OpenFileMapping failed: %d\n", GetLastError()); return 1; }

    char *ptr = (char*)MapViewOfFile(h_map, FILE_MAP_ALL_ACCESS, 0, 0, SHM_SIZE);
    if (!ptr) { printf("MapViewOfFile failed: %d\n", GetLastError()); CloseHandle(h_map); return 1; }

    printf("C read: %s\n", ptr);
    snprintf(ptr, SHM_SIZE, "Hello from C on Windows!");

    // Cleanup
    UnmapViewOfFile(ptr);
    CloseHandle(h_map);
    return 0;
}

To guarantee both processes read/write the same region, follow these rules:

• Use a unique, consistent name: Both programs must use the exact same shared memory name (e.g., /my_unique_shared_memory on Unix, Local\\MyUniqueSharedMemory on Windows). Avoid generic names to prevent collisions.
• Match the memory size: Always map the same size of memory in both Python and C. Mismatched sizes can cause crashes or truncated data.
• Add synchronization (critical for concurrent access): If both processes will read/write at the same time, use locks or semaphores to avoid data corruption. For example:
  • On Unix, use fcntl.flock() in Python or pthread_mutex_t in C.
  • On Windows, use CreateMutex() in both languages.

Stick with ctypes in Python for 90% of use cases—here's why:

• Simplicity: ctypes lets you define plain C-style data structures (structs, arrays, basic types) in Python, then map them directly to the shared memory. No need to mess with Python's internal object layout or reference counting.
• Low coupling: You avoid tying your C code to Python's C API, which can break between Python versions.
• Ease of maintenance: Both teams (Python and C) can work with familiar, simple data formats.

Example: Shared Struct with ctypes #

C Struct Definition

typedef struct {
    int user_id;
    char status_message[64];
    float sensor_value;
} SharedData;

Python ctypes Mapping

import ctypes

class SharedData(ctypes.Structure):
    _fields_ = [
        ("user_id", ctypes.c_int),
        ("status_message", ctypes.c_char * 64),
        ("sensor_value", ctypes.c_float)
    ]

# Map the shared memory to the struct
shared_data = SharedData.from_buffer(mm)

# Modify data
shared_data.user_id = 456
shared_data.status_message = b"System online"
shared_data.sensor_value = 27.5

# Read data
print(f"ID: {shared_data.user_id}, Status: {shared_data.status_message.decode()}, Value: {shared_data.sensor_value}")

Only use Python's C API to parse Python objects if you absolutely need to pass complex Python data structures (like dictionaries or lists) directly. This is far more complex and only worth it for niche use cases.

内容的提问来源于stack exchange，提问作者Greg