You need to enable JavaScript to run this app. Cython并行调用基于FFTW的SSHT外部C库时出现内存释放及FFTW断言错误
优惠活动
大模型
产品
解决方案
定价
更多

Cython并行调用基于FFTW的SSHT外部C库时出现内存释放及FFTW断言错误

我的尝试

我此前使用Python的multiprocess库并行实现了一段计算代码,可正常运行但速度较慢、内存占用较高,因此决定尝试将其改写为Cython代码。我对Cython、C语言经验较少,下面的示例已尽可能简化,该代码在串行模式下可正常运行,但一旦开启并行就无法工作。由于并行运行的特性,我梳理了全部代码并关闭了GIL。

该代码依赖外部C库SSHT(球谐变换工具库),编译说明见对应项目的README文档,底层调用了FFTW。SSHT仓库自带的Cython封装中调用了我正在使用的同一个C函数ssht_core_mw_inverse_sov_sym_ss,与我的实现逻辑最接近的仓库内置Cython代码如下:

def ssht_inverse_mwss_complex(
    np.ndarray[ double complex, ndim=1, mode="c"] f_lm not None,
    int L,
    int spin
):
    cdef ssht_dl_method_t dl_method = SSHT_DL_RISBO
    f_mwss_c = np.empty([L+1,2*L,], dtype=complex)
    ssht_core_mw_inverse_sov_sym_ss(
        <double complex*> np.PyArray_DATA(f_mwss_c),
        <const double complex*> np.PyArray_DATA(f_lm),
        L,
        spin,
        dl_method,
        0
    )
    return f_mwss_c

由于我需要无GIL运行的版本,因此不得不本地重新实现了该封装逻辑。

问题现象

使用Cython模块运行脚本时会触发段错误,每次报错略有不同:要么没有明确报错信息,要么提示指针内存分配相关错误:

malloc: *** error for object 0x7f83de8b58e0: pointer being freed was not allocated
python(21855,0x70000d333000) malloc: Double free of object 0x7f83de8b58e0
python(21855,0x70000d536000) malloc: *** set a breakpoint in malloc_error_break to debug

要么报错和FFTW相关:

fftw: /Users/runner/.conan/data/fftw/3.3.8/_/_/build/55f3919d9a41efc78a625ee65e5d1ea60d02b2ff/source_subfolder/kernel/planner.c:261: assertion failed: SLVNDX(slot) == slvndx

我查阅技术资料时发现有同类问题的反馈,因此担心是FFTW的设计特性导致我无法实现目标(毕竟我C语言经验不足)。

已尝试的操作

我尝试调用libc.stdlib提供的free接口,但没有解决问题;也尝试使用cython.view创建数组,但无法生成SSHT库要求的double complex类型;还尝试配置Cython调试模式,但在我的Mac上运行遇到了问题。我已经为此耗费了两天时间。

我的运行环境

我使用常规命令python setup.py build_ext --inplace编译扩展,使用python3.8.5、Cython==0.29.21,运行在macOS 11.0.1系统上。

相关代码

我的Cython文件如下:

import numpy as np
from libc.stdio cimport printf
from libc.stdlib cimport calloc, malloc
from cython.parallel import parallel, prange
from openmp cimport omp_get_thread_num

# needed to recreate without importing (for nogil)
cdef extern from "ssht/ssht.h" nogil:
    ctypedef enum ssht_dl_method_t:
        SSHT_DL_RISBO, SSHT_DL_TRAPANI
    void ssht_core_mw_inverse_sov_sym_ss(
        double complex *f,
        const double complex *flm,
        int L,
        int spin,
        ssht_dl_method_t dl_method,
        int verbosity
    )

def my_cython_module(int L, int threads):
    """
    dummy function more to show that parallel loops fails
    """
    cdef int ell, tid
    with nogil, parallel(num_threads=threads):
        tid = omp_get_thread_num()
        for ell in prange(L * L, schedule="guided"):
            printf("ell: %i\n", ell)
            _ssht_inverse(L, ell)

cdef double complex * _ssht_inverse(int L, int ind) nogil:
    """
    function creates a 1D complex array flm  with zeros and a 1
    then calls c function to get 2D complex array f
    not returning anything as it's just for demonstration
    """
    cdef ssht_dl_method_t dl_method = SSHT_DL_RISBO
    cdef double complex *flm = NULL
    cdef double complex *f = NULL
    flm = <double complex *> calloc(L * L, sizeof(double complex))
    flm[ind] = 1
    f = <double complex *> malloc((L + 1) * (2 * L) * sizeof(double complex))
    ssht_core_mw_inverse_sov_sym_ss(f, flm, L, 0, dl_method, 0)
    return f

我的setup.py文件如下:

import os
from Cython.Build import cythonize
from setuptools import Extension, setup

# running on mac so need GCC instead of clang
os.environ["CC"] = "gcc-10"

setup(
    ext_modules=cythonize(
        Extension(
            "test",
            ["*.pyx"],
            extra_compile_args=["-fopenmp"],
            extra_link_args=["-fopenmp"],
            include_dirs=["/usr/local/include"],
        ),
        annotate=True,
        language_level=3,
        compiler_directives=dict(boundscheck=False, embedsignature=True),
    ),
)
可正常运行的Python并行最小示例

以下示例(可通过pip install pyssht安装依赖)可正常并行运行,因此判断问题出在C/Cython实现环节:

# the cython wrapper from the external library
from pyssht import ssht_inverse_mwss_complex
import numpy as np
from multiprocess import Pool

def my_python_implementation(L, threads):
    """
    the python equivalent in parallel
    """
    def func(chunk):
        """
        deals with each chunk
        """
        for ell in chunk:
            print(f"ell: {ell}")
            flm = np.zeros(L * L, dtype=np.complex_)
            flm[ell] = 1
            ssht_inverse_mwss_complex(flm, L, 0)

    chunks = np.array_split(np.arange(L * L), threads)
    with Pool(processes=threads) as p:
        p.map(func, chunks)
后续期待

既然能通过Python实现并行运行,非常希望能找到Cython并行方案的可行实现路径。

内容的提问来源于Stack Exchange,提问作者Paddy Roddy

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.11.15 16:45:03