Geofileops大数据量相交运算结果异常问题求助及解决记录

问题场景 #

用geofileops替代geopandas原生运算提升性能，处理约60万行的RPG空间数据与部门边界的相交操作时，结果行数为0或明显异常。限制数据量（如仅取部分数据）时，相交运算能得到正确结果；大数据量运行时CPU使用率拉满，曾怀疑是并行化问题。

复现代码 #

from detection_algorithm.utils.database_connection import DatabaseConnection
import geopandas as gpd
import datetime
from io import BytesIO
import geofileops as gfo
import time
crs = 'EPSG:2154'
base_year = datetime.datetime.now().year - 2
query = f"SELECT geometry FROM departments WHERE \"INSEE_DEP\" = '71'"
database_engine = DatabaseConnection().con
department_gdf = gpd.read_postgis(query, database_engine, geom_col="geometry")
department_gdf.crs = crs

department_gdf.to_file("/tmp/department.gpkg")

current_difference_layer = gpd.read_postgis(
    "SELECT geometry FROM oso WHERE departement='71' and \"Classe\" IN (5, 6, 7, 8, 9, 10, 11, 12, 13)",
    database_engine,
    geom_col="geometry"
)
print("JUST AFTER OSO")
current_difference_layer.to_file("/tmp/oso.gpkg",)
current_difference_layer.crs = crs

for year in range(base_year, base_year - 1, -1):
    print("HER")
    rpg_gdf = gpd.read_postgis(
        "select geometry from rpg where region=1 and annee_d_export={}".format(year),
        database_engine,
        geom_col="geometry"
    )
    strat_time=time.time()

    # Convert GeoDataFrame to GeoPackage (GPKG) format in memory
    file_path = "/tmp/rpg.gpkg"
    gdf=rpg_gdf.to_crs(crs)
    gdf.to_file(file_path)
 
    
    
    gfo.intersection(file_path,"/tmp/department.gpkg","/tmp/rpg_dep.gpkg")
    rpg=gpd.read_file("/tmp/rpg_dep.gpkg")
    print("rpg LEN: "+str(len(rpg)))

问题根源 #

RPG数据中存在多部件几何（如MultiPolygon、MultiLineString）。小数据量下这类几何不会触发运算异常，但大数据量时，geofileops的并行处理逻辑无法正确解析多部件几何，导致相交运算结果异常。

解决方法 #

在将RPG数据转换为GPKG并执行相交运算前，添加explode操作拆分多部件几何：

# 在转换CRS之后、保存到文件之前添加以下代码
gdf = rpg_gdf.to_crs(crs)
# 拆分多部件几何为单个独立几何，index_parts=True保留原索引关联关系
gdf = gdf.explode(index_parts=True)
gdf.to_file(file_path)

添加该语句后，大数据量下的相交运算可得到正确结果。

内容的提问来源于stack exchange，提问作者Mohamed Zied Brahmi