Python使用SDV GaussianCopula生成数据时报SVD未收敛错误

问题背景 #

我目前正在使用SDV和GaussianCopula训练模型，加载给定数据集进行训练，生成数据集时得到如下错误：

Saving Model to path D:/.../GaussianCopula/model_MLB_1.pkl
Generating 22479 rows of synthetic data
Traceback (most recent call last):
  File ".\generate_gaussian_model.py", line 47, in <module>
    samples = gaussianCopula.sample(len(data.index))
  File "C:\Users\...\AppData\Local\Programs\Python\Python37\lib\site-packages\sdv\tabular\base.py", line 442, in sample
    return self._sample_batch(num_rows, max_retries, max_rows_multiplier)
  File "C:\Users\...\AppData\Local\Programs\Python\Python37\lib\site-packages\sdv\tabular\base.py", line 300, in _sample_batch
    num_rows, conditions, transformed_conditions, float_rtol)
  File "C:\Users\...\AppData\Local\Programs\Python\Python37\lib\site-packages\sdv\tabular\base.py", line 228, in _sample_rows
    sampled = self._sample(num_rows)
  File "C:\Users\...\AppData\Local\Programs\Python\Python37\lib\site-packages\sdv\tabular\copulas.py", line 319, in _sample
    return self._model.sample(num_rows, conditions=conditions)
  File "C:\Users\...\AppData\Local\Programs\Python\Python37\lib\site-packages\copulas\__init__.py", line 36, in wrapper
    return function(self, *args, **kwargs)
  File "C:\Users\...\AppData\Local\Programs\Python\Python37\lib\site-packages\copulas\multivariate\gaussian.py", line 249, in sample
    samples = self._get_normal_samples(num_rows, conditions)
  File "C:\Users\...\AppData\Local\Programs\Python\Python37\lib\site-packages\copulas\multivariate\gaussian.py", line 223, in _get_normal_samples
    samples = np.random.multivariate_normal(means, covariance, size=num_rows)
  File "mtrand.pyx", line 4120, in numpy.random.mtrand.RandomState.multivariate_normal
  File "<__array_function__ internals>", line 6, in svd
  File "C:\Users\...\AppData\Local\Programs\Python\Python37\lib\site-packages\numpy\linalg\linalg.py", line 1660, in svd
    u, s, vh = gufunc(a, signature=signature, extobj=extobj)
  File "C:\Users\...\AppData\Local\Programs\Python\Python37\lib\site-packages\numpy\linalg\linalg.py", line 97, in _raise_linalgerror_svd_nonconvergence
    raise LinAlgError("SVD did not converge")
numpy.linalg.LinAlgError: SVD did not converge

我已尝试公开的相关解决方案未生效，我的训练代码如下：

from sdv.tabular import GaussianCopula
import pickle
import pandas as pd
from pandas.core.indexes.base import Index

header_import_path = "C:/Users/.../headers/all_headers.txt"

all_mlb_names = ['MLB_1', 'MLB_7', 'MLB_19', 'MLB_31', 'MLB_41', 'MLB_45', 'MLB_49', 'MLB_53', 'MLB_58']
with open(header_import_path, 'rb') as fp:
    all_headers = pickle.load(fp)

for mlb_file_name in all_mlb_names:
    #Create separate model for each MLB Table
    model_export_path = "D:/.../GaussianCopula/model_{0}.pkl".format(mlb_file_name)
    synth_data_export_path = "C:/Users/.../models/generated/{0}_samples.csv".format(mlb_file_name)
    data_import_path = "C:/Users/.../models/original/{0}.csv".format(mlb_file_name)
    headers = all_headers[mlb_file_name]

    print("Read data for table {0}".format(mlb_file_name))
    data = pd.read_csv(data_import_path, sep='|', names=headers)

    # This is necessary to remove invalid columns from my original dataset
    for colname in data.columns:
        if colname.startswith("Calculation"):
            data = data.drop(axis=1, labels=[colname])

    # Thought this would fix my issue but it didn't
    data.dropna(inplace=True)
    
    #print("Takes a third of the dataset")
    data = data.sample(frac=0.3)
   
    print(data)
    gaussianCopula = GaussianCopula()
    print("Start training of GaussianCopula Model")
    gaussianCopula.fit(data)

    print("Saving Model to path {0}".format(model_export_path))
    gaussianCopula.save(model_export_path)

    print("Generating {0} rows of synthetic data".format(len(data.index)))
    
    # Here it begins to crash
    samples = gaussianCopula.sample(len(data.index))
    samples.to_csv(synth_data_export_path, header=True, sep='|', index=False)

目前已验证使用data = data.sample(n=1000)取1000条样本运行可以成功，但该数据量无法满足需求。

报错原因 #

核心原因是大样本训练后得到的协方差矩阵存在数值稳定性问题，导致SVD分解无法收敛，具体触发原因通常为以下几种：

1. 数据特征间存在高度共线性，协方差矩阵接近奇异，大样本下数值精度问题被放大
2. 不同特征的数值范围差异过大，协方差矩阵条件数过高，超出SVD算法的稳定处理范围
3. 未添加正则项的协方差矩阵在样本量变大后出现浮点精度误差，导致分解失败

可行解决方案 #

以下方案可单独使用也可组合使用：

• 开启协方差矩阵正则化：初始化GaussianCopula时传入正则参数，给协方差矩阵添加极小的单位矩阵偏移量，避免矩阵奇异

  gaussianCopula = GaussianCopula(
      random_state=42,
      copula_kwargs={"sigma": 1e-6}
  )
  

• 预处理数据降低协方差矩阵条件数：对所有数值特征做标准化，缩小特征间的数值范围差异

  from sklearn.preprocessing import StandardScaler
  # 训练前标准化数值列
  numeric_cols = data.select_dtypes(include=['int64', 'float64']).columns
  scaler = StandardScaler()
  data[numeric_cols] = scaler.fit_transform(data[numeric_cols])
  
  # 生成数据后转换回原始范围
  samples[numeric_cols] = scaler.inverse_transform(samples[numeric_cols])
  

• 去除高度共线性特征：计算特征间的皮尔逊相关系数，删除相关系数绝对值高于0.95的冗余特征，降低协方差矩阵的秩压力
• 拆分采样过程：将大样本采样拆分为多次小批量采样后合并，避开单次大量采样的数值不稳定问题

  total_rows = len(data.index)
  batch_size = 2000
  sample_list = []
  remain_rows = total_rows
  
  while remain_rows > 0:
      cur_batch = min(batch_size, remain_rows)
      sample_list.append(gaussianCopula.sample(cur_batch))
      remain_rows -= cur_batch
  
  samples = pd.concat(sample_list, ignore_index=True)
  

内容的提问来源于stack exchange，提问作者41 72 6c