Sidekit创建UBM时代码冻结问题求助

Let’s break down why your tiny 3.8MB feature file is causing such extreme hang times, and walk through actionable fixes to get things moving:

1. MPI Configuration & Worker Process Failures #

Even with MPI enabled, silent worker process issues often cause queue deadlocks:

• Run your script with verbose MPI logging (e.g., mpiexec -n 2 -v python your_script.py on Linux, or use Microsoft MPI’s verbose flags on Windows) to check if workers are crashing, failing to initialize, or not communicating with the main process.
• Avoid over-subscribing CPU cores—start with a small number of MPI processes (like 2) instead of maxing out your core count. Over-subscription can create resource conflicts that block queue operations.
• On Windows, double-check that your MPI implementation (e.g., Microsoft MPI) is fully compatible with Sidekit. Some users report process communication gaps between Python and Windows MPI that lead to indefinite hangs.

2. Validate Your Feature File & Feature List #

Your feat.h5 might have hidden issues that trip up the EM_Split logic:

• Manually inspect the HDF5 file to confirm all 24 feature sets are properly structured with consistent dimensions (no corrupted or malformed entries). Use this quick script:

  import h5py
  with h5py.File('feat.h5', 'r') as f:
      for key in f.keys():
          print(f"Feature {key}: {f[key].shape}")
  

• When using feature_list=all_files, verify that all_files exactly matches the key names in your HDF5 file (no typos, missing entries, or extra paths). A mismatched list can make the function wait for data that never arrives.

3. Sidekit’s EM_Split Edge Cases with Small Datasets #

24 audio files is a very small dataset for EM_Split’s default parallel logic:

• Test with no parallelism first to rule out queue/MPI issues entirely:

  # Disable parallel processing temporarily
  ubm.EM_Split(feature_list=all_files, distrib_nb=1, num_thread=1)
  

• Upgrade Sidekit to the latest version—older releases had bugs related to queue handling for small datasets, which are often fixed in newer builds. Use pip install --upgrade sidekit (or pull the latest source if you’re using a custom build).

4. Diagnose Queue Deadlock #

The queue_in.join() line points to a classic producer-consumer deadlock scenario:

• Add debug logging (either in your script or by modifying Sidekit’s source temporarily) to track when items are added to or removed from the queue. This will tell you if the queue is being populated but not consumed, or vice versa.
• Ensure worker processes aren’t crashing silently. If a worker dies without notifying the main process, the queue will wait forever for that worker to finish its tasks.

5. Unexpected Memory Overhead (Even with 32GB RAM) #

High-dimensional features can cause memory bloat even with small file sizes:

• Check the dimension of your feature vectors (e.g., 39-dimensional MFCCs vs. 128+ dimensional embeddings). EM iterations multiply memory usage, and swapping to disk can make the process look frozen (it’s just extremely slow).
• Monitor system memory usage while the script runs. If RAM spikes to 100%, try reducing the batch size or using lower-dimensional features for testing.

内容的提问来源于stack exchange，提问作者nahomyaja