TensorFlow训练时sess.run()卡顿求助：PointNet零件分割自定义数据问题

Let’s break down the possible causes and actionable fixes for your training freeze issue when scaling PointNet’s input from (2048,3) to (4096,3):

1. GPU Memory Limits & Fragmentation #

Sudden freezes during sess.run() often tie back to GPU memory constraints, especially since doubling your point count per sample roughly doubles the model’s memory footprint:

• Use nvidia-smi (for NVIDIA GPUs) in a separate terminal to track real-time memory usage. If usage spikes to 100% right before the freeze, that’s your culprit.
• Quick fixes:
  • Cut your batch size in half (or more) to reduce immediate memory load.
  • Enable TensorFlow’s dynamic memory growth to avoid pre-allocating all VRAM at once:

    gpus = tf.config.experimental.list_physical_devices('GPU')
    if gpus:
        try:
            for gpu in gpus:
                tf.config.experimental.set_memory_growth(gpu, True)
        except RuntimeError as e:
            print(e)
    

  • Add tf.keras.backend.clear_session() at the end of each epoch to clear leftover GPU cache and reduce fragmentation.

2. Data Pipeline Bottlenecks (Even with Non-Empty feed_dict) #

A non-empty feed_dict doesn’t guarantee your data pipeline is stable. Check these points:

• If you’re loading data synchronously from disk during training, switch to tf.data.Dataset for asynchronous loading — this decouples data preparation from model training and prevents stalls.
• Add a sanity check in your data loader to ensure every sample is exactly (4096,3):

  for point_cloud in your_dataset:
      assert point_cloud.shape == (4096, 3), f"Invalid sample shape: {point_cloud.shape}"
  

  A single malformed sample with mismatched dimensions can cause the session to hang instead of throwing an explicit error.

3. TensorFlow Session Resource Leaks #

If you’re using raw TensorFlow sessions (not Keras wrappers), repeated sess.run() calls without proper cleanup can lead to leaks that cause freezes over epochs:

• Ensure all graph construction (layer definitions, loss functions, optimizers) happens once before starting the training loop — don’t redefine variables inside the loop.
• If you restart training runs without restarting your Python process, explicitly reset resources:

  sess.close()
  tf.reset_default_graph()
  

4. Adjust PointNet’s Hardcoded Point Count Logic #

PointNet is originally built for 2048 points, so hardcoded values in the network might break with 4096-point inputs:

• Check the farthest point sampling (fps) and sample_and_group functions. If they hardcode num_points = 2048, update this to match your input size or make it a configurable parameter.
• Verify that any layer expecting a fixed number of input points (like the initial feature extraction layers) is adjusted to accept 4096 points.

5. Isolate the Issue with Minimal Tests #

Narrow down the root cause by testing edge cases:

• Train with a batch size of 1 and a tiny subset of your data. If the freeze still occurs, the problem is likely in the network architecture or data formatting, not memory.
• Re-run the original PointNet code with (2048,3) data to confirm it works, then gradually switch to (4096,3) to pinpoint exactly when the freeze starts.

内容的提问来源于stack exchange，提问作者Aditya Khadilkar