TensorFlow迭代耗时递增问题求助及tf.reset_default_graph()报错咨询

Hey there, let's break down your problems one by one—first that confusing graph mismatch error, then the frustrating slowdown across iterations.

1. Fixing the tf.reset_default_graph() Error #

That error message is telling you exactly what's off: you're trying to use tensors from two separate TensorFlow computation graphs together. When you call tf.reset_default_graph(), you wipe the current default graph and create a brand new empty one. If you then mix tensors created before the reset (like that Const:0 tensor) with new tensors made after the reset (the softmax cross entropy tensor), TensorFlow throws this mismatch error because they don't belong to the same graph context.

Here's how to fix this:

• Stop calling tf.reset_default_graph() inside your training loop. This command is meant for resetting graphs between separate training runs (e.g., training multiple models back-to-back), not during a single training process.
• Keep all graph-related code in one place. Define your model, loss function, optimizer, and placeholders once before starting your iteration loop, all within the same default graph. For example:

  # Define all graph components FIRST, outside the loop
  tf.reset_default_graph()  # Only use this if you're starting a fresh training run
  
  # Create input placeholders
  keep_probability = tf.placeholder(tf.float32)
  input_M = tf.placeholder(tf.float32, shape=(your_input_shape))
  V_a = tf.placeholder(tf.float32, shape=(your_Va_shape))
  L = tf.placeholder(tf.int32, shape=(your_L_shape))
  
  # Build your model once
  pred_matrix = ModelA(keep_probability, input_M, V_a, L)
  
  # Define loss and optimizer
  loss = tf.nn.softmax_cross_entropy_with_logits(...)  # Match your actual loss setup
  optimizer = tf.train.AdamOptimizer(learning_rate=0.001).minimize(loss)
  
  # Now run your training loop
  with tf.Session() as sess:
      sess.run(tf.global_variables_initializer())
      for iteration in range(20):
          # Feed data and run operations
          _, current_loss = sess.run(
              [optimizer, loss],
              feed_dict={
                  keep_probability: 0.5,
                  input_M: your_batch_input,
                  V_a: your_batch_Va,
                  L: your_batch_L
              }
          )
          print(f"Iteration {iteration+1}: Loss = {current_loss}")
  

• Never mix tensors from different graphs. If you do need to reset the graph (for multiple sequential training runs), make sure you redefine every single tensor, model component, and operation from scratch after the reset.

2. Fixing the Iteration Slowdown #

Your first iteration taking 5s and the 20th taking 50s is a classic sign of a growing computation graph—meaning you're adding new operations to the graph every iteration, making it larger and slower to run. The most likely cause and fix:

• You're redefining your model (ModelA) inside the training loop. If you call ModelA() every iteration, you're creating new matmul, reshape, and other operations each time, which pile up in the graph. This makes each subsequent run take longer as TensorFlow has to process more nodes.
  • Fix: Move the ModelA() call outside the loop, as shown in the code example above. Define your model once, then just run the optimizer/loss in the loop.
• Other possible causes to check:
  • Memory leaks: Monitor your GPU/CPU memory usage—if it climbs steadily with each iteration, ensure you're not holding onto unnecessary tensors or data in your loop.
  • Data loading bottlenecks: If your data loading code is inefficient (e.g., reloading data every iteration instead of preloading batches), it could slow down over time. Optimize your data pipeline to load batches smoothly.
  • Graph bloat: Use print(len(tf.get_default_graph().get_operations())) at the start and end of each iteration. If the number of operations increases, you're adding nodes to the graph—fix this by moving all graph definitions outside the loop.

If you're still stuck after these fixes, you can use TensorFlow's profiler (tf.profiler) to pinpoint exactly which parts of your code are taking the most time.

内容的提问来源于stack exchange，提问作者jv3768