TensorFlow计算图修改致神经网络权重初始化变化问题咨询

Got it, let's tackle this problem step by step. I've dealt with similar reproducibility headaches in TF 1.x before, so here's exactly what's going wrong and how to fix it:

First, the root issue: your graph-level seed tf.set_random_seed(777) isn't enough on its own. TensorFlow's PRNG works with two layers: the graph-level seed you set, and per-operation seeds. When you don't explicitly define an operation-level seed, TensorFlow generates one dynamically based on the graph-level seed and the order that random operations are added to the graph.

If you modify your input pipeline, you're changing the sequence of operations added before your network's weight initializers. This shifts those dynamic operation seeds, leading to different weight initializations even with the same graph-level seed.

Here's how to lock in consistent initial weights:

1. Explicitly Assign Fixed Operation-Level Seeds to All Weight Initializers #

Instead of relying on the graph-level seed alone, give every weight initialization operation a fixed, hardcoded seed. This overrides dynamic seed generation, so the same weights are initialized no matter what changes you make to the input pipeline.

For example:

# ❌ Bad (no operation seed, relies on graph order)
weights = tf.get_variable("weights", shape=[32, 64], initializer=tf.contrib.layers.xavier_initializer())

# ✅ Good (fixed operation seed)
weights = tf.get_variable(
    "weights",
    shape=[32, 64],
    initializer=tf.contrib.layers.xavier_initializer(seed=123)  # Fixed seed here
)

# For tf.Variable with random_normal:
bias = tf.Variable(tf.random_normal([64], seed=456))  # Fixed seed for bias

Use a consistent sequence of seeds (like 123, 456, 789) for each unique weight variable, and keep these seeds identical across all input pipeline variants.

2. Isolate Network Definition from Input Pipeline Code #

Wrap your entire neural network layer and weight initialization logic in a separate function that you call after defining your input pipeline. This ensures the order of weight initialization operations in the graph stays identical every time, regardless of how the input pipeline is structured.

Example structure:

def build_core_network(inputs):
    # All weight initializers here have fixed operation seeds
    with tf.variable_scope("core_net"):
        dense1 = tf.layers.dense(
            inputs, 64, 
            kernel_initializer=tf.contrib.layers.xavier_initializer(seed=123)
        )
        dense2 = tf.layers.dense(
            dense1, 32, 
            kernel_initializer=tf.contrib.layers.xavier_initializer(seed=456)
        )
        output = tf.layers.dense(
            dense2, 10, 
            kernel_initializer=tf.contrib.layers.xavier_initializer(seed=789)
        )
    return output

# Variant 1: Input Pipeline A (placeholders)
with tf.Graph().as_default():
    tf.set_random_seed(777)
    inputs_a = tf.placeholder(tf.float32, shape=[None, 784])
    logits_a = build_core_network(inputs_a)
    
    # Verify initial weights
    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        weights_a = sess.run(tf.get_variable("core_net/dense/kernel"))
        print("Pipeline A initial weight sample:", weights_a[0][0])

# Variant 2: Input Pipeline B (tf.data)
with tf.Graph().as_default():
    tf.set_random_seed(777)
    # Different input pipeline structure
    dataset = tf.data.Dataset.from_tensor_slices(tf.random_normal([1000, 784]))
    iterator = dataset.make_initializable_iterator()
    inputs_b = iterator.get_next()
    logits_b = build_core_network(inputs_b)
    
    # Verify initial weights match
    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        sess.run(iterator.initializer)
        weights_b = sess.run(tf.get_variable("core_net/dense/kernel"))
        print("Pipeline B initial weight sample:", weights_b[0][0])

In this setup, weights_a and weights_b will be identical—because the network's weight initializers have fixed seeds, and their order in the graph is consistent relative to each other.

3. Keep Random Input Ops Separate (If Needed) #

If your input pipeline uses random operations (like random data augmentation), give those fixed seeds too (but use different seeds than your network weights). This prevents those input-side random ops from altering the sequence of your network's initialization operations.

Key Takeaway #

Graph-level seeds only guarantee reproducibility if the entire graph structure (operation order) is identical. By adding fixed operation-level seeds to all your network's weight initializers and isolating the network definition, you decouple weight initialization from input pipeline changes—ensuring you start with exactly the same weights every time you train.

内容的提问来源于stack exchange，提问作者Ferran Parés