TensorFlow 1.14中正交权重初始化报错求助

错误原因 #

tf.orthogonal_initializer()仅支持初始化至少2维的张量，但你的代码中所有偏置变量（b）都是1维的（比如shape=(hdim)、shape=(ddim)）。当你把变量作用域的默认初始化器替换成正交初始化器后，偏置变量会继承这个初始化器，从而触发ValueError。

修复方案 #

核心思路：只让权重变量使用正交初始化器，偏置变量使用适合1维张量的初始化器（比如零初始化、随机均匀初始化）。

修改后的代码示例 #

def __dictNN(self, x):
    # Parameters
    dim = self.__dim
    hdim = self.__hdim
    ddim = self.__ddim
    kmatdim = ddim + 1 + dim
    num_layers = self.__num_layers
    std = 1.0 / np.sqrt(hdim)
    std_proj = 1.0 / np.sqrt(dim)
    
    # 输入投影层：权重用正交初始化
    with tf.variable_scope("Input_projection"):
        P = tf.get_variable(name='weights',
                            shape=(dim,hdim),
                            dtype=tf.float64,
                            initializer=tf.orthogonal_initializer())
        res_in = tf.matmul(x, P)
    
    # 残差层：每层权重用正交初始化，偏置用零初始化
    with tf.variable_scope("Residual"):
        for j in range(self.__num_layers):
            layer_name = "Layer_"+str(j)
            with tf.variable_scope(layer_name):
                W = tf.get_variable(name="weights", shape=(hdim,hdim),
                                    dtype=tf.float64,
                                    initializer=tf.orthogonal_initializer())
                b = tf.get_variable(name="biases", shape=(hdim),
                                    dtype=tf.float64,
                                    initializer=tf.zeros_initializer())
                if j==0: # first layer
                    res_out = res_in + self.__tf_nlr(
                        tf.matmul(res_in, W) + b)
                else: # subsequent layers
                    res_out = res_out + self.__tf_nlr(
                        tf.matmul(res_out, W) + b)
    
    # 输出投影层：权重正交初始化，偏置用零初始化
    with tf.variable_scope("Output_projection"):
        W = tf.get_variable(name="weights", shape=(hdim, ddim),
                            dtype=tf.float64,
                            initializer=tf.orthogonal_initializer())
        b = tf.get_variable(name="biases", shape=(ddim),
                            dtype=tf.float64,
                            initializer=tf.zeros_initializer())
        out = tf.matmul(res_out, W) + b
    return out

另一种简化写法（作用域默认+单独覆盖） #

如果你想保留变量作用域的默认初始化器为正交，只需要给偏置单独指定初始化器：

with tf.variable_scope("Residual", initializer=tf.orthogonal_initializer()):
    for j in range(self.__num_layers):
        layer_name = "Layer_"+str(j)
        with tf.variable_scope(layer_name):
            W = tf.get_variable(name="weights", shape=(hdim,hdim),
                                dtype=tf.float64)  # 继承作用域的正交初始化
            b = tf.get_variable(name="biases", shape=(hdim),
                                dtype=tf.float64,
                                initializer=tf.zeros_initializer())  # 覆盖为零初始化
            # ... 后续代码不变

补充说明 #

• 正交初始化器的设计目的是保持梯度在深层网络中稳定，通常只用于权重矩阵（2维及以上），偏置一般用零初始化或小范围随机初始化即可。
• 如果你一定要给偏置用类似的初始化效果，可以把偏置的shape改成(hdim, 1)，但这会增加不必要的维度，不推荐。

内容的提问来源于stack exchange，提问作者Carlos