Keras实现：将全连接层中特定权重设置为不可训练

Absolutely, you can pull this off in Keras! There are a couple of clean ways to implement a partially fixed weight matrix between your second hidden layer and output layer—here’s how to do it step by step:

Method 1: Custom Dense Layer with Fixed Zero Weights #

This approach encapsulates the fixed-weight logic into a custom layer, making your model code clean and easy to maintain. We’ll create a layer that splits weights into trainable and fixed (zero) parts based on a mask you define.

First, import the necessary libraries:

import tensorflow as tf
from tensorflow.keras.layers import Dense, Input
from tensorflow.keras.models import Model

Then define the custom layer:

class PartiallyFixedDense(Dense):
    def __init__(self, units, fixed_mask, **kwargs):
        self.fixed_mask = fixed_mask  # Boolean mask: True = trainable, False = fixed to 0
        super().__init__(units, **kwargs)
    
    def build(self, input_shape):
        # Initialize trainable weights
        self.kernel_trainable = self.add_weight(
            name='kernel_trainable',
            shape=(input_shape[-1], self.units),
            initializer=self.kernel_initializer,
            regularizer=self.kernel_regularizer,
            constraint=self.kernel_constraint,
            trainable=True
        )
        # Initialize fixed zero weights (non-trainable)
        self.kernel_fixed = tf.zeros_like(self.kernel_trainable, trainable=False)
        # Combine weights: use trainable values where mask is True, 0 otherwise
        self.kernel = tf.where(self.fixed_mask, self.kernel_trainable, self.kernel_fixed)
        super().build(input_shape)
    
    def call(self, inputs):
        # Use the combined weight matrix for computations
        output = tf.matmul(inputs, self.kernel)
        if self.use_bias:
            output = tf.nn.bias_add(output, self.bias)
        if self.activation is not None:
            output = self.activation(output)
        return output

Now build your model with this custom layer. Let’s say your second hidden layer has 10 neurons, the output layer has 5, and you want to fix specific weight positions to zero:

# Define your fixed-weight mask
# Example: fix positions (0,0), (1,0), (2,1), (3,2), (4,3) to zero
fixed_mask = tf.ones((10, 5), dtype=tf.bool)
fixed_mask = tf.tensor_scatter_nd_update(
    fixed_mask,
    indices=[[0,0], [1,0], [2,1], [3,2], [4,3]],
    updates=[False, False, False, False, False]
)

# Build the full model
inputs = Input(shape=(8,))  # Adjust input shape to match your data
hidden_layer_1 = Dense(10, activation='relu')(inputs)
hidden_layer_2 = Dense(10, activation='relu')(hidden_layer_1)
output_layer = PartiallyFixedDense(5, fixed_mask, activation='softmax')(hidden_layer_2)

model = Model(inputs=inputs, outputs=output_layer)
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

Method 2: Modify a Standard Dense Layer & Freeze Specific Weights #

If you prefer to work with a standard Dense layer, you can manually set specific weights to zero and override the training step to prevent those weights from updating.

Start by building a basic model with standard layers:

import tensorflow as tf
from tensorflow.keras.layers import Dense, Input
from tensorflow.keras.models import Model

# Build base model
inputs = Input(shape=(8,))
hidden_layer_1 = Dense(10, activation='relu')(inputs)
hidden_layer_2 = Dense(10, activation='relu')(hidden_layer_1)
output_layer = Dense(5, activation='softmax')
outputs = output_layer(hidden_layer_2)

model = Model(inputs=inputs, outputs=outputs)

Next, set the desired weights to zero:

# Define which weight positions to fix to zero
fixed_indices = [[0,0], [1,0], [2,1], [3,2], [4,3]]

# Get the current kernel weights of the output layer
kernel_weights = output_layer.kernel.numpy()

# Set specified positions to zero
for idx in fixed_indices:
    kernel_weights[idx[0], idx[1]] = 0

# Update the layer's weights with the modified values
output_layer.kernel.assign(kernel_weights)

Finally, create a custom training step to block gradient updates for the fixed positions:

@tf.function
def train_step(x, y):
    with tf.GradientTape() as tape:
        predictions = model(x, training=True)
        loss = model.compiled_loss(y, predictions, regularization_losses=model.losses)
    
    # Calculate gradients
    gradients = tape.gradient(loss, model.trainable_variables)
    
    # Zero out gradients for fixed weight positions
    for var, grad in zip(model.trainable_variables, gradients):
        if var.name == output_layer.kernel.name:
            # Create a mask to zero gradients at fixed positions
            grad_mask = tf.ones_like(grad)
            grad_mask = tf.tensor_scatter_nd_update(
                grad_mask,
                indices=fixed_indices,
                updates=[0.]*len(fixed_indices)
            )
            # Apply the mask to the gradient
            gradients[gradients.index(grad)] = grad * grad_mask
    
    # Apply modified gradients
    model.optimizer.apply_gradients(zip(gradients, model.trainable_variables))
    
    # Update metrics
    model.compiled_metrics.update_state(y, predictions)
    return {m.name: m.result() for m in model.metrics}

You can then use this train_step function in your training loop instead of the standard model.fit():

# Example training loop (adjust for your dataset)
epochs = 10
for epoch in range(epochs):
    print(f"Epoch {epoch+1}/{epochs}")
    for x_batch, y_batch in your_training_dataset:
        metrics = train_step(x_batch, y_batch)
    print(f"Loss: {metrics['loss'].numpy():.4f}, Accuracy: {metrics['accuracy'].numpy():.4f}")

Key Notes #

• The fixed_mask (Method 1) or fixed_indices (Method 2) lets you precisely control which weights stay zero and non-trainable.
• Method 1 is more scalable and cleaner for long-term use, while Method 2 is great for quick modifications to existing models.

内容的提问来源于stack exchange，提问作者user5016984