Hey there! Let's break down exactly what's happening with the Normalization layer's parameters in TensorFlow Keras, and how it differs from BatchNormalization.

参数数量的计算逻辑（对应你的观测） #

First, let's align with your observations:

• For an input shape of [None, 1] (1 feature per sample), you see 3 non-trainable parameters
• For [None, 9] (9 features per sample), you see 19 non-trainable parameters

This pattern follows the formula 2 * number_of_features + 1, which tells us the non-trainable parameters are made up of three parts:

1. Per-feature mean: 1 parameter for each input feature (1 or 9 total)
2. Per-feature standard deviation (or inverse std): 1 parameter for each input feature (1 or 9 total)
3. Global offset: A single fixed parameter (this is likely an implementation detail from the older experimental version of the layer you're using; modern TensorFlow versions use per-feature trainable offsets instead)

In current mainstream TensorFlow versions (2.10+), the Normalization layer's non-trainable parameters are only the per-feature mean and variance (total 2 * number_of_features), with additional trainable parameters for per-feature offsets and scaling factors (another 2 * number_of_features if enabled). But regardless of version, the core logic centers on mean, std, and optional adjustments.

What These Parameters Actually Mean #

Each non-trainable parameter ties directly to the standard Z-score normalization process:

• Mean: Calculated from your training data via the adapt() method, this shifts each feature's distribution to be centered around 0 (x - mean).
• Standard Deviation: Also computed during adapt(), this scales each feature to have a variance of 1 ((x - mean) / std).
• Global Offset (if present): A fixed shift applied after normalization to adjust the overall distribution of the output, though modern versions let you use trainable per-feature offsets instead.

Crucially, these mean and std values are fixed once computed—they don't update during model training, which is a key difference from BatchNormalization.

Theoretical Background #

The Normalization layer implements offline feature normalization (Z-score normalization), a staple preprocessing technique in machine learning. The formula is straightforward:
$$x_{normalized} = \frac{x - \mu}{\sigma}$$
where $\mu$ is the feature's mean and $\sigma$ is its standard deviation.

This method eliminates scale differences between features, helping neural networks converge faster and more reliably. It's a fundamental statistical technique rather than a novel deep learning innovation, so it's covered in most machine learning textbooks (like Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow) and introductory statistics resources, rather than a single "reference paper."

Key Differences from BatchNormalization #

You're right that BatchNormalization has a completely different parameter logic—here's the breakdown:

1. Statistic Calculation:
   • Normalization: Computes mean/std once from your full training dataset (via adapt()) and keeps them fixed.
   • BatchNormalization: Computes mean/std per training batch during training, using a moving average to update global stats for inference.
2. Parameter Type:
   • Normalization: Non-trainable parameters are precomputed mean/std; trainable parameters (if enabled) are optional offsets/scales.
   • BatchNormalization: Has 4 parameters per feature: trainable scale ($\gamma$) and offset ($\beta$), plus non-trainable moving average mean and variance.
3. Use Case:
   • Normalization: Best for preprocessing to fix data distribution before training, especially when you have a large, stable dataset.
   • BatchNormalization: Designed to dynamically stabilize layer inputs during training, reducing internal covariate shift in deep neural networks.

Quick Code Examples #

To reproduce your observations (using an older experimental layer version):

import tensorflow as tf

# 1D input example
norm_layer_1d = tf.keras.layers.experimental.preprocessing.Normalization(input_shape=(1,))
norm_layer_1d.adapt(tf.random.normal((1000, 1)))
model_1d = tf.keras.Sequential([norm_layer_1d])
model_1d.summary()  # Shows 3 non-trainable params in older versions

# 9D input example
norm_layer_9d = tf.keras.layers.experimental.preprocessing.Normalization(input_shape=(9,))
norm_layer_9d.adapt(tf.random.normal((1000, 9)))
model_9d = tf.keras.Sequential([norm_layer_9d])
model_9d.summary()  # Shows 19 non-trainable params in older versions

内容的提问来源于stack exchange，提问作者uom0