基于TensorFlow计算向量组合范数及解决余弦范数溢出问题

Hey there! Let's work through your problem step by step—covering how to compute pairwise norms (L2 and cosine), fix that inf issue you ran into, and set up a way to enforce the proximity rules for your tr embeddings.

Your data is structured as 13×13 grids, each with 5×100-dimensional embeddings. To compute pairwise norms between all embeddings across your two batches, you first need to flatten this nested structure into a simple list of embeddings.

Assuming each batch has a shape like (13, 13, 5, 100) (one sequence per batch), here's how to flatten them in TensorFlow:

import tensorflow as tf

# Example batch data (replace with your actual tensors)
batch1 = tf.random.normal(shape=(13, 13, 5, 100))
batch2 = tf.random.normal(shape=(13, 13, 5, 100))

# Flatten to (total_embeds, 100) — total_embeds = 13*13*5 = 845
flat_batch1 = tf.reshape(batch1, (-1, 100))
flat_batch2 = tf.reshape(batch2, (-1, 100))

Calculating pairwise L2 distances is straightforward with TensorFlow's tf.norm. This will give you a matrix where each element [i,j] is the L2 distance between the i-th embedding in batch1 and j-th embedding in batch2:

# Shape: (845, 845) — pairwise L2 distances between all embeddings
pairwise_l2 = tf.norm(flat_batch1[:, None, :] - flat_batch2[None, :, :], axis=-1)

Your cosine norm calculation was hitting inf because you likely had an embedding with a zero L2 norm, leading to division by zero. To fix this, add a tiny epsilon value to the denominator to avoid division by zero.

Here's how to compute valid pairwise cosine similarity (and distance, if you prefer):

epsilon = 1e-8  # Tiny value to prevent division by zero

# Compute L2 norms for each embedding (add epsilon to avoid zeros)
norm_batch1 = tf.norm(flat_batch1, axis=-1, keepdims=True) + epsilon
norm_batch2 = tf.norm(flat_batch2, axis=-1, keepdims=True) + epsilon

# Compute dot product between all pairs of embeddings
dot_products = tf.matmul(flat_batch1, flat_batch2, transpose_b=True)

# Pairwise cosine similarity (range: [-1, 1])
cosine_similarity = dot_products / tf.matmul(norm_batch1, norm_batch2, transpose_b=True)

# If you need cosine distance (range: [0, 2]), use:
cosine_distance = 1 - cosine_similarity

This will eliminate any inf values because we've guaranteed the denominator is never zero.

To make tr embeddings close to each other and far from non-tr embeddings, you'll need to define a loss function that rewards the desired behavior. First, use a mask to separate tr and non-tr embeddings:

# Example mask: Assume `tr_mask` is a boolean tensor of shape (845,)
# (Replace this with your actual logic to identify `tr` embeddings)
tr_mask = tf.random.uniform(shape=(845,)) < 0.1  # Randomly pick 10% as `tr`

# Separate `tr` and non-`tr` embeddings
tr_embeds1 = tf.boolean_mask(flat_batch1, tr_mask)
tr_embeds2 = tf.boolean_mask(flat_batch2, tr_mask)
non_tr_embeds1 = tf.boolean_mask(flat_batch1, ~tr_mask)
non_tr_embeds2 = tf.boolean_mask(flat_batch2, ~tr_mask)

# Loss 1: Minimize distances between `tr` embeddings (make them close)
tr_proximity_loss = tf.reduce_mean(tf.norm(tr_embeds1[:, None, :] - tr_embeds2[None, :, :], axis=-1))

# Loss 2: Maximize distances between `tr` and non-`tr` embeddings
# We use a threshold (e.g., 5.0) — if distance is below this, add penalty
tr_separation_loss = tf.reduce_mean(tf.maximum(0.0, 5.0 - tf.norm(tr_embeds1[:, None, :] - non_tr_embeds2[None, :, :], axis=-1)))

# Total loss to optimize
total_loss = tr_proximity_loss + tr_separation_loss

When you optimize total_loss, your model will learn to push tr embeddings closer together and pull them away from non-tr embeddings.

内容的提问来源于stack exchange，提问作者mosnoi ion