TensorFlow中Skip-gram版Word2Vec实现困惑及方案咨询

Let me break down your confusion step by step and clarify the correct way to implement skip-gram:

Core Skip-Gram Logic #

Yes, skip-gram does aim to predict C context words (each from a vocabulary of size V) given a central word. But here's the key point: each context word is an independent classification task—you don't need to concatenate their one-hot vectors. Instead, you treat each context prediction as a separate (but related) loss term, then combine all those losses to optimize the shared embedding parameters.

Why Your Initial Approaches Have Issues #

1. Concatenating C*V vectors: This approach misframes the problem. Softmax is designed to output a probability distribution over V classes for a single prediction. Concatenating C one-hot vectors turns it into a misaligned multi-output task that doesn't match skip-gram's goal of predicting each context word individually.

2. Using C separate optimizers: This is unnecessary and inefficient. All your context prediction tasks are optimizing the same embedding matrices (input and output layers). Using multiple optimizers would lead to redundant parameter updates and potential conflicts. Instead, you should sum all individual losses into a total loss and use one optimizer to minimize that total.

Correct TensorFlow Implementation Steps #

Here's a simplified example using TensorFlow 1.x syntax (aligned with your mention of sessions):

1. Define Inputs #

Assume you have:

• center_words: Batch of central word indices (shape [batch_size])
• context_words: List of C tensors, each being a batch of context word indices (each shape [batch_size])
• vocab_size = V, embedding_size = N (size of your word vectors)

2. Build Embedding Layers #

import math
import tensorflow as tf

# Input embedding matrix (V x N)
embeddings = tf.Variable(tf.random_uniform([vocab_size, embedding_size], -1.0, 1.0))
# Look up embedding for central word: shape [batch_size, embedding_size]
center_embed = tf.nn.embedding_lookup(embeddings, center_words)

# Output weight matrix (N x V)
output_weights = tf.Variable(tf.truncated_normal([vocab_size, embedding_size], stddev=1.0 / math.sqrt(embedding_size)))
output_biases = tf.Variable(tf.zeros([vocab_size]))

3. Calculate Combined Loss #

total_loss = 0.0
for context in context_words:
    # Compute logits for this context: [batch_size, vocab_size]
    logits = tf.matmul(center_embed, tf.transpose(output_weights)) + output_biases
    # Sparse cross-entropy (uses word indices instead of one-hot vectors)
    loss = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=context, logits=logits)
    total_loss += tf.reduce_mean(loss)

# Average loss across all context windows
total_loss /= len(context_words)

4. Optimize with a Single Optimizer #

optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.1).minimize(total_loss)

# Training loop in session
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for epoch in range(num_epochs):
        # Fetch batch data (replace with your data pipeline)
        batch_center, batch_contexts = get_training_batch()
        # Prepare feed dict for all context tensors
        feed_dict = {center_words: batch_center}
        for i in range(len(context_words)):
            feed_dict[context_words[i]] = batch_contexts[i]
        # Run optimizer and track loss
        _, current_loss = sess.run([optimizer, total_loss], feed_dict=feed_dict)

Key Optimization for Large Vocabularies #

When your vocabulary size V is large (which it almost always is), full softmax becomes computationally expensive. Instead, use negative sampling to approximate the loss—this is the standard approach in real-world word2vec implementations:

# Replace per-context loss calculation with negative sampling
loss = tf.nn.nce_loss(
    weights=output_weights,
    biases=output_biases,
    labels=tf.expand_dims(context, 1),
    inputs=center_embed,
    num_sampled=5,  # Number of negative samples to use (adjust based on your data)
    num_classes=vocab_size
)
total_loss += tf.reduce_mean(loss)

Quick Recap #

• Skip-gram predicts each context word independently, not as a concatenated vector.
• Combine all individual context losses into a single total loss.
• Use one optimizer to minimize this total loss—multiple optimizers are unnecessary and counterproductive.
• For large vocabularies, use negative sampling instead of full softmax to speed up training.

内容的提问来源于stack exchange，提问作者de-loke