关于FastText算法输入单元及学习流程示例的技术问询

Hey there! Let's break down your two questions about FastText clearly:

1. FastText's Input Units: Words/Subwords vs. Sentences #

It depends on which task you're using FastText for:

• When training word vectors, the core input units are words and their decomposed subwords. The model learns word representations where each word's vector is the sum of its own unique vector plus all the vectors of its subwords.
• When doing text classification, FastText fully supports taking an entire sentence as input. It computes a sentence-level vector by averaging (or summing) the vectors of all words in the sentence (each word's vector already incorporates subword info), then feeds this vector into a classifier layer to predict the label.

2. Step-by-Step FastText Learning Procedure Example #

Let's use your sample sentence and parameters to walk through this, comparing it to Word2Vec as you requested:

I go to school with lunch box that my mother wrapped every morning

We'll use the CBOW mode (matching the Word2Vec CBOW logic you mentioned) with a window size of 2, and set subword n-grams to length 3 (we'll also include full words and special boundary markers, which FastText uses by default).

Step 1: Preprocess Words & Generate Subwords #

FastText adds special < (start) and > (end) markers to each word to capture boundary information. For the target word school, we first convert it to <school>, then extract all 3-gram subwords:

• <sc
• sch
• cho
• hoo
• ool
• ol>
  Plus, the full processed word <school> is kept as a special subword (to preserve the full word's semantic identity).

Step 2: Context Window & Vector Representations #

Just like Word2Vec CBOW, with a window size of 2, the context words for target school are ['go', 'to', 'with', 'lunch']. Here's where FastText differs:
Each context word's vector isn't a single unique vector—it's the sum of its own unique vector plus all the vectors of its subwords.
For example, take the context word to:

• Process it to <to>
• Extract 3-gram subwords: <to, to>
• Its vector = <to> (full word vector) + <to (subword vector) + to> (subword vector)

Step 3: CBOW Prediction & Training #

The overall flow mirrors Word2Vec CBOW, but with subword-aware vectors:

1. Compute the average of all context word vectors (each being the sum of word + subword vectors) to get a single context aggregate vector.
2. Use this aggregate vector to predict the target word school. The target's representation is also the sum of its own vector plus all its subword vectors.
3. During backpropagation, the model updates all relevant vectors: the unique vectors of context/target words, and all the subword vectors associated with them.

Key Difference from Word2Vec #

Word2Vec assigns one unique vector per word. FastText's word vectors are sums of multiple vectors, which means:

• Out-of-vocabulary (OOV) words can still get meaningful vectors by combining their subwords (as long those subwords were seen during training).
• Morphologically similar words (like school and schools) share subword vectors, leading to more semantically aligned representations.

内容的提问来源于stack exchange，提问作者Isaac Sim