关于spaCy生成非词汇词向量的原理及向量维度调整的技术咨询

Great questions! Let's unpack each one clearly:

When you feed random gibberish like fjoeij into spaCy, the way it produces a vector depends on the model you're using. For the en_core_web_sm model you're working with:

• It uses a hash embedding system. Every token you input gets a hash calculated, and that hash is used to look up a position in a pre-trained embedding table.
• For out-of-vocabulary (OOV) tokens (like your random strings), the hash maps to some entry in that table, which gives you the 96-dimensional vector you see. No fancy subword breakdown here—just a deterministic hash-to-vector mapping built into the small model.
• If you switched to a larger model like en_core_web_lg or en_core_web_trf, things change. Those models use subword segmentation (think Byte Pair Encoding) to split OOV tokens into smaller, known subword pieces. The final vector is then a combination of the vectors for those subwords.

This depends on whether you're using a pre-built spaCy model or rolling your own:

• Pre-trained spaCy models: No, you can't change the vector dimension. Each model's vector size is fixed during training—en_core_web_sm uses 96 dimensions, en_core_web_lg uses 300, and transformer-based models like en_core_web_trf match their underlying Transformer's output (e.g., 768 for BERT-base). These are hardcoded into the model's architecture.
• Custom models/embeddings: Yes, you absolutely can. Here's how:
  • Use the spacy init vectors command to create a custom vector table from your own corpus or external word vectors, specifying your desired dimension (like 128, 256, etc.). You can then train a spaCy model using this custom vector set.
  • When building a custom spaCy model from scratch, you can set the vectors_dim parameter in your config file to define the embedding dimension, then train the embedding layer alongside the rest of the model.

内容的提问来源于stack exchange，提问作者Feng Chen