FastText Model
Relevant source files
The following files were used as context for generating this wiki page:
The FastText model in Gensim is an implementation of the word representation learning method described in "Enriching Word Vectors with Subword Information." Its primary purpose is to learn word embeddings from a training corpus while providing the additional capability to obtain vector representations for out-of-vocabulary (OOV) words by leveraging subword information (character n-grams).
The module includes a fast native C implementation with Python interfaces, rather than simply wrapping external binaries. It maintains full compatibility with Facebook's original FastText implementation, supporting the loading of .bin files and continuing training from pretrained models.
Sources: gensim/models/fasttext.py:14-23, gensim/models/fasttext.py:236-240
Architecture and Components
The FastText implementation extends the Word2Vec architecture but introduces specialized sub-components to handle character n-grams and bucket-based hashing.
Class Hierarchy
The relationship between the core classes is structured as follows:
The FastText class handles the training logic and overall model state, while FastTextKeyedVectors manages the word and n-gram vectors.
Sources: gensim/models/fasttext.py:234, gensim/models/fasttext.py:848, gensim/models/word2vec.py:129
Component Overview
| Component | Description |
|---|---|
FastText | The main class for training and model persistence. Inherits from Word2Vec. |
FastTextKeyedVectors | Specialized container for word and subword vectors. Handles OOV inference. |
vectors_vocab | Learned vectors for full-word tokens in the vocabulary. |
vectors_ngrams | Learned vectors for character n-gram buckets. |
buckets_word | A cache mapping each vocabulary word to its respective n-gram bucket indices. |
Sources: gensim/models/fasttext.py:234-315, gensim/models/fasttext.py:850-900
Logic and Data Flow
Training Flow
Training involves scanning the corpus, building a vocabulary, and optimizing weights using either Skip-gram or CBOW architectures.
- Vocabulary Building: The model scans the corpus to count word frequencies and identifies unique tokens.
- Subword Generation: For each word, character n-grams are generated between
min_nandmax_nlengths. - Hashing: These n-grams are hashed into a fixed number of buckets (
bucket). - Weight Initialization: Vectors are initialized for both words and n-gram buckets.
- Optimization: During training, updates are applied to both word vectors and the constituent n-gram vectors.
- Adjustment: After training,
adjust_vectors()composes the final word vectors from their token vectors and n-gram averages.
Sources: gensim/models/fasttext.py:317-380, gensim/models/fasttext.py:1003-1025, gensim/models/word2vec.py:465-470
Vector Inference for OOV Words
When a word is not found in the vocabulary, FastText synthesized a vector by averaging the vectors of its character n-grams.
Sources: gensim/models/fasttext.py:970-1001
Configuration and Parameters
FastText shares many parameters with Word2Vec but includes specific settings for subword information.
| Parameter | Type | Default | Description |
|---|---|---|---|
vector_size | int | 100 | Dimensionality of the word vectors. |
min_n | int | 3 | Minimum length of char n-grams. |
max_n | int | 6 | Maximum length of char n-grams. |
bucket | int | 2,000,000 | Number of buckets for hashing n-grams. |
sg | int | 0 | 1 for Skip-gram, 0 for CBOW. |
hs | int | 0 | 1 to use Hierarchical Softmax. |
negative | int | 5 | Number of noise words for negative sampling. |
word_ngrams | int | 1 | Max length of word n-grams (Gensim supports 1 only). |
Sources: gensim/models/fasttext.py:334-390, gensim/models/fasttext.py:440-450
Model Persistence and Facebook Compatibility
Gensim provides specialized functions to interact with the original Facebook FastText format.
Loading Facebook Models
Developers can load .bin files directly using load_facebook_model. This process reconstructs the internal state of the model to allow for continued training.
from gensim.models.fasttext import load_facebook_model
fb_model = load_facebook_model('path/to/model.bin')
Sources: gensim/models/fasttext.py:688-720
KeyedVectors Only
If training is not required, load_facebook_vectors can be used to load only the embeddings, which significantly reduces memory consumption.
Sources: gensim/models/fasttext.py:749-760
Performance and Cython Implementation
The project utilizes Cython to optimize the inner loops of the training process. C and C++ extensions are compiled for performance-critical components.
gensim.models.fasttext_inner: Contains optimized C routines for batch training and n-gram hashing.gensim.models.fasttext_corpusfile: Implements multi-threaded training directly from files to avoid Python Global Interpreter Lock (GIL) bottlenecks.
Sources: setup.py:20-33, gensim/models/fasttext.py:220-230
Conclusion
The FastText model in Gensim provides a robust and high-performance implementation for subword-enhanced word embeddings. By inheriting from the mature Word2Vec architecture and integrating optimized Cython components, it offers an efficient solution for large-scale NLP tasks, particularly those involving morphologically rich languages or frequent out-of-vocabulary terms.