Doc2Vec Model
Relevant source files
The following files were used as context for generating this wiki page:
The Doc2Vec model, also known as Paragraph Vector, is an unsupervised algorithm that learns fixed-length feature representations from variable-length pieces of texts, such as sentences, paragraphs, or documents. In Gensim, this is implemented via the Doc2Vec class, which extends the Word2Vec Model functionality to include document-level embeddings. It allows for representing documents in a continuous vector space where semantically similar documents are positioned close to one another.
Sources: gensim/models/doc2vec.py:1-60, README.md:14-25
Architecture and Training Modes
The Doc2Vec implementation supports several distinct training modes that determine how document and word vectors are learned and combined during training.
Distributed Memory (PV-DM)
In Distributed Memory mode (dm=1), the model attempts to predict the next word in a context using both the context words and a unique document identifier (document vector). This acts as a memory of what is missing from the current context.
- DM/Mean: (
dm_mean=1) Context word vectors are averaged. - DM/Sum: (
dm_mean=0) Context word vectors are summed. - DM/Concatenate: (
dm_concat=1) Context word vectors and the document vector are concatenated, resulting in a larger input layer.
Distributed Bag of Words (PV-DBOW)
In Distributed Bag of Words mode (dm=0), the model ignores the context words in the input and instead forced the model to predict words randomly sampled from the document using only the document vector. This mode is computationally faster and requires less storage.
Sources: gensim/models/doc2vec.py:81-160, gensim/test/test_doc2vec.py:380-450
Training Flow Diagram
The following diagram illustrates the high-level data flow during a Doc2Vec training cycle.
This diagram represents how the input corpus is processed through vocabulary building and weight initialization before entering specific training paths based on the dm parameter.
Sources: gensim/models/doc2vec.py:643-730, gensim/models/word2vec.py:460-510
Key Components and Data Structures
Classes and Iterators
| Component | Description |
|---|---|
Doc2Vec | The primary model class containing the training logic and vector storage. |
TaggedDocument | A single document, represented as a list of words and a list of tags (labels). |
DocvecsArray (Internal) | Specifically handles the storage and retrieval of document vectors in self.dv. |
Sources: gensim/models/doc2vec.py:61-75, gensim/models/doc2vec.py:1335-1350
Parameter Configuration
The Doc2Vec constructor accepts numerous parameters that tune performance and memory usage.
| Parameter | Type | Default | Description |
|---|---|---|---|
vector_size | int | 100 | Dimensionality of the feature vectors. |
dm | int {0, 1} | 1 | Defines training algorithm: 1 for PV-DM, 0 for PV-DBOW. |
window | int | 5 | Max distance between current and predicted word. |
min_count | int | 5 | Ignores words with total frequency lower than this. |
hs | int {0, 1} | 0 | If 1, hierarchical softmax is used. |
negative | int | 5 | If > 0, negative sampling is used with this many noise words. |
epochs | int | 10 | Number of iterations over the corpus. |
Sources: gensim/models/doc2vec.py:643-780
Vector Operations
Inference of New Documents
The model provides the infer_vector method to estimate a vector for a previously unseen document. This process keeps the underlying model weights fixed and performs a number of training steps (gradient descent) to find the document vector that best fits the existing word embeddings.
# Example of vector inference
from gensim.models.doc2vec import Doc2Vec
model = Doc2Vec.load("my_model.d2v")
new_vector = model.infer_vector(["this", "is", "a", "new", "document"])
Sources: gensim/models/doc2vec.py:1001-1050, gensim/test/test_doc2vec.py:534-545
Similarity Search
Document similarity is handled via the self.dv (Document Vectors) attribute, which is a KeyedVectors instance.
most_similar(): Finds the top-N most similar documents (by tag) or vectors.similarity(): Computes the cosine similarity between two specific document tags.
Sources: gensim/models/doc2vec.py:410-450, gensim/test/test_doc2vec.py:548-570
Model Persistence
Doc2Vec models can be saved and loaded using Gensim's standard SaveLoad protocol. Because document vectors can be very large, they are often saved in separate files using memory-mapping (mmap) to allow fast loading and sharing across processes.
Saving Options
| Feature | Implementation |
|---|---|
| Native Save | model.save(fname) stores the entire object state. |
| Word2Vec Format | model.save_word2vec_format() saves vectors in the original C format. |
| Separate Arrays | sep_limit can force large internal arrays into separate files. |
Sources: gensim/models/doc2vec.py:930-970, setup.py:20-30, gensim/test/test_doc2vec.py:50-80
Performance Optimizations
The implementation leverages Cython to achieve high training speeds. Specific C++ extensions are used for the inner loops of the training algorithms.
| Extension Name | Source File |
|---|---|
gensim.models.doc2vec_inner | gensim/models/doc2vec_inner.cpp |
gensim.models.doc2vec_corpusfile | gensim/models/doc2vec_corpusfile.cpp |
Sources: setup.py:28-35, gensim/models/word2vec.py:105-125
Conclusion
The Doc2Vec model in Gensim provides a robust framework for document-level embedding. By extending the Word2Vec architecture with the Paragraph Vector concept, it enables sophisticated NLP tasks such as document classification, clustering, and semantic retrieval, all while maintaining memory efficiency through streaming APIs and Cython-optimized routines.