Document Similarity Queries

Relevant source files

The following files were used as context for generating this wiki page:

• gensim/similarities/docsim.py
• gensim/interfaces.py
• gensim/test/test_similarities.py
• gensim/test/simspeed.py
• gensim/examples/dmlcz/gensim_xml.py
• README.md

The Document Similarity Queries system in Gensim provides efficient retrieval of similar documents from a large corpus using the Vector Space Model. It allows users to build an index of documents and subsequently query that index with a new document (or an entire corpus) to determine which existing documents are most similar, typically using cosine similarity or specialized distance metrics like Word Mover's Distance (WMD).

The system is designed to be memory-independent, handling datasets larger than the available RAM through a sharding mechanism. It offers several specialized indexing classes suited for different data structures (dense vs. sparse) and memory constraints, ensuring high-performance BLAS-backed matrix operations for similarity computations.

Sources: gensim/similarities/docsim.py:12-42, README.md:14-17, gensim/interfaces.py:241-255

Core Architecture and Interfaces

All similarity retrieval modules inherit from a common abstract base class that defines the protocol for indexing and querying.

SimilarityABC

The SimilarityABC class defines the standard interface for similarity search. It requires subclasses to implement get_similarities(doc), which calculates the similarity of a query document against the built index. The interface supports:

• Single Querying: Finding similarities for one document.
• Batch Querying: Processing an entire corpus of queries at once for better performance.
• Normalization: Automatically applying unit vector normalization to queries if configured.
• Top-N Results: Optionally returning only the num_best results rather than a full similarity vector.

Sources: gensim/interfaces.py:241-325

Similarity Query Flow

The following diagram illustrates the high-level flow of a similarity query through the system:

The diagram shows the logic within SimilarityABC.__getitem__ which handles input normalization and result clipping.
Sources: gensim/interfaces.py:268-300

Scalable Indexing with Sharding

For large-scale applications where the index does not fit in memory, the Similarity class implements a disk-based sharding strategy.

The Similarity Class

This class acts as a coordinator that splits a massive index into several smaller sub-indexes called "shards." Each shard is a standalone index file stored on disk and mmap'ed into memory only when needed during a query.

• Sharding Logic: When documents are added via add_documents(), they are buffered in memory. Once the buffer reaches shardsize, it is converted into a matrix (dense or sparse) and written to disk as a new shard.
• Parallel Processing: The system can optionally parallelize queries across shards using the multiprocessing module if PARALLEL_SHARDS is enabled.
• Dynamic Indexing: Unlike static matrix implementations, the Similarity class supports adding new documents to the index dynamically.

Sources: gensim/similarities/docsim.py:202-315, gensim/similarities/docsim.py:15-19

Shard Management

The Similarity class uses Shard proxies to interact with disk-based sub-indexes.
Sources: gensim/similarities/docsim.py:84-142, gensim/similarities/docsim.py:149-200

Specialized Indexing Classes

Gensim provides specific implementations based on the nature of the document vectors and the similarity metrics required.

Class	Use Case	Implementation Detail
MatrixSimilarity	Dense vectors, index fits in RAM.	Uses a 2D numpy.ndarray.
SparseMatrixSimilarity	Sparse vectors (e.g., BoW, TF-IDF), index fits in RAM.	Uses scipy.sparse.csr_matrix.
SoftCosineSimilarity	Similarity considering term relations.	Requires a SparseTermSimilarityMatrix.
WmdSimilarity	Similarity based on Word Mover's Distance.	Requires a pre-trained KeyedVectors model.

Sources: gensim/similarities/docsim.py:440-445, gensim/similarities/docsim.py:702-740, gensim/similarities/docsim.py:539-555, gensim/similarities/docsim.py:644-665

MatrixSimilarity vs. SparseMatrixSimilarity

The choice between dense and sparse indexing is critical for performance. MatrixSimilarity is optimized for dense vectors (like LSI or RP) using BLAS dot products. SparseMatrixSimilarity is designed for high-dimensional sparse data like raw Bag-of-Words, where it maintains memory efficiency by avoiding the storage of zero entries.

Sources: gensim/similarities/docsim.py:441-450, gensim/similarities/docsim.py:734-738

Similarity Computation Logic

The similarity result depends on the input query type and the configuration of the indexer.

Sequence of Operations during __getitem__

1. Shard Closure: If any documents were recently added to a Similarity index, the current "fresh" shard is flushed to disk.
2. Normalization: If index.norm is set (e.g., 'l2'), the query vector is normalized to unit length.
3. Cross-Shard Querying: The query is sent to each shard.
4. Result Aggregation:
   • If num_best is None: Results from all shards are concatenated using numpy.hstack.
   • If num_best is set: The system uses a heap-based _nlargest function to extract the top matches across all shards, applying necessary offsets to document IDs.

Sources: gensim/similarities/docsim.py:348-400, gensim/similarities/docsim.py:168-178

Top-N Retrieval Sequence

The sequence shows how Similarity aggregates top-N results from multiple independent shards.
Sources: gensim/similarities/docsim.py:356-390

Performance Optimization

Gensim employs several strategies to maximize query throughput:

• Chunking: When querying with a corpus, documents are processed in chunks (defined by chunksize). This allows for large matrix-matrix multiplications which are significantly faster than multiple matrix-vector multiplications.
• Mmap: Shards are loaded using memory-mapping (mmap='r'), allowing the OS to manage memory demand and sharing.
• BLAS Integration: Low-level operations are delegated to highly optimized Fortran/C libraries like MKL or OpenBLAS via NumPy.

Sources: gensim/similarities/docsim.py:31-38, gensim/similarities/docsim.py:126-128, README.md:75-80

Summary

The Gensim Document Similarity Queries system provides a flexible, scalable framework for retrieving related documents. By abstracting the complexities of memory management and parallelization via the Similarity and Shard classes, it allows developers to perform high-speed similarity searches on corpora that exceed physical memory limits while maintaining a simple, Pythonic API.

Sources: gensim/similarities/docsim.py:20-40, gensim/interfaces.py:241-250