Term Similarity Measures

Relevant source files

The following files were used as context for generating this wiki page:

• gensim/similarities/docsim.py
• gensim/test/test_similarities.py
• gensim/topic_coherence/direct_confirmation_measure.py
• gensim/test/test_coherencemodel.py
• setup.py
• README.md

Term Similarity Measures in Gensim provide a suite of tools for calculating the relatedness between individual tokens or terms. These measures are foundational for higher-level algorithms like Soft Cosine Similarity and various topic coherence evaluations. The system enables developers to represent relationships between terms beyond simple identity, accounting for semantic or lexical proximity.

The architecture leverages several specialized indices and matrices to handle term-to-term comparisons efficiently. These include indices based on word embeddings (Word2Vec, FastText), lexical edit distances (Levenshtein), and statistical co-occurrence measures (PMI, NPMI) used in topic modeling diagnostics.

Architecture and Components

The term similarity system is organized around "Similarity Indices" which define how two terms are compared, and "Similarity Matrices" which organize these comparisons for use in vector space models.

Term Similarity Indices

Gensim provides several index types to compute similarities between specific terms:

• UniformTermSimilarityIndex: A baseline index that returns a constant similarity for all pairs.
• WordEmbeddingSimilarityIndex: Uses pre-trained vectors (like Word2Vec) to compute cosine similarity between term embeddings.
• LevenshteinSimilarityIndex: Calculates lexical similarity based on the Levenshtein edit distance between strings.

SparseTermSimilarityMatrix

The SparseTermSimilarityMatrix is a core component that encapsulates the relationships between all terms in a given dictionary. It is primarily used to perform "Soft Cosine Similarity," allowing for the comparison of documents that do not share the exact same vocabulary but share semantically related terms.

Logic and Mathematical Measures

Statistical term similarities are extensively used in topic coherence models to validate the quality of generated topics.

Co-occurrence Measures

The system implements several measures based on term probability and co-occurrence:

Measure	Formula Description	Use Case
Log Conditional Probability	$log \frac{P(W', W^{}) + \epsilon}{P(W^{})}$	Used by $U_{mass}$ coherence.
PMI (Log Ratio)	$log \frac{P(W', W^{}) + \epsilon}{P(W') * P(W^{})}$	Pointwise Mutual Information.
NPMI	$\frac{PMI}{-log(P(W', W^{*}) + \epsilon)}$	Normalized PMI for $c_v$ coherence.

Sources: gensim/topic_coherence/direct_confirmation_measure.py:18-25, gensim/topic_coherence/direct_confirmation_measure.py:108-125

Implementation Details

The direct_confirmation_measure.py module handles these calculations using an InvertedIndexAccumulator to retrieve document frequencies.

def log_conditional_probability(segmented_topics, accumulator, with_std=False, with_support=False):
    # ...
    w_star_count = accumulator[w_star]
    co_occur_count = accumulator[w_prime, w_star]
    m_lc_i = np.log(((co_occur_count / num_docs) + EPSILON) / (w_star_count / num_docs))
    # ...

Source: gensim/topic_coherence/direct_confirmation_measure.py:58-63

Advanced Similarity Classes

Gensim provides specialized classes for high-level similarity retrieval that utilize term-level data.

SoftCosineSimilarity

This class computes the similarity between documents using the Soft Cosine Measure. Unlike standard cosine similarity, it uses a term similarity matrix to account for related features.

Configuration Parameters:

Parameter	Type	Description
similarity_matrix	SparseTermSimilarityMatrix	The matrix containing term-to-term similarity scores.
normalize_queries	bool	Whether to L2-normalize query vectors during basis change.
normalize_documents	bool	Whether to L2-normalize document vectors.

Sources: gensim/similarities/docsim.py:756-778

WmdSimilarity

The Word Mover's Distance (WMD) index calculates the distance between documents by finding the minimum "travel cost" to transform one document's word embeddings into another's.

Lexical Similarity via FastSS

For string-based similarities, Gensim uses a Cython-optimized implementation of the FastSS algorithm and edit distance. This is particularly relevant for the LevenshteinSimilarityIndex.

• editdist(s1, s2): Computes the number of edits (insertions, deletions, substitutions) required to change one string into another.
• Performance: The gensim.similarities.fastss extension is implemented in C for high performance.

Sources: setup.py:30, gensim/test/test_similarities.py:970-990

Summary

Term Similarity Measures in Gensim bridge the gap between simple bag-of-words models and semantic understanding. By integrating lexical edit distances, word embeddings, and statistical co-occurrence probabilities, the system enables complex document analysis tasks such as Soft Cosine Similarity and Topic Coherence evaluation. These components are designed for efficiency, utilizing sparse matrix representations and C-extensions to handle large-scale natural language processing tasks.

Sources: README.md:16-30, gensim/similarities/docsim.py:726-745