Phrases and Collocations
Relevant source files
The following files were used as context for generating this wiki page:
Phrases and Collocations in Gensim refers to the automated detection of multi-word expressions (MWEs) or n-gram collocations from a stream of sentences. The primary goal is to identify groups of words that appear together more frequently than would be expected by chance and join them into single tokens (e.g., "new" and "york" becoming "new_york"). This module is often used as a preprocessing step for other models like Word2Vec to improve the semantic representation of concepts that span multiple words.
The implementation is heavily inspired by Mikolov's work on distributed representations and Bouma's research on Normalized Pointwise Mutual Information (NPMI). It supports streaming data processing, allowing it to handle corpora larger than the available RAM by utilizing iterators and generators.
Sources: gensim/models/phrases.py:10-40, README.md
Core Architecture and Components
The system is built around a transformation pipeline that identifies and joins adjacent tokens. It utilizes two main classes: Phrases, which handles the learning of statistics from a corpus, and FrozenPhrases, a read-only, memory-efficient version of the model used for inference.
Class Hierarchy
The following diagram illustrates the relationship between the primary classes and the base transformation interface.
Logic Flow for Phrase Detection
The analyze_sentence method implements a state-machine logic to process tokens. It identifies "connector words" (like "of" or "the") which can exist within a phrase but do not contribute to its scoring.
Scoring Mechanisms
Gensim provides two built-in scoring functions to determine if a bigram should be joined. These functions use unigram and bigram frequency counts collected during the training phase.
| Scorer | Formula | Characteristics |
|---|---|---|
| Original (Mikolov) | (bigram_count - min_count) * len_vocab / (worda_count * wordb_count) | Default; fast to calculate; PMI-like scoring. |
| NPMI (Bouma) | ln(pab / (pa * pb)) / -ln(pab) | More robust for common words; ranges from -1 to 1; slower to calculate. |
Sources: gensim/models/phrases.py:70-98, gensim/models/phrases.py:101-140
Parameters for Custom Scorers
Users can define custom scoring functions, which must accept the following arguments:
worda_count: Occurrences of the first token.wordb_count: Occurrences of the second token.bigram_count: Occurrences of the combined phrase.len_vocab: Total unique tokens in vocabulary.min_count: The configured minimum count threshold.corpus_word_count: Total tokens (non-unique) in the corpus.
Sources: gensim/models/phrases.py:325-333, gensim/test/test_phrases.py:165-170
Vocabulary Management and Pruning
To maintain memory efficiency, the Phrases model employs a pruning strategy during vocabulary collection.
- Collection:
_learn_vocabiterates through sentences, counting unigrams and bigrams. - Constraint: If the vocabulary size exceeds
max_vocab_size(default 40M), pruning is triggered. - Pruning: The
utils.prune_vocabfunction removes tokens with counts less than the currentmin_reducethreshold. - Scaling: The
min_reducevalue is incremented after each pruning event to ensure continued memory control.
Sources: gensim/models/phrases.py:392-423, gensim/models/phrases.py:302-310
Integration with WikiCorpus
The WikiCorpus class and associated scripts use tokenization and filtering that can be fed into phrase models. The segment_wiki.py script specifically handles extracting sections and titles, which can be used to generate the sentence streams required for phrase training.
Configuration and Defaults
The behavior of phrase detection is controlled via several parameters in the Phrases constructor.
| Parameter | Type | Default | Description |
|---|---|---|---|
min_count | int | 5 | Minimum occurrences required for a bigram to be considered. |
threshold | float | 10.0 | Score threshold; higher values lead to fewer phrases. |
max_vocab_size | int | 40,000,000 | Max tokens in memory; controls pruning. |
delimiter | str | '_' | Character used to join tokens (e.g., 'new_york'). |
connector_words | set | frozenset() | Words that can appear inside a phrase without scoring. |
scoring | str/func | 'default' | Scoring algorithm used ('default', 'npmi', or custom). |
Sources: gensim/models/phrases.py:284-320, gensim/test/test_phrases.py:220-230
Summary of Usage Logic
Phrases are detected in a greedy, one-pass manner within each sentence. If a sequence is identified as a phrase based on the scoring function and threshold, it is yielded as a single joined token. The system supports multi-word expressions beyond bigrams by applying the transformation multiple times (e.g., applying a bigram model to a corpus that has already been processed for bigrams creates a trigram model).
Sources: gensim/models/phrases.py:240-258, gensim/test/test_phrases.py:100-110