Dictionaries and Vocabularies
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
Dictionaries and Vocabularies in Gensim serve as the fundamental mapping between raw text tokens and their unique integer identifiers. This mapping is essential for converting human-readable documents into numerical bag-of-words (BoW) vectors, which are the primary input for most topic modeling and similarity retrieval algorithms. By abstracting the vocabulary, Gensim enables memory-efficient processing of massive datasets that exceed available RAM.
The core component, Dictionary, encapsulates the word-to-ID mapping and maintains global statistics such as document frequencies. For extremely large or streaming datasets where a full vocabulary cannot be maintained in memory during the initial pass, the HashDictionary provides an alternative by using a hashing function to map tokens to a fixed range of IDs.
Sources: gensim/corpora/dictionary.py, gensim/corpora/hashdictionary.py, gensim/scripts/make_wiki_online.py:38-46
Core Dictionary Mechanisms
The Dictionary Class
The Dictionary class is the primary tool for creating a mapping between words and IDs. It processes a collection of documents (iterables of tokens) to build its internal state. It tracks how many documents each word appears in and assigns a unique integer ID to each unique token encountered.
The flow above illustrates how the doc2bow method converts a list of tokens into a sparse vector of (word_id, count) tuples.
Sources: gensim/corpora/dictionary.py, gensim/examples/dmlcz/dmlcorpus.py:90-94
Vocabulary Filtering
To improve model quality and reduce computational overhead, Dictionaries support "extreme" filtering. This process removes tokens that are either too rare or too common based on their document frequency.
| Method | Purpose | Key Parameters |
|---|---|---|
filter_extremes | Removes rare and common tokens | no_below, no_above, keep_n |
filter_tokens | Removes specific IDs from the mapping | bad_ids, good_ids |
compactify | Reassigns IDs to fill gaps after filtering | N/A |
Sources: gensim/corpora/dictionary.py, gensim/scripts/make_wiki_online.py:75-76
Large-Scale Vocabulary Management
HashDictionary for Streaming
HashDictionary is designed for "online" or "one-pass" processing. Instead of building a precise mapping from scratch, it uses a hash function to project tokens into a predefined id_range. This allows the system to begin processing bag-of-words vectors immediately without a prior scan of the entire corpus to determine the vocabulary size.
The HashDictionary is particularly useful in scenarios like processing the entire Wikipedia dump where the total number of distinct types (~7M) might be too large for efficient memory handling in early stages.
Sources: gensim/corpora/hashdictionary.py, gensim/scripts/make_wiki_online.py:46-51
Wikipedia Corpus Integration
The WikiCorpus class utilizes dictionaries to manage the massive vocabulary found in MediaWiki dumps. It can either be initialized with an existing dictionary or build one automatically by scanning the dump.
- Min Token Length: Defaults to 2 characters to filter out noise.
- Max Token Length: Defaults to 15 characters.
- Namespace Filtering: Ignores non-article namespaces like "Template" or "Category".
Sources: gensim/corpora/wikicorpus.py:41-43, gensim/corpora/wikicorpus.py:464-470
Integration with Corpora
Dictionaries are often embedded within corpus classes to provide seamless conversion. For instance, the DmlCorpus uses a dictionary to keep track of word IDs while processing articles from different sources.
Key Classes and Functions
Dictionary Class Methods
doc2bow(document, allow_update=False): Converts a list of words into a BoW format. Ifallow_updateis True, it adds new words to the dictionary.save_as_text(fname): Saves the ID-to-word mapping in a human-readable format, often used in Wikipedia processing scripts.load_from_text(fname): Reconstructs a dictionary from a text file, saving memory by not loading auxiliary statistics if they aren't needed.
Sources: gensim/corpora/dictionary.py, gensim/scripts/make_wiki_online.py:87-88
Wiki Processing Constants
Scripts like make_wiki_online.py define standard thresholds for vocabulary management in large datasets:
DEFAULT_DICT_SIZE: Set to 100,000 for Wikipedia models.no_above: Frequently set to 0.1 (filtering words appearing in >10% of documents).no_below: Frequently set to 20 (filtering words appearing in <20 documents).
Sources: gensim/scripts/make_wiki_online.py:38-42, gensim/scripts/make_wiki_online.py:75-76
Summary
Dictionaries and Vocabularies provide the essential bridge between unstructured text and structured numerical data in Gensim. Through the Dictionary class and its specialized variants like HashDictionary, the system can handle vocabularies ranging from small specialized collections to the multi-million token sets found in Wikipedia. The inclusion of robust filtering and compactification methods ensures that the resulting vector space models remain computationally efficient and statistically significant.