Utility Scripts
Relevant source files
The following files were used as context for generating this wiki page:
Utility scripts in the Gensim project provide essential command-line interfaces for processing large-scale datasets, converting between model formats, and preparing corpora for topic modeling. These scripts are designed to handle data that exceeds available RAM by utilizing streaming techniques and multiprocessing, ensuring efficiency for massive datasets like Wikipedia dumps.
The primary scope of these utilities includes Wikipedia extraction and segmentation, vector space model creation (Bag-of-Words and TF-IDF), and interoperability with visualization tools like the TensorBoard Embedding Projector.
Wikipedia Processing Utilities
Gensim provides several scripts to convert raw MediaWiki XML dumps into structured formats suitable for machine learning. These scripts handle decompression on-the-fly and filter content based on configurable thresholds.
Corpus Creation (make_wiki.py / make_wikicorpus.py)
These scripts convert Wikipedia articles into sparse vectors. The process involves scanning for distinct word types, removing tokens appearing in more than 10% of articles, and keeping the most frequent terms (defaulting to 100,000).
The following diagram illustrates the data flow from an XML dump to finalized vector files:
Wikipedia Segmentation (segment_wiki.py)
The segment_wiki.py script extracts plain text from Wikipedia dumps, preserving article titles and section structures. It utilizes multiprocessing to stream through XML articles and outputs data in a JSON-line format.
| Parameter | Type | Default | Description |
|---|---|---|---|
--file | string | Required | Path to MediaWiki database dump (.xml.bz2). |
--output | string | None | Path to output JSON-line file. Supports .gz or .bz2. |
--workers | int | CPU_COUNT - 1 | Number of parallel worker processes. |
--min-article-character | int | 200 | Minimum characters to keep an article. |
--include-interlinks | flag | False | Includes article interlinks in output. |
Sources: gensim/scripts/segment_wiki.py:186-215
Vector and Model Converters
Converters allow Gensim models to be used in external environments or transition between internal representations.
Word2Vec to Tensor (word2vec2tensor.py)
This script converts a trained Word2Vec model into a format compatible with the TSV (Tab-Separated Values) format used by the TensorBoard Embedding Projector.
Logic Flow:
- Loads the Word2Vec model.
- Iterates through the vocabulary.
- Writes two TSV files:
- Tensors: The raw vector weights.
- Metadata: The corresponding words/labels.
Sources: gensim/scripts/word2vec2tensor.py (derived from contextual reference in gensim/test/test_scripts.py:113-117)
Release and Maintenance Scripts
The repository includes scripts specifically for maintainers to assist in the software release cycle. These are referenced in the release documentation and are typically used to automate versioning and distribution tasks.
Sources: release/README.md:1-12
Architecture of Streaming Processing
Gensim's utility scripts follow a specific architectural pattern to ensure memory independence. Instead of loading an entire corpus into memory, they use a "Stream and Process" approach.
Summary
The utility scripts in Gensim are robust command-line tools that bridge the gap between raw data (like Wikipedia dumps) and numerical models. By leveraging multiprocessing and streaming readers like WikiCorpus, they enable users to process multi-gigabyte datasets on standard hardware. Key outputs include Matrix Market (.mm) files for sparse vectors and text-based mappings for word identifiers.
Sources: gensim/scripts/make_wiki.py:59-106, gensim/scripts/segment_wiki.py:40-60