Supported Corpus Formats
Relevant source files
The following files were used as context for generating this wiki page:
Gensim provides a robust suite of modules for handling diverse corpus formats, enabling the processing of large-scale text data for topic modeling and document similarity. These formats range from standard scientific data exchange formats like Matrix Market to specialized formats for large dumps like Wikipedia.
The architecture emphasizes memory independence, allowing users to stream data from disk without loading entire datasets into RAM. This is achieved through iterator-based interfaces and streaming processing pipelines.
Sources: README.md:38-46, gensim/corpora/wikicorpus.py:596-600
Matrix Market (MmCorpus)
The Matrix Market format is a primary interchange format in Gensim, typically used for storing sparse bag-of-words (BoW) vectors.
Implementation Details
- MmCorpus: A class for reading and writing corpora in the Matrix Market format. It supports fast random access via an index file (
.index). - Compression: While it requires random access (seek) for writing, it supports reading from compressed formats like
.gzand.bz2. - Metadata: It can store document titles or other identifiers in a
.metadata.cpicklefile.
The diagram above shows the serialization flow of a corpus into the Matrix Market format and its subsequent accessibility.
Sources: gensim/test/test_corpora.py:165-185, gensim/scripts/make_wiki.py:19-25
Wikipedia XML Dumps (WikiCorpus)
Gensim provides specialized support for MediaWiki database dumps, allowing for the extraction of plain text from massive XML files.
Key Features
- On-the-fly Extraction: Decompresses and parses XML articles using multiple cores without requiring full decompression to disk.
- Preprocessing: Includes sophisticated regex-based filtering to remove wiki markup, templates, tables, and namespaces.
- Section Segmentation: Capable of splitting articles into sections (e.g., Introduction, History) and extracting interlinks.
Data Processing Flow
The sequence diagram illustrates the parallelized processing of Wikipedia dumps into usable vectors or text segments.
Sources: gensim/corpora/wikicorpus.py:592-620, gensim/scripts/segment_wiki.py:12-25, gensim/scripts/segment_wiki.py:276-290
Mallet Format
The Mallet format is a line-based text format where each line represents a single document instance.
Format Structure
[document_id] [label] [text_content]
Implementation
- MalletCorpus: Inherits from
LowCorpus. It ignores the label field by default but can preservedocument_idandlangas metadata. - Metadata Support: When initialized with
metadata=True, iterating yields(doc_bow, (doc_id, doc_lang)).
Sources: gensim/corpora/malletcorpus.py:19-40, gensim/corpora/malletcorpus.py:143-155
Digital Library (DML-CZ) Format
Specialized corpus support is included for Digital Library documents, specifically for projects like DML-CZ and Numdam.
Components
| Component | Description |
|---|---|
| DmlConfig | Aggregates multiple article sources and filters by language. |
| DmlSource | Handles articles where directories start with #, containing fulltext.txt and meta.xml. |
| DmlCorpus | An article collection implementing CorpusABC, capable of building its own dictionary. |
Sources: gensim/examples/dmlcz/dmlcorpus.py:24-42, gensim/examples/dmlcz/sources.py:65-75
Other Supported Formats
Gensim supports several other common formats through its testing and implementation suites:
- BleiCorpus: Used for Latent Dirichlet Allocation (LDA) tools. It follows the format:
unique_word_count index1:count1 index2:count2.... - SvmLightCorpus: Compatible with SVMLight, storing labeled sparse vectors.
- LowCorpus: A "list-of-words" format where each line is a document.
- UciCorpus: Supports the University of California, Irvine (UCI) bag-of-words format.
- TextDirectoryCorpus: Recursively crawls a directory and treats each file (or line within a file) as a document.
Sources: gensim/test/test_corpora.py:315-325, gensim/test/test_corpora.py:302-310, gensim/test/test_corpora.py:343-355, gensim/test/test_corpora.py:534-550
Summary of Configuration Options
| Format Class | Key Parameter | Default | Description |
|---|---|---|---|
WikiCorpus | article_min_tokens | 50 | Ignores articles shorter than this limit. |
WikiCorpus | token_min_len | 2 | Minimum character length for tokens. |
MalletCorpus | metadata | False | Whether to return document IDs and labels. |
TextCorpus | tokenizer | None | Custom function to split raw text into tokens. |
Sources: gensim/corpora/wikicorpus.py:623-645, gensim/corpora/malletcorpus.py:65-75