Text Processing and Parsing
Relevant source files
The following files were used as context for generating this wiki page:
Text processing and parsing within the project provides a robust suite of tools for converting raw, unstructured data into structured formats suitable for topic modeling and document indexing. The system handles various input formats, including large-scale XML Wikipedia dumps and specialized Digital Library (DML) formats, applying rigorous cleaning and tokenization pipelines.
The core objective is to reduce noise and normalize text through a series of transformations such as markup removal, numeric filtering, and case folding. This allows downstream algorithms to operate on clean, representative tokens while maintaining memory efficiency through streaming and multiprocessing techniques. Sources: README.md, gensim/corpora/wikicorpus.py:1-25
Wikipedia Processing Pipeline
The project includes specialized logic for parsing MediaWiki database dumps. This involves streaming through massive XML files, stripping complex Wiki markup, and extracting plain text from specific namespaces and sections.
Wiki Markup Removal
The remove_markup function utilizes a series of regular expressions to strip comments, footnotes, external links, math content, and table formatting. Because Wiki markup can be recursive, the system applies these transformations in a loop until the text stabilizes or a fixed number of iterations is reached. Sources: gensim/corpora/wikicorpus.py:151-200
The diagram shows the iterative approach used to handle nested Wiki markup effectively. Sources: gensim/corpora/wikicorpus.py:171-209
Section Extraction and Segmentation
The segment_wiki.py script further structures the output by breaking articles into sections. It uses regex to identify top-level headings and yields a structured object containing article titles, section headings, and content. Sources: gensim/scripts/segment_wiki.py:160-200
| Component | Description | Logic |
|---|---|---|
extract_page_xmls | Yields raw XML strings for <page> tags. | Uses ElementTree.iterparse to avoid loading full XML into RAM. |
segment | Parses section headings and content. | Splits text by \n==[^=].*[^=]==\n. |
find_interlinks | Extracts links to other articles. | Regex matches [[Article|Text]] patterns. |
Sources: gensim/scripts/segment_wiki.py:126-220, gensim/corpora/wikicorpus.py:126-146
Tokenization and Normalization
Once text is cleaned, it is converted into a list of tokens. The standard tokenizer filters out tokens that are too short or too long (defaulting to a range of 2 to 15 characters) and removes tokens starting with an underscore. Sources: gensim/corpora/wikicorpus.py:270-292
Transformation Pipeline
- Tokenization: Splitting text into individual words using
utils.tokenize. - Case Folding: Converting all tokens to lowercase (optional).
- Length Filtering: Discarding tokens outside the defined
TOKEN_MIN_LENandTOKEN_MAX_LEN. - Digit Removal: Sources such as
DmlSourceexplicitly filter out numeric-only tokens to focus on linguistic content.
Sources: gensim/corpora/wikicorpus.py:270-292, gensim/examples/dmlcz/sources.py:128-132
Digital Library Document Sources
The system abstracts document retrieval through an ArticleSource interface, supporting diverse formats like DML-CZ and arxmliv.
Source Architecture
The ArticleSource class defines the interface for article identification (URIs), content retrieval, and metadata parsing. Specific implementations handle different filesystem structures:
- DmlSource: Articles are directories starting with
#containingfulltext.txtandmeta.xml. - ArxmlivSource: Uses SAX event parsing to extract text and math tokens (prefixed with
$) fromtex.xml.
This hierarchy illustrates how the system standardizes document access across different research data sources. Sources: gensim/examples/dmlcz/sources.py:36-60, 203-240
Performance and Parallelization
Parsing large dumps is computationally expensive. The WikiCorpus class and segment_wiki script use multiprocessing.Pool to parallelize processing.
Parallel Processing Flow
- The main process iterates through the XML file using a lazy generator.
- Data is chunked (using
utils.chunkize) to prevent filling RAM. - Worker processes execute
_process_article(cleaning and tokenizing) in parallel. - Results are yielded back to the main process for serialization or indexing.
Sources: gensim/corpora/wikicorpus.py:445-470, gensim/scripts/segment_wiki.py:236-250
Summary
The text processing and parsing module provides the foundational infrastructure required to transform massive, messy datasets into clean token streams. By leveraging iterative regex-based markup removal, flexible document source abstractions, and parallel processing, it ensures that the project can scale to datasets as large as the entire Wikipedia dump while maintaining high technical accuracy in token representation. Sources: README.md, gensim/corpora/wikicorpus.py:480-490