Installation and Setup
Relevant source files
The following files were used as context for generating this wiki page:
Gensim is a Python library designed for topic modelling, document indexing, and similarity retrieval with large corpora. The installation process focuses on providing a memory-efficient environment that leverages low-level optimization libraries for high-performance scientific computing.
The scope of setup ranges from simple package installation via pip for general users to complex editable installs for contributors involving virtual environments and test dependencies. Gensim targets the Natural Language Processing (NLP) and Information Retrieval (IR) communities, ensuring compatibility across major operating systems through its reliance on NumPy and SciPy.
Sources: README.md:14-18, setup.py:125-130
Prerequisites
Before installing Gensim, the system must have specific scientific computing dependencies. Gensim relies heavily on NumPy and SciPy. It is highly recommended to link NumPy to a fast BLAS (Basic Linear Algebra Subprograms) library such as MKL, ATLAS, or OpenBLAS to improve performance by an order of magnitude.
Sources: README.md:37-43, setup.py:133-135
Core Dependencies
| Dependency | Version Requirement | Purpose |
|---|---|---|
| Python | >= 3.8 | Base runtime environment |
| NumPy | >= 1.18.5, < 2.0 | Numerical operations and matrix handling |
| SciPy | >= 1.7.0, < 1.14.0 | Scientific computing routines |
| smart_open | >= 1.8.1 | Streamed file I/O |
| Sources: setup.py:228-237, setup.py:317-318 |
Installation Methods
Standard User Installation
The most straightforward way to install the latest stable version is via the Python Package Index (PyPI).
pip install --upgrade gensim
Alternatively, if installing from a source distribution (.tar.gz), users should navigate to the unzipped directory and run:
python setup.py install
Sources: README.md:46-56, setup.py:139-145
Developer Setup
Contributors must perform a more detailed setup to ensure the environment supports testing and documentation building. This involves forking the repository and using "editable" installs.
Sources: CONTRIBUTING.md:15-30
The diagram shows the workflow for setting up a development environment, highlighting the divergence between Unix-like and Windows systems for activation and dependency selection.
Sources: CONTRIBUTING.md:16-29
Build Architecture and Extensions
Gensim utilizes C and C++ extensions to accelerate performance-critical components such as Word2Vec and FastText. The setup.py script includes a bootstrapping mechanism to handle numpy and Cython during the build process even if they are not initially present at the module level.
Sources: setup.py:23-40, setup.py:85-89
Compiled Extensions
The build process targets several core modules for compilation:
- C Extensions:
word2vec_inner,_mmreader,fasttext_inner,_matutils,nmf_pgd, andfastss. - C++ Extensions:
doc2vec_inner,word2vec_corpusfile,fasttext_corpusfile, anddoc2vec_corpusfile.
Sources: setup.py:23-40
Gensim's CustomBuildExt ensures that headers from NumPy are included during compilation via numpy.get_include().
Sources: setup.py:44-50, setup.py:108-111
Validation and Environment Info
After installation, users can verify their setup using the package_info script. This utility provides a dictionary containing versions of Gensim and its primary dependencies, along with platform details.
Sources: gensim/scripts/package_info.py:34-51
Verification Command
python -m gensim.scripts.package_info --info
This command outputs the following information:
- Platform: The operating system and version.
- Python: The specific version of the Python interpreter.
- NumPy/SciPy: Versions of the scientific stack.
- Location: The absolute path where Gensim is installed.
Sources: gensim/scripts/package_info.py:34-51
Testing the Installation
To ensure the installation is functional, especially for developers, the suite of unit tests should be executed using pytest.
Sources: CONTRIBUTING.md:31-36
| Test Command | Scope |
|---|---|
pytest -v gensim/test | Runs full suite of unit tests |
flake8 gensim | Validates PEP8 compliance |
make -C docs/src html | Builds documentation (macOS/Linux only) |
| Sources: CONTRIBUTING.md:33-35 |
Gensim is continuously tested under all supported Python versions (currently 3.8 through 3.12) to ensure stability across releases.
Sources: setup.py:292-297, README.md:58-61