Release and CI Process
Relevant source files
The following files were used as context for generating this wiki page:
The Release and CI Process in Gensim is a structured workflow designed to manage versioning, document changes via changelogs, and ensure stability through continuous integration. The process involves transition between specialized git branches (develop, release-X.X.X, and master) and automated scripts for version bumping and artifact verification.
The high-level workflow transitions from feature development on forks and the develop branch into a dedicated release branch where versions are bumped and changelogs are finalized, before eventually merging into master for tagging and distribution. Sources: CONTRIBUTING.md:14-36, release/prepare.sh:11-20, release/merge.sh:1-25
Release Workflow Architecture
The release process is governed by a series of shell scripts and Python utilities that automate the preparation, documentation, and merging phases.
Release Preparation Phase
The process begins with the prepare.sh script, which initializes a new release by fetching the latest code from the upstream develop branch and creating a temporary local release branch. Sources: release/prepare.sh:11-30
The diagram above illustrates the sequential steps taken to prepare a local environment for a new version release. Sources: release/prepare.sh:11-47
Branch Management and Merging
Gensim utilizes three primary branch types during a release:
- develop: The primary integration branch where all PRs are merged.
- release-$RELEASE: A local-only branch for version-specific commits (version bumps, CHANGELOG updates).
- master: The stable branch containing official releases and tags.
Sources: release/merge.sh:3-10
| Branch Type | Role | Persistence |
|---|---|---|
develop | Integration of new features and PRs | Permanent |
release-X.X.X | Staging area for version-specific release metadata | Temporary (deleted after merge) |
master | Stable release branch with official tags | Permanent |
Sources: release/merge.sh:3-10, CONTRIBUTING.md:16-18
The sequence diagram shows the data flow and merge points between the three critical branches. Sources: release/merge.sh:12-25
Changelog Generation and Management
Automation is used to summarize activity since the previous release. The project utilizes the GitHub API to identify merged pull requests and closed issues.
Update Logic
- Summarization:
update_changelog.pyfetches the most recent release tag and timestamp from GitHub. Sources: release/update_changelog.py:15-22 - Filtering: It identifies all PRs merged after the previous release's publication date. Sources: release/update_changelog.py:25-28
- Formatting: Each PR is converted into a markdown list item containing the PR title, author, and a link to the PR. Sources: release/update_changelog.py:30-36
- Templating: The script prepends a standardized template to
CHANGELOG.mdwith sections for New Features, Bug Fixes, and Deprecations. Sources: release/update_changelog.py:53-73
Issue Tracking
The generate_changelog.py script further enhances this by identifying closed issues that are not linked to PRs, ensuring no significant changes are omitted from the documentation. Sources: release/generate_changelog.py:46-64
Continuous Integration and Testing
Gensim employs GitHub Actions and standard Python testing tools to maintain code quality.
CI Requirements and Environments
The setup.py file defines distinct test environments for different platforms and purposes. Sources: setup.py:228-265
- Core Test Environment: Includes
pytest,pytest-cov, andtestfixtures. - Windows Build: Skips certain problematic packages like
POTornmslibdepending on Python version. - Linux Build: Includes additional visualization requirements like
visdom. - Docs Build: Includes the full suite of dependencies plus Sphinx and its extensions for reproducible documentation.
Sources: setup.py:228-250
Extension Building
A custom build step is implemented in setup.py via the CustomBuildExt class. This handles the bootstrapping of numpy and Cython to compile C and C++ extensions (e.g., word2vec_inner.c, doc2vec_inner.cpp) on the fly during installation. Sources: setup.py:20-40, setup.py:100-130
Diagram showing the conditional compilation logic for Gensim's high-performance extensions. Sources: setup.py:55-130
Contributor CI Workflow
Before code is merged into the develop branch, contributors must follow a verification pipeline:
- Linting: Code is checked against PEP8 using
flake8with specific ignores and line lengths (120). Sources: CONTRIBUTING.md:32 - Documentation: Documentation must be buildable via
make -C docs/src html(on Linux/MacOS). Sources: CONTRIBUTING.md:33 - Unit Testing: Tests are executed via
pytest -v gensim/test. Sources: CONTRIBUTING.md:34
Summary
The Gensim Release and CI process combines automated scripts (prepare.sh, merge.sh) with manual verification and rigorous testing environments defined in setup.py. This ensures that releases are consistently versioned, changes are documented through automated GitHub API integration, and performance-critical extensions are properly compiled across platforms. Sources: release/prepare.sh, release/merge.sh, setup.py