Quickstart & Local Setup
Relevant source files
The following files were used as context for generating this wiki page:
DataHub's local setup provides developers with a streamlined environment to run the platform using Docker, develop the React-based frontend, and contribute to the metadata ingestion framework. The goal of the quickstart is to provide the bare-minimum steps required to get DataHub running, optionally configure SSO, and ingest at least one metadata source.
The local environment is divided into three primary development tracks: the core platform (typically run via Docker), the React Web Application for UI development, and the Python-based Metadata Ingestion framework for building connectors.
Sources: docs-website/README.md:104-111, docs-website/generateDocsDir.ts:601-611
Core Platform Quickstart
The fastest way to get started with a local DataHub instance is using the DataHub CLI and Docker. This setup deploys the standard DataHub stack, including the Metadata Service (GMS), the frontend, and supporting infrastructure like Kafka, Elasticsearch, and MySQL.
Standard Quickstart Commands
python3 -m pip install --upgrade pip wheel setuptools
python3 -m pip install --upgrade acryl-datahub
datahub docker quickstart
Sources: docs-website/generateDocsDir.ts:601-611
Local Infrastructure Flow
The following diagram illustrates how a user interacts with a locally deployed DataHub instance via the Quickstart method.
React UI Local Development
The React application (located in datahub-web-react) serves as the primary user interface. For UI development, you can run a standalone development server that proxies requests to a running DataHub Backend.
UI Development Setup
- Requirements: Node.js (Active LTS 16.13.0 recommended) and Yarn.
- Execution:
This starts a forwarding server atyarn install && yarn run starthttp://localhost:3000. It requires thedatahub-frontendserver (usually athttp://localhost:9002) to service GraphQL API requests.
Sources: datahub-web-react/README.md:41-45, datahub-web-react/README.md:65-68
Proxy Configuration
You can customize where the local UI fetches data by modifying environment variables.
| Variable | Description | Default Value |
|---|---|---|
REACT_APP_PROXY_TARGET | Target DataHub GMS/Frontend server | http://localhost:9002 |
REACT_APP_LOGO_URL | Custom logo URL for homepage/header | N/A |
REACT_APP_THEME | Selection of theme defined in themes.ts | N/A |
Sources: datahub-web-react/README.md:48-51, datahub-web-react/README.md:80-82
Metadata Ingestion Environment
Developing ingestion sources requires a Python 3.9+ environment. DataHub uses a plugin-based architecture where each connector (e.g., Snowflake, BigQuery) is an optional "extra" dependency.
Ingestion Development Workflow
The development process is managed primarily through Gradle tasks that automate virtual environment creation and dependency management.
Setup Commands
| Action | Command |
|---|---|
| Install Dev Env | ../gradlew :metadata-ingestion:installDev |
| Activate Venv | source venv/bin/activate |
| Run Fast Tests | ../gradlew :metadata-ingestion:testQuick |
| Full Linting | ../gradlew :metadata-ingestion:lint |
Sources: metadata-ingestion/CLAUDE.md:6-14, metadata-ingestion/developing.md:15-18
Environment Variables for Ingestion
DATAHUB_VENV_USE_COPIES: Set totrueif symlinks are not supported (Nix, Windows, or certain container filesystems). This uses the--copiesflag when creating the Python virtual environment.
Sources: metadata-ingestion/CLAUDE.md:23-28, metadata-ingestion/developing.md:65-71
Documentation Website Setup
The DataHub documentation site is built with Docusaurus 2. It follows a complex assembly process that gathers markdown files from across the repository.
Local Docs Development
To view changes to documentation locally:
- Install Dependencies:
yarn install(insidedocs-website). - Start Server:
../gradlew yarnStart. - Real-time Updates: Use
../gradlew fastReloadwhenever a markdown file is changed.
Sources: docs-website/README.md:12-22
Documentation Assembly Process
The generateDocsDir.ts script is responsible for:
- Discovery: Using
git ls-filesto find all.mdfiles in the repo. - Transformation: Injecting frontmatter (slugs, titles) and rewriting relative links to point to the GitHub repository or internal doc paths.
- Directives: Processing
{{ inline ... }}for code snippets and{{ command-output ... }}for dynamic content.
Sources: docs-website/generateDocsDir.ts:100-110, docs-website/generateDocsDir.ts:320-330, docs-website/AGENTS.md:25-35
Troubleshooting Local Setup
Common UI Issues
- OpenSSL Error:
Error: error:0308010C:digital envelope routines::unsupported. This is a known issue with Node 17+. Revert to Node 16.13.0. - Safari Security: Safari may drop secure cookies over HTTP. Use
yarn start:httpsfor local testing on Safari.
Sources: datahub-web-react/README.md:65-68, datahub-web-react/README.md:52-54
Common Ingestion Issues
- Wheel Build Failures: Ensure
pip install --upgrade pip wheel setuptoolsis run first. - Pydantic Conflicts: DataHub core supports Pydantic 1.x and 2.x, but certain sources may require Pydantic 1.x due to transitive dependencies.
Sources: metadata-ingestion/developing.md:79-81, metadata-ingestion/developing.md:94-100
Local setup for DataHub involves coordinating between a Docker-based backend, a React frontend, and a Python ingestion environment. By utilizing the provided Gradle wrappers and CLI commands, developers can establish a functional environment for full-stack DataHub development.
Sources: metadata-ingestion/developing.md:1-10, datahub-web-react/README.md:1-10