Database Setup and Configuration

Relevant source files

The following files were used as context for generating this wiki page:

• deployments/docker-compose.yml
• deployments/mongo-init.sh
• api/util/api/api.go
• api/db/postgres/postgres_handler.go
• api/db/postgres_test.go
• cli/types/types.go
• client/integration/sonarqube/testdata/analysis/vulnerable_go_project.json

Introduction

huskyCI orchestrates security tests and centralizes all results into a database for analysis and metrics. The primary database used is MongoDB, which stores analysis results, security test configurations, and user data. The system is designed with a database abstraction layer that also includes an incomplete implementation for PostgreSQL, suggesting potential for future support.

This document outlines the database setup, configuration, and initialization processes, primarily focusing on the Docker Compose environment, which is the recommended method for local deployment.

Supported Databases

huskyCI is configured to work with MongoDB out-of-the-box. There is also experimental code for PostgreSQL integration, but it is not fully implemented or enabled in the default deployment configuration.

• MongoDB: The primary and fully supported database.
• PostgreSQL: An experimental and incomplete integration. The API has a handler, but it is not used in the main application flow.

Sources: deployments/docker-compose.yml:81-93, api/db/postgres/postgres_handler.go

Docker Compose Setup

The docker-compose.yml file defines the services required to run huskyCI, including the database. The api service depends on the mongodb service, ensuring the database is healthy before the API starts.

This diagram shows the service dependency within the Docker Compose setup.

MongoDB Service

The mongodb service is configured using the official mongo:4.2.2 image. It exposes the standard MongoDB ports and uses a volume for data persistence.

# deployments/docker-compose.yml:53-69
    mongodb:
        container_name: huskyCI_MongoDB
        image: mongo:4.2.2
        environment:
            MONGO_INITDB_ROOT_USERNAME: huskyCIUser
            MONGO_INITDB_ROOT_PASSWORD: huskyCIPassword
            MONGO_INITDB_DATABASE: huskyCIDB
        ports:
          - "27017:27017"
          - "27018:27018"
        volumes:
            - mongo_vol:/data/db
            - ./mongo-init.sh:/docker-entrypoint-initdb.d/mongo-init.sh
        networks:
            - huskyCI_net
        healthcheck:
            test: ["CMD-SHELL", "echo 'db.stats().ok' | mongo localhost:27017/huskyDB --quiet"]
            interval: 30s
            timeout: 10s
            retries: 3

Sources: deployments/docker-compose.yml:53-69

MongoDB Environment Variables

The MongoDB container is configured via environment variables that set up the initial user and database.

Variable	Value	Description
MONGO_INITDB_ROOT_USERNAME	huskyCIUser	The root username for the database.
MONGO_INITDB_ROOT_PASSWORD	huskyCIPassword	The password for the root user.
MONGO_INITDB_DATABASE	huskyCIDB	The name of the database to be created.

Sources: deployments/docker-compose.yml:56-58

MongoDB Initialization

On its first run, the MongoDB container executes the mongo-init.sh script to perform initial setup. This script creates a dedicated user and grants it roles on the huskyCIDB database.

# deployments/mongo-init.sh
#!/bin/bash

mongo <<EOF
use admin
db.auth("huskyCIUser", "huskyCIPassword")
use huskyCIDB
db.createUser({
  user: "huskyCIUser",
  pwd: "huskyCIPassword",
  roles: [
    {
      role: "readWrite",
      db: "huskyCIDB",
    },
  ],
});
EOF

Sources: deployments/mongo-init.sh

PostgreSQL Service (Experimental)

The docker-compose.yml file contains a commented-out service definition for PostgreSQL, indicating it is not enabled by default.

# deployments/docker-compose.yml:81-93
    # TODO: Finish postgres APIs integration before ussing it. 
    # postgres:
    #     container_name: huskyCI_Postgres
    #     environment:
    #         POSTGRES_USER: huskyCIUser
    #         POSTGRES_PASSWORD: huskyCIPassword
    #         POSTGRES_DB: huskyCIDB
    #     build:
    #         context: ../
    #         dockerfile: deployments/dockerfiles/postgres.Dockerfile
    #     ports:
    #         - "5432:5432"
    #     networks:
    #         - huskyCI_net

Sources: deployments/docker-compose.yml:81-93

API Database Connection

The huskyCI API connects to the database using credentials and addresses provided by environment variables. At startup, the CheckHuskyRequirements function calls checkDB to verify the connection.

Sources: api/util/api/api.go:22-25, api/util/api/api.go:49-52

This sequence diagram illustrates the database health check process during API startup.

API Environment Variables for Database

The API service in docker-compose.yml is configured with the following environment variables to connect to the MongoDB instance.

Variable	Value in docker-compose.yml	Description
HUSKYCI_DATABASE_DB_ADDR	mongodb	The address of the database service.
HUSKYCI_DATABASE_DB_NAME	huskyCIDB	The name of the database.
HUSKYCI_DATABASE_DB_USERNAME	huskyCIUser	The username for database authentication.
HUSKYCI_DATABASE_DB_PASSWORD	huskyCIPassword	The password for database authentication.

Sources: deployments/docker-compose.yml:21-24, api/util/api/api.go:70-77

Data Models

huskyCI stores analysis data in the database. The primary data structures are defined in Go. The main entity is Analysis, which contains information about a specific scan, including the repository, status, and results.

Sources: cli/types/types.go:23-35

Entity Relationship Diagram

This diagram shows the relationships between the main data entities stored in the database, inferred from the type definitions and test data.

Analysis Structure

The Analysis struct is the core data model for storing scan results.

// cli/types/types.go:23-35
type Analysis struct {
	ID             bson.ObjectId  `bson:"_id,omitempty"`
	RID            string         `bson:"RID" json:"RID"`
	URL            string         `bson:"repositoryURL" json:"repositoryURL"`
	Branch         string         `bson:"repositoryBranch" json:"repositoryBranch"`
	Status         string         `bson:"status" json:"status"`
	Result         string         `bson:"result" json:"result"`
	Containers     []Container    `bson:"containers" json:"containers"`
	ErrorFound     string         `bson:"errorFound" json:"errorFound"`
	StartedAt      time.Time      `bson:"startedAt" json:"startedAt"`
	FinishedAt     time.Time      `bson:"finishedAt" json:"finishedAt"`
	Codes          []Code         `bson:"codes" json:"codes"`
	HuskyCIResults HuskyCIResults `bson:"huskyciresults,omitempty" json:"huskyciresults"`
}

Sources: cli/types/types.go:23-35

Database Abstraction Layer

The codebase includes a DB interface, which abstracts database operations. MongoRequests is the concrete implementation for MongoDB. A PostgresRequests struct also exists, but its methods are largely placeholders, reinforcing its experimental status.

This design allows for easier integration of other databases in the future by creating new types that satisfy the DB interface.

Sources: api/db/postgres/postgres_handler.go:25-30, api/db/postgres_test.go:17-43

DB Interface and Implementations

This class diagram shows the DB interface and its implementations.

Summary

huskyCI's database architecture is centered around MongoDB, configured and deployed via Docker Compose for ease of setup. The system uses environment variables for configuration and an initialization script to set up the necessary users and permissions. While there is an abstraction layer and some code for PostgreSQL, it is not fully implemented, and MongoDB remains the sole supported database for production and development environments. The data models are defined in Go and are structured to store detailed results from each security analysis.