API Service

Relevant source files

The following files were used as context for generating this wiki page:

• api/server.go
• api/util/api/api.go
• api/util/util.go
• api/db/huskystats.go
• deployments/docker-compose.yml
• README.md
• client/cmd/main.go

The huskyCI API is the central component of the system, acting as an orchestrator for security tests. It is responsible for receiving analysis requests from clients (like the huskyci-client), managing the lifecycle of security scans within a containerized environment (Docker or Kubernetes), and persisting the results to a database for later retrieval and metric generation. The API exposes a set of RESTful endpoints to interact with the system's core functionalities.

Sources: README.md, api/server.go

Architecture Overview

The API service is a Go application built using the Echo framework. It integrates with a MongoDB database for data persistence and an orchestration backend (Docker or Kubernetes) to run the security tests.

This diagram illustrates the high-level interaction between the API and other system components.

Startup and Initialization

On startup, the API server performs a series of prerequisite checks to ensure the environment is correctly configured. This process is handled by the CheckHuskyRequirements function.

Sources: api/server.go:47-51

The initialization sequence is as follows:

1. Load the API configuration.
2. Initialize the logging system (Graylog).
3. Execute CheckHuskyRequirements to validate the environment.
4. If all checks pass, start the Echo web server.

The following flowchart details the steps within CheckHuskyRequirements.

Requirement Checks

• Environment Variables: Verifies that all necessary environment variables are set. This includes database credentials, API user details, and infrastructure-specific settings.
  Sources: api/util/api/api.go:73-122
• Infrastructure Health: Checks the health of the selected orchestration backend.
  • For Docker, it pings the Docker API endpoint.
  • For Kubernetes, it checks the Kubernetes API server.
    Sources: api/util/api/api.go:30-45, api/util/api/api.go:124-138
• Database Connection: Attempts to connect to the MongoDB instance to ensure it's accessible.
  Sources: api/util/api/api.go:140-153
• Default Security Tests: Ensures that the default configurations for all supported security tools (e.g., gosec, bandit, gitleaks) are present in the database, inserting them if they are missing.
  Sources: api/util/api/api.go:155-166
• Default User: Checks if the default API user exists in the database and creates it if it doesn't.
  Sources: api/util/api/api.go:168-181

Configuration

The API service is configured primarily through environment variables. These are read at startup to establish connections to the database and orchestration backend.

Environment Variable	Description	Example from docker-compose.yml
HUSKYCI_DATABASE_DB_ADDR	Address of the MongoDB server.	mongodb
HUSKYCI_DATABASE_DB_NAME	The name of the database to use.	huskyCIDB
HUSKYCI_DATABASE_DB_USERNAME	Username for MongoDB authentication.	huskyCIUser
HUSKYCI_DATABASE_DB_PASSWORD	Password for MongoDB authentication.	huskyCIPassword
HUSKYCI_API_DEFAULT_USERNAME	Default username for API basic auth.	huskyCIUser
HUSKYCI_API_DEFAULT_PASSWORD	Default password for API basic auth.	huskyCIPassword
HUSKYCI_API_ALLOW_ORIGIN_CORS	CORS AllowOrigins value.	*
HUSKYCI_INFRASTRUCTURE_USE	The container orchestration backend to use.	docker
HUSKYCI_DOCKERAPI_ADDR	Address of the Docker API (if docker is used).	dockerapi
HUSKYCI_DOCKERAPI_CERT_PATH	Path to Docker API TLS certificates.	/go/src/github.com/globocom/huskyCI/

Sources: api/util/api/api.go:76-90, deployments/docker-compose.yml:13-25

API Endpoints

The API exposes several endpoints for managing and retrieving analysis data. The server uses the Echo framework to define routes.

Sources: api/server.go:66-104

Method	Path	Description	Authentication
GET	/healthcheck	Performs a health check of the API service.	None
GET	/version	Returns the current API version.	None
POST	/analysis	Submits a new repository for security analysis.	None
GET	/analysis/:id	Retrieves the status and results of a specific analysis by its RID.	None
GET	/stats/:metric_type	Retrieves aggregated statistics based on the metric type.	None
POST	/api/1.0/token	(Not fully implemented) Handles token creation.	Basic Auth
POST	/api/1.0/token/deactivate	(Not fully implemented) Handles token deactivation.	Basic Auth
PUT	/user	(Not fully implemented) Updates a user.	None

Analysis Request Flow

The primary function of the API is to handle new analysis requests sent to the /analysis endpoint. The huskyci-client initiates this process.

The following sequence diagram shows the flow for a new analysis request.

When a POST /analysis request is received:

1. The handler (ReceiveRequest) is invoked.
2. The input repository URL and branch are validated to prevent malicious inputs using regex checks in CheckMaliciousRepoURL and CheckMaliciousRepoBranch.
3. If validation is successful, a new analysis record is created in the database with a status of started and a unique Request ID (RID).
4. The RID is returned to the client, which then polls the GET /analysis/:id endpoint until the analysis status changes to finished, error, or warning.

Sources: api/util/util.go:142-195, client/cmd/main.go:49-58

Statistics Endpoint

The /stats/:metric_type endpoint provides aggregated metrics from historical analysis data stored in MongoDB. It uses MongoDB's aggregation framework to generate statistics.

Sources: api/server.go:95, api/db/huskystats.go:19-27

The following metric types are supported:

Metric Type	Description
language	Count of analyses grouped by programming language.
container	Count of analyses grouped by the security tool container used.
analysis	Count of analyses grouped by final result (passed, failed, error, warning).
repository	Total count of unique repositories and branches analyzed.
author	Total count of unique commit authors.
severity	Count of vulnerabilities grouped by severity (low, medium, high).
historyanalysis	A time-series count of analysis results per hour.

Sources: api/db/huskystats.go:29-183

The endpoint also supports a time_range query parameter to filter results. Valid values are today, yesterday, last7days, and last30days.

Sources: api/db/huskystats.go:201-213