Analysis Workflow

Relevant source files

The following files were used as context for generating this wiki page:

• api/analysis/analysis.go
• api/securitytest/securitytest.go
• client/cmd/main.go
• client/analysis/analysis.go
• client/analysis/output.go
• cli/analysis/analysis.go
• cli/types/types.go
• api/securitytest/npmaudit.go
• client/integration/sonarqube/sonarqube.go

The huskyCI analysis workflow is a comprehensive process designed to orchestrate various security tests against a code repository and centralize the results. It is initiated by a client, typically within a CI/CD pipeline, which communicates with the huskyCI API. The API then manages the execution of security tools in containerized environments (Docker or Kubernetes), gathers the outputs, and stores the results for the client to retrieve and act upon.

This workflow is divided into two main parts: the client-side process, which triggers and monitors the analysis, and the API-side process, which executes the security scans. The system is designed to be extensible, allowing for the addition of new security tools that can be run against different programming languages.

Client-Side Workflow

The client-side workflow is the entry point for developers and CI/CD systems. The huskyci-client binary orchestrates the interaction with the huskyCI API.

Sources: client/cmd/main.go, client/analysis/analysis.go

Execution Flow

The client follows a sequence of steps to perform an analysis:

1. Configuration: It first checks for necessary environment variables (HUSKYCI_API_ADDR, HUSKYCI_API_TOKEN, etc.) and sets up the configuration.
2. Start Analysis: It sends a POST request to the /analysis endpoint of the huskyCI API with the repository URL and branch. The API responds with a unique Request ID (RID).
3. Monitor Analysis: The client then enters a monitoring loop, periodically sending GET requests to the /analysis/{RID} endpoint. It waits until the analysis status field is either finished or error running. This process has a timeout of 60 minutes.
4. Process Results: Once the analysis is complete, the client retrieves the full analysis object, which contains the results from all security tests.
5. Print Output: The results are formatted and printed to standard output. The output can be in a human-readable format or JSON, depending on the command-line arguments.
6. SonarQube Integration: An optional step generates a sonarqube.json file, which aggregates all vulnerabilities into a format compatible with SonarQube's generic issue import feature.
7. Exit Status: The client exits with a specific status code. An exit code of 190 indicates that HIGH or MEDIUM severity vulnerabilities were found, which can be used to fail a CI/CD pipeline. An exit code of 0 is used if no blocking vulnerabilities are found.

Sources: client/cmd/main.go:21-127, client/analysis/analysis.go:20-136, client/integration/sonarqube/sonarqube.go:20-120

Client-API Sequence Diagram

The following diagram illustrates the interactions between the CI pipeline (running the client) and the huskyCI API.

API-Side Workflow

When the huskyCI API receives an analysis request, it orchestrates the execution of multiple security tests.

Sources: api/analysis/analysis.go, api/securitytest/securitytest.go

Execution Flow

1. Register Analysis: A new Analysis document is created in the database with a running status.
2. Language Detection: The first security test to run is always enry, which analyzes the repository to detect the programming languages present.
3. Select & Run Tests: Based on the languages detected by enry, the API selects the appropriate security tests to run. Generic tests (like gitleaks) are also included. These tests are executed in parallel.
4. Container Execution: For each security test, the API runs a container (Docker or Kubernetes) with the corresponding tool's image and command. The repository code is cloned inside the container for analysis.
5. Analyze Output: After a container finishes, its output is processed by a dedicated analysis function (e.g., analyzeNpmaudit, analyzeBrakeman). This function parses the tool's raw output and standardizes it into the HuskyCIVulnerability format.
6. Update Analysis: The results from all containers, including any vulnerabilities found, are aggregated. The main Analysis document in the database is updated with the final status (failed, passed, error), results, and container details.

Sources: api/analysis/analysis.go:22-83, api/securitytest/securitytest.go:76-117

API-Side Flowchart

This diagram shows the workflow within the huskyCI API after receiving a request.

Data Structures

The analysis workflow relies on several key data structures to manage state and results.

Sources: cli/types/types.go, api/analysis/analysis.go, api/securitytest/securitytest.go

Core Data Models

Struct	File	Description
types.Analysis	cli/types/types.go	The top-level object representing a single analysis run. It holds the status, results, and all associated containers.
types.Container	cli/types/types.go	Represents the execution of a single security test tool within a container.
types.SecurityTest	cli/types/types.go	Defines a security test, including its name, container image, and command.
securitytest.SecTestScanInfo	api/securitytest/securitytest.go	A transient struct on the API side that holds all information needed to run and analyze a single security test.
types.HuskyCIVulnerability	cli/types/types.go	A standardized structure for a single vulnerability finding, abstracting over the different formats of each security tool.

Data Model Relationship Diagram

This diagram shows how the main data structures are related.

Result Processing and Output

A critical part of the workflow is parsing the varied outputs from different security tools and presenting them in a unified format.

API-Side Parsing

On the server side, each security tool has a corresponding analyze<ToolName> function.

• Example (npmaudit): The analyzeNpmaudit function in api/securitytest/npmaudit.go is responsible for processing the JSON output from the npm audit command.
  • It unmarshals the JSON into the NpmAuditOutput struct.
  • It then iterates through the vulnerabilities, creating a standardized types.HuskyCIVulnerability for each one.
  • It maps the tool-specific severity levels (e.g., moderate, critical) to huskyCI's levels (medium, high).
  • This standardized list is then stored in the SecTestScanInfo.Vulnerabilities field.

Sources: api/securitytest/npmaudit.go:130-205

Client-Side Formatting

On the client side, the client/analysis/output.go file handles the final presentation of the results.

• The prepareAllSummary function aggregates vulnerability counts (low, medium, high) for each tool and for the total analysis.
• The printSTDOUTOutput function iterates through the results and prints a detailed, human-readable report for each vulnerability.
• The printJSONOutput function marshals the entire result object into a single JSON string.
• The printAllSummary function prints a summary table for each tool that found issues, showing the count of vulnerabilities by severity.

Sources: client/analysis/output.go:94-444

Available Security Tests

The following security tests are available, mapped by language. This logic is defined on the client-side for pre-analysis checks.

Language	Security Tools
Go	huskyci/gosec
Python	huskyci/bandit, huskyci/safety
Ruby	huskyci/brakeman
JavaScript	huskyci/npmaudit, huskyci/yarnaudit
Java	huskyci/spotbugs
HCL	huskyci/tfsec
C#	huskyci/securitycodescan
Generic	huskyci/gitleaks

Sources: cli/analysis/analysis.go:167-191

Summary

The huskyCI analysis workflow is a robust, container-based system for automating security testing. It decouples the test execution (API) from the test invocation (client), making it a flexible addition to any CI/CD pipeline. By standardizing the results from various open-source security tools, it provides a centralized and consistent view of a project's security posture, enabling developers to identify and remediate vulnerabilities early in the development lifecycle. The ability to fail a pipeline based on vulnerability severity makes it a powerful tool for enforcing security policies.