API Endpoints: Statistics

Relevant source files

The following files were used as context for generating this wiki page:

• api/routes/stats.go
• api/db/huskystats.go
• api/server.go
• cli/types/types.go
• client/analysis/output.go

The huskyCI statistics API provides a set of endpoints to retrieve aggregated data from security analyses. These metrics are generated by performing MongoDB aggregation queries on the AnalysisCollection, which stores the results of every scan. This feature allows users to gain high-level insights into security trends, tool usage, vulnerability severities, and historical data.

The primary endpoint is designed to be flexible, allowing clients to request different types of metrics through a single, unified interface. It supports filtering by time ranges, enabling trend analysis and reporting over specific periods.

Endpoint Definition

The statistics functionality is exposed via a single GET endpoint that uses a path parameter to determine which metric to calculate and return.

Sources: api/server.go:61

Method	Path	Description
GET	/api/1.0/stats/:metric_type	Retrieves aggregated statistics for a given metric type.

Path Parameters

The metric_type parameter is mandatory and specifies the aggregation to be performed.

Parameter	Type	Description
metric_type	string	The type of metric to retrieve. Valid values are: language container analysis repository author severity historyanalysis

Sources: api/db/huskystats.go:17-25, api/db/huskystats.go:210-217

Query Parameters

The endpoint supports optional query parameters for filtering.

Parameter	Type	Description
time_range	string	Filters the statistics to a specific time period. Valid values are: today yesterday last7days last30days

Sources: api/db/huskystats.go:15, api/db/huskystats.go:198-206

Data Flow

The following diagram illustrates the sequence of operations when a request is made to the statistics endpoint.

This diagram shows the flow of a request for statistics from the client to the final JSON response.

Sources: api/routes/stats.go:13-28, api/db/huskystats.go:183-197

1. A GET request is sent to the /api/1.0/stats/:metric_type endpoint.
2. The GetMetric handler in api/routes/stats.go is invoked. It extracts the metric_type from the URL path and any query parameters.
3. The handler calls db.GetMetricByType, passing the metric type and query parameters.
4. Inside api/db/huskystats.go, the metric type and parameters are validated. An invalid metric_type or time_range will result in an error.
5. A base MongoDB aggregation pipeline is selected from the statsQueryBase map based on the metric_type.
6. If a valid time_range is provided, a $match stage is prepended to the pipeline to filter results by the finishedAt timestamp.
7. The final aggregation pipeline is executed on the AnalysisCollection in MongoDB.
8. The results are returned to the handler, which sends them back to the client as a JSON response.

Metric Types and Aggregation

Each metric_type corresponds to a predefined MongoDB aggregation pipeline designed to extract a specific statistic. These pipelines are defined in api/db/huskystats.go.

Sources: api/db/huskystats.go:27-221

| Metric Type | Description | Aggregation Logic Summary - |
| language | Counts the number of analyses per programming language. | $unwind the codes array and $group by codes.language. - |
| container | Counts the usage of each security test container (e.g., gosec, bandit). | $unwind the containers array and $group by containers.securityTest.name. - |
| analysis | Counts analyses by their final result (passed, failed, error). | $group by the top-level result field and $sum the occurrences. - |
| repository | Counts the total number of repositories and branches analyzed. | Multi-stage $group first by repositoryURL and repositoryBranch, then by repositoryURL to sum branches, and finally a total count. - |
| author | Counts the total number of unique commit authors. | $unwind the commitAuthors array and $group by author email to get a unique count. - |
| severity | Counts vulnerabilities by severity (low, medium, high). | A multi-stage pipeline using $objectToArray and $unwind to traverse the nested huskyciresults object and count vulnerabilities within each severity array (e.g., lowvulns). - |
| historyanalysis | Provides a time-series of analysis results, aggregated by the hour. | Uses date arithmetic to group analyses by the hour of their finishedAt timestamp. The result "warning" is mapped to "passed". The output is sorted by date descending. |

Time-Based Filtering

The time_range query parameter allows for filtering statistics over a set period. This is handled by the getTimeFilterStage function, which prepends a MongoDB $match stage to the aggregation pipeline.

This diagram shows the logic for applying a time filter.

The $match stage filters documents where the finishedAt field falls within the calculated date range.

time_range Value	Date Range Filter -
today	Filters from the beginning of the current day to the end of the current day. -
yesterday	Filters for the entire duration of the previous day. -
last7days	Filters from the beginning of 6 days ago to the end of the current day. -

• |
  | last30days | Filters from the beginning of 29 days ago to the end of the current day. - |

Sources: api/db/huskystats.go:246-259, api/db/huskystats.go:198-206

Relevant Data Structures

The statistics are aggregated from Analysis documents stored in MongoDB. The structure of this data is critical to how the aggregation pipelines are built. Key fields from the Analysis and related structs are used as the source for metrics.

Sources: cli/types/types.go

// Source: cli/types/types.go:31-41
type Analysis struct {
	ID             bson.ObjectId  `bson:"_id,omitempty"`
	RID            string         `bson:"RID" json:"RID"`
	URL            string         `bson:"repositoryURL" json:"repositoryURL"`
	Branch         string         `bson:"repositoryBranch" json:"repositoryBranch"`
	Status         string         `bson:"status" json:"status"`
	Result         string         `bson:"result" json:"result"`
	Containers     []Container    `bson:"containers" json:"containers"`
	ErrorFound     string         `bson:"errorFound" json:"errorFound"`
	StartedAt      time.Time      `bson:"startedAt" json:"startedAt"`
	FinishedAt     time.Time      `bson:"finishedAt" json:"finishedAt"`
	Codes          []Code         `bson:"codes" json:"codes"`
	HuskyCIResults HuskyCIResults `bson:"huskyciresults,omitempty" json:"huskyciresults"`
}

// Source: cli/types/types.go:50-56
type HuskyCIResults struct {
	GoResults         GoResults         `bson:"goresults,omitempty" json:"goresults,omitempty"`
	PythonResults     PythonResults     `bson:"pythonresults,omitempty" json:"pythonresults,omitempty"`
	JavaScriptResults JavaScriptResults `bson:"javascriptresults,omitempty" json:"javascriptresults,omitempty"`
	RubyResults       RubyResults       `bson:"rubyresults,omitempty" json:"rubyresults,omitempty"`
	JavaResults       JavaResults       `bson:"javaresults,omitempty" json:"javaresults,omitempty"`
	GenericResults    GenericResults    `bson:"genericresults,omitempty" json:"genericresults,omitempty"`
}

// Source: cli/types/types.go:117-123
type HuskyCISecurityTestOutput struct {
	NoSecVulns  []HuskyCIVulnerability `bson:"nosecvulns,omitempty" json:"nosecvulns,omitempty"`
	LowVulns    []HuskyCIVulnerability `bson:"lowvulns,omitempty" json:"lowvulns,omitempty"`
	MediumVulns []HuskyCIVulnerability `bson:"mediumvulns,omitempty" json:"mediumvulns,omitempty"`
	HighVulns   []HuskyCIVulnerability `bson:"highvulns,omitempty" json:"highvulns,omitempty"`
}

• The Result, FinishedAt, Codes, Containers, and CommitAuthors fields from the Analysis document are the primary sources for the various metric types.
• The severity metric, in particular, relies on deeply traversing the nested HuskyCIResults and HuskyCISecurityTestOutput structures to count vulnerabilities.

Conclusion

The huskyCI statistics API offers a robust and efficient method for querying aggregated security analysis data. By leveraging MongoDB's native aggregation framework, it can process large volumes of historical scan data to produce meaningful metrics on language distribution, tool usage, vulnerability counts, and historical trends. The flexible design of the /stats/:metric_type endpoint, combined with time-based filtering, makes it a powerful tool for monitoring and reporting on the overall security posture of projects integrated with huskyCI.