Relevant source files

The following files were used as context for generating this wiki page: - [api/routes/stats.go](https://github.com/rafaveira3/huskyCI/blob/main/api/routes/stats.go) - [api/db/huskystats.go](https://github.com/rafaveira3/huskyCI/blob/main/api/db/huskystats.go) - [api/server.go](https://github.com/rafaveira3/huskyCI/blob/main/api/server.go) - [cli/types/types.go](https://github.com/rafaveira3/huskyCI/blob/main/cli/types/types.go) - [client/analysis/output.go](https://github.com/rafaveira3/huskyCI/blob/main/client/analysis/output.go)

# API Endpoints: Statistics The huskyCI statistics API provides a set of endpoints to retrieve aggregated data from security analyses. These metrics are generated by performing MongoDB aggregation queries on the `AnalysisCollection`, which stores the results of every scan. This feature allows users to gain high-level insights into security trends, tool usage, vulnerability severities, and historical data. The primary endpoint is designed to be flexible, allowing clients to request different types of metrics through a single, unified interface. It supports filtering by time ranges, enabling trend analysis and reporting over specific periods. ## Endpoint Definition The statistics functionality is exposed via a single GET endpoint that uses a path parameter to determine which metric to calculate and return. *Sources: [api/server.go:61]()* | Method | Path | Description | | :----- | :------------------------------ | :------------------------------------------------- | | `GET` | `/api/1.0/stats/:metric_type` | Retrieves aggregated statistics for a given metric type. | ### Path Parameters The `metric_type` parameter is mandatory and specifies the aggregation to be performed. | Parameter | Type | Description | | :------------ | :------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `metric_type` | `string` | The type of metric to retrieve. Valid values are:

`language`
`container`
`analysis`
`repository`
`author`
`severity`
`historyanalysis`

| *Sources: [api/db/huskystats.go:17-25](), [api/db/huskystats.go:210-217]()* ### Query Parameters The endpoint supports optional query parameters for filtering. | Parameter | Type | Description | | :----------- | :------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `time_range` | `string` | Filters the statistics to a specific time period. Valid values are:

`today`
`yesterday`
`last7days`
`last30days`

| *Sources: [api/db/huskystats.go:15](), [api/db/huskystats.go:198-206]()* ## Data Flow The following diagram illustrates the sequence of operations when a request is made to the statistics endpoint. ```mermaid sequenceDiagram participant Client participant API_Server as "API Server (api/server.go)" participant StatsHandler as "Stats Handler (api/routes/stats.go)" participant StatsDB as "Stats DB (api/db/huskystats.go)" participant MongoDB Client->>API_Server: GET /api/1.0/stats/{metric_type}?time_range=... API_Server->>StatsHandler: GetMetric(c) StatsHandler->>StatsDB: GetMetricByType(metricType, queryParams) Note right of StatsDB: 1. Validate metric type
2. Validate query params StatsDB->>MongoDB: Run Aggregation Pipeline MongoDB-->>StatsDB: Aggregation Results StatsDB-->>StatsHandler: return results, nil StatsHandler-->>API_Server: c.JSON(http.StatusOK, results) API_Server-->>Client: 200 OK with JSON data ``` This diagram shows the flow of a request for statistics from the client to the final JSON response. *Sources: [api/routes/stats.go:13-28](), [api/db/huskystats.go:183-197]()* 1. A `GET` request is sent to the `/api/1.0/stats/:metric_type` endpoint. 2. The `GetMetric` handler in `api/routes/stats.go` is invoked. It extracts the `metric_type` from the URL path and any query parameters. 3. The handler calls `db.GetMetricByType`, passing the metric type and query parameters. 4. Inside `api/db/huskystats.go`, the metric type and parameters are validated. An invalid `metric_type` or `time_range` will result in an error. 5. A base MongoDB aggregation pipeline is selected from the `statsQueryBase` map based on the `metric_type`. 6. If a valid `time_range` is provided, a `$match` stage is prepended to the pipeline to filter results by the `finishedAt` timestamp. 7. The final aggregation pipeline is executed on the `AnalysisCollection` in MongoDB. 8. The results are returned to the handler, which sends them back to the client as a JSON response. ## Metric Types and Aggregation Each `metric_type` corresponds to a predefined MongoDB aggregation pipeline designed to extract a specific statistic. These pipelines are defined in `api/db/huskystats.go`. *Sources: [api/db/huskystats.go:27-221]()* | Metric Type | Description | Aggregation Logic Summary - | | `language` | Counts the number of analyses per programming language. | `$unwind` the `codes` array and `$group` by `codes.language`. - | | `container` | Counts the usage of each security test container (e.g., `gosec`, `bandit`). | `$unwind` the `containers` array and `$group` by `containers.securityTest.name`. - | | `analysis` | Counts analyses by their final result (`passed`, `failed`, `error`). | `$group` by the top-level `result` field and `$sum` the occurrences. - | | `repository` | Counts the total number of repositories and branches analyzed. | Multi-stage `$group` first by `repositoryURL` and `repositoryBranch`, then by `repositoryURL` to sum branches, and finally a total count. - | | `author` | Counts the total number of unique commit authors. | `$unwind` the `commitAuthors` array and `$group` by author email to get a unique count. - | | `severity` | Counts vulnerabilities by severity (`low`, `medium`, `high`). | A multi-stage pipeline using `$objectToArray` and `$unwind` to traverse the nested `huskyciresults` object and count vulnerabilities within each severity array (e.g., `lowvulns`). - | | `historyanalysis` | Provides a time-series of analysis results, aggregated by the hour. | Uses date arithmetic to group analyses by the hour of their `finishedAt` timestamp. The `result` "warning" is mapped to "passed". The output is sorted by date descending. | ## Time-Based Filtering The `time_range` query parameter allows for filtering statistics over a set period. This is handled by the `getTimeFilterStage` function, which prepends a MongoDB `$match` stage to the aggregation pipeline. This diagram shows the logic for applying a time filter. ```mermaid flowchart TD A[Request with `time_range`] --> B{GetMetricByType}; B --> C{validateParams}; C -- Valid --> D{getTimeFilterStage}; D --> E[Generate `$match` stage on `finishedAt`]; E --> F[Prepend stage to pipeline]; F --> G[Execute Aggregation]; G --> H[Return Filtered Results]; C -- Invalid --> I[Return Error]; ``` The `$match` stage filters documents where the `finishedAt` field falls within the calculated date range. | `time_range` Value | Date Range Filter - | | :----------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `today` | Filters from the beginning of the current day to the end of the current day. - | | `yesterday` | Filters for the entire duration of the previous day. - | | `last7days` | Filters from the beginning of 6 days ago to the end of the current day. - | - | | `last30days` | Filters from the beginning of 29 days ago to the end of the current day. - | *Sources: [api/db/huskystats.go:246-259](), [api/db/huskystats.go:198-206]()* ## Relevant Data Structures The statistics are aggregated from `Analysis` documents stored in MongoDB. The structure of this data is critical to how the aggregation pipelines are built. Key fields from the `Analysis` and related structs are used as the source for metrics. *Sources: [cli/types/types.go]()* ```go // Source: cli/types/types.go:31-41 type Analysis struct { ID bson.ObjectId `bson:"_id,omitempty"` RID string `bson:"RID" json:"RID"` URL string `bson:"repositoryURL" json:"repositoryURL"` Branch string `bson:"repositoryBranch" json:"repositoryBranch"` Status string `bson:"status" json:"status"` Result string `bson:"result" json:"result"` Containers []Container `bson:"containers" json:"containers"` ErrorFound string `bson:"errorFound" json:"errorFound"` StartedAt time.Time `bson:"startedAt" json:"startedAt"` FinishedAt time.Time `bson:"finishedAt" json:"finishedAt"` Codes []Code `bson:"codes" json:"codes"` HuskyCIResults HuskyCIResults `bson:"huskyciresults,omitempty" json:"huskyciresults"` } // Source: cli/types/types.go:50-56 type HuskyCIResults struct { GoResults GoResults `bson:"goresults,omitempty" json:"goresults,omitempty"` PythonResults PythonResults `bson:"pythonresults,omitempty" json:"pythonresults,omitempty"` JavaScriptResults JavaScriptResults `bson:"javascriptresults,omitempty" json:"javascriptresults,omitempty"` RubyResults RubyResults `bson:"rubyresults,omitempty" json:"rubyresults,omitempty"` JavaResults JavaResults `bson:"javaresults,omitempty" json:"javaresults,omitempty"` GenericResults GenericResults `bson:"genericresults,omitempty" json:"genericresults,omitempty"` } // Source: cli/types/types.go:117-123 type HuskyCISecurityTestOutput struct { NoSecVulns []HuskyCIVulnerability `bson:"nosecvulns,omitempty" json:"nosecvulns,omitempty"` LowVulns []HuskyCIVulnerability `bson:"lowvulns,omitempty" json:"lowvulns,omitempty"` MediumVulns []HuskyCIVulnerability `bson:"mediumvulns,omitempty" json:"mediumvulns,omitempty"` HighVulns []HuskyCIVulnerability `bson:"highvulns,omitempty" json:"highvulns,omitempty"` } ``` - The `Result`, `FinishedAt`, `Codes`, `Containers`, and `CommitAuthors` fields from the `Analysis` document are the primary sources for the various metric types. - The `severity` metric, in particular, relies on deeply traversing the nested `HuskyCIResults` and `HuskyCISecurityTestOutput` structures to count vulnerabilities. ## Conclusion The huskyCI statistics API offers a robust and efficient method for querying aggregated security analysis data. By leveraging MongoDB's native aggregation framework, it can process large volumes of historical scan data to produce meaningful metrics on language distribution, tool usage, vulnerability counts, and historical trends. The flexible design of the `/stats/:metric_type` endpoint, combined with time-based filtering, makes it a powerful tool for monitoring and reporting on the overall security posture of projects integrated with huskyCI.