Container Orchestration (Docker & Kubernetes)
Relevant source files
The following files were used as context for generating this wiki page:
huskyCI leverages containerization to execute security tests in isolated and ephemeral environments. This ensures that each analysis runs with a clean state and that the security tools do not interfere with the host system or each other. The system is designed to be flexible, supporting both Docker and Kubernetes as the underlying container orchestration platforms.
The choice of orchestrator is determined at startup and is consistent for all subsequent security scans. The huskyCI API component is responsible for communicating with the chosen orchestrator's API to create, monitor, and destroy containers for each security test.
Sources: api/util/api/api.go:27-41, api/securitytest/securitytest.go:102-117
Configuration
The container orchestration platform is selected via the HUSKYCI_INFRASTRUCTURE_USE environment variable. The huskyCI API checks this variable during its startup sequence to determine whether to initialize connections to the Docker API or the Kubernetes API.
Sources: api/util/api/api.go:27-41, deployments/docker-compose.yml:29
Environment Variables
The following environment variables control the container orchestration settings:
| Variable | Description | Default in docker-compose.yml |
|---|---|---|
HUSKYCI_INFRASTRUCTURE_USE | Specifies the container infrastructure to use. Can be docker or kubernetes. | docker |
HUSKYCI_DOCKERAPI_ADDR | The address of the Docker API host. Required if HUSKYCI_INFRASTRUCTURE_USE is docker. | dockerapi |
HUSKYCI_DOCKERAPI_TLS_VERIFY | Whether to verify the Docker API's TLS certificate. | 0 |
HUSKYCI_DOCKERAPI_CERT_PATH | Path to the Docker API TLS certificates. | /go/src/github.com/globocom/huskyCI/ |
Sources: api/util/api/api.go:72-108, deployments/docker-compose.yml:23-30
Orchestration Selection Flow
The diagram below illustrates how huskyCI selects and health-checks the orchestration engine at startup.
This diagram shows the conditional logic in the CheckHuskyRequirements function.
Sources: api/util/api/api.go:27-41
Docker Orchestration
When configured to use Docker, huskyCI relies on a Docker daemon to run the security test containers. In the provided docker-compose.yml setup, this is handled by a dedicated dockerapi service running a Docker-in-Docker (dind) image.
The api service communicates with the dockerapi service over TCP on port 2376 to manage the lifecycle of the containers.
Sources: deployments/docker-compose.yml:5-15, deployments/docker-compose.yml:30
Docker Execution Flow
The securitytest package orchestrates the scan. The Start() function calls dockerRun() if the infrastructure is set to docker. This function then calls huskydocker.DockerRun to interact with the Docker API, creating a container, executing the command, and retrieving the output.
This sequence shows the interaction between huskyCI's internal packages and the Docker API to run a single security test.
Sources: api/securitytest/securitytest.go:108-114, api/securitytest/securitytest.go:120-131
Docker Compose Services
The local development environment is defined in docker-compose.yml and includes the following key services for orchestration:
| Service | Image | Description |
|---|---|---|
dockerapi | docker:24-rc-dind | Provides the Docker-in-Docker environment, exposing the Docker API on port 2376. |
api | Built from deployments/dockerfiles/api.Dockerfile | The main huskyCI API. It connects to dockerapi to run security tests. |
mongodb | mongo:4.2.2 | The database used by the API to store analysis results. |
The following diagram shows the relationship between the services:
Sources: deployments/docker-compose.yml:4-63
Kubernetes Orchestration
When HUSKYCI_INFRASTRUCTURE_USE is set to kubernetes, huskyCI interacts with a Kubernetes cluster to run security tests. Each test is executed within a dedicated Kubernetes Pod.
Sources: api/util/api/api.go:35-39, api/securitytest/securitytest.go:103-107
Kubernetes Execution Flow
The securitytest.Start() function calls kubeRun(), which in turn uses the huskykube.KubeRun function. This function encapsulates the entire lifecycle of a test Pod:
- A Kubernetes client is created.
- A new Pod is created using the specified security tool's image and command.
- The system waits for the Pod to complete its execution.
- The logs (output) from the completed Pod are retrieved.
- The Pod is deleted from the cluster.
This sequence illustrates the process of running a security test as a transient Pod in Kubernetes.
Sources: api/securitytest/securitytest.go:103-107, api/securitytest/securitytest.go:133-146, api/kubernetes/huskykube.go:28-76
Security Test Containers
Each security analysis tool (e.g., Bandit, Gosec, Gitleaks) is packaged as a separate Docker image. This modular approach allows tools to be updated independently and ensures that each tool runs with its specific dependencies without conflict.
The Dockerfiles for these tools are located in the deployments/dockerfiles/ directory. A shell script, build-containers.sh, is provided to build all the security test images.
Sources: deployments/dockerfiles/ , deployments/scripts/build-containers.sh
Example Tool Images
The build-containers.sh script builds the following images, among others:
| Tool | Image Name | Dockerfile Path |
|---|---|---|
| Bandit | huskyci/bandit:latest | deployments/dockerfiles/bandit/ |
| Brakeman | huskyci/brakeman:latest | deployments/dockerfiles/brakeman/ |
| Gosec | huskyci/gosec:latest | deployments/dockerfiles/gosec/ |
| Gitleaks | huskyci/gitleaks:latest | deployments/dockerfiles/gitleaks/ |
| NpmAudit | huskyci/npmaudit:latest | deployments/dockerfiles/npmaudit/ |
| Safety | huskyci/safety:latest | deployments/dockerfiles/safety/ |
| SpotBugs | huskyci/spotbugs:latest | deployments/dockerfiles/spotbugs/ |
| TFSec | huskyci/tfsec:latest | deployments/dockerfiles/tfsec/ |
Sources: deployments/scripts/build-containers.sh:9-20
Summary
huskyCI's architecture is built upon containerization, providing flexibility through its support for both Docker and Kubernetes. This design ensures that security scans are executed in consistent, isolated, and disposable environments. The choice of orchestrator is a simple configuration step, allowing huskyCI to adapt to different operational environments, from a local docker-compose setup for development to a full-fledged Kubernetes cluster for production-scale deployments. This container-based approach is fundamental to the reliability and scalability of the huskyCI analysis pipeline.
Sources: api/util/api/api.go:27-41, api/securitytest/securitytest.go:102-117