CI/CD Setup
Relevant source files
The following files were used as context for generating this wiki page:This document outlines the automated CI/CD pipeline for the project, which uses GitHub Actions to build, test, scan, and push Docker images for the various microservices. The system is designed to ensure code quality, security, and consistent deployments.
The process is split into two main automated pipelines: one for continuous integration on every push and pull request, and another for building and publishing release artifacts to Docker Hub. The final deployment to production hosts is a manual step using helper scripts that pull these versioned images. For more information on the manual deployment process, see the Deployment Guide.
Sources: CI_CD_SETUP.md:3-4, deploys/README.md:3-5
Architecture Overview
The CI/CD process is orchestrated by GitHub Actions. It automates the lifecycle from code commit to a deployable Docker image stored in Docker Hub. The Continuous Deployment (CD) part involves manually running a script on target hosts to pull and run the new image version.
This diagram illustrates the end-to-end flow from a developer pushing code to the application being deployed.
Initial Setup
Before the CI/CD pipelines can run successfully, you must configure secrets in the GitHub repository and set up a Docker Hub account.
1. Docker Hub Account and Access Token
An access token is required for the pipeline to push images to Docker Hub. This is more secure than using your account password.
- Create a Docker Hub account if you don't have one.
- Navigate to Account Settings -> Security.
- Click "Generate new token".
- Give the token a name (e.g.,
deployment) and set its permissions to "Read & Write". - Copy the generated token. This will be used as the
DOCKER_PASSWORDsecret.
Sources: DOCKER_HUB_SETUP.md:27-44
2. GitHub Secrets
The pipelines require secrets to be configured in the repository settings under Settings > Secrets and Variables > Actions.
| Secret | Description | Example Value |
|---|---|---|
DOCKER_USERNAME | Your Docker Hub username. | haongo123 |
DOCKER_PASSWORD | The Docker Hub access token you generated. | dckr_pat_xxxxxxxx |
Sources: CI_CD_SETUP.md:27-31, DOCKER_HUB_SETUP.md:46-55
CI/CD Pipelines
The project utilizes two distinct GitHub Actions workflows located in .github/workflows/.
Pipeline 1: Docker Build & Test
This pipeline focuses on continuous integration, ensuring that every change is validated.
- Workflow File:
.github/workflows/docker-build-test.yml - Triggers:
- Push to
mainordevelopbranches. - Pull Request targeting
mainordevelop. - Manual trigger (
workflow_dispatch).
- Push to
- Purpose: To validate code and Docker configurations without publishing any artifacts.
The following diagram shows the parallel jobs executed in this pipeline.
Sources: CI_CD_SETUP.md:7-15, 51-89
Jobs:
| Job | Tool | Description |
|---|---|---|
lint-dockerfiles | Hadolint | Validates Dockerfile syntax and checks for best practices. |
build-images | Docker Buildx | Builds all 8 microservice images in parallel. |
scan-images | Trivy | Scans the built images for security vulnerabilities. Results appear in the GitHub Security tab. |
test-containers | Docker Run | Runs tests for the gateway and user-app to validate container startup. |
report | GitHub Actions | Provides a summary of the workflow run in the GitHub Actions UI. |
Sources: CI_CD_SETUP.md:65-89
Pipeline 2: Build & Push to Docker Hub
This pipeline is responsible for creating and publishing official release images.
- Workflow File:
.github/workflows/docker-push.yml - Triggers:
- Push to the
mainbranch. - Pushing a Git tag (e.g.,
v1.0.0). - Manual trigger with a version input.
- Push to the
- Purpose: To build and push tagged Docker images to Docker Hub and create a GitHub Release for Git tags.
Versioning Strategy:
- A push to
maintags the images withlatest. - A push of a Git tag (e.g.,
v1.1.0) tags the images with that version number. - A manual run allows specifying a custom version tag.
This pipeline uses Docker's build cache to speed up subsequent builds.
cache-from: type=registry,ref=...
cache-to: type=registry,ref=...
Sources: CI_CD_SETUP.md:17-22, 91-113, 149-153
Deployment Process
Once the CI pipeline successfully builds and pushes a versioned image to Docker Hub, the final step is to deploy it to the target hosts. This is a manual process facilitated by shell scripts.
Deployment Scripts
The deployment is managed by scripts located in the deploys/ directory.
| Script | Location | Purpose |
|---|---|---|
build-all-images.sh | scripts/ | Builds all microservice Docker images locally. Used for manual builds or as part of the CI process. |
push-all-images.sh | scripts/ | Pushes all locally built and tagged images to Docker Hub. |
deploy.sh | deploys/ | The main deployment script run on a host. It pulls a specific service's image and restarts its container. |
Sources: docker/README.md:21-38, deploys/README.md:108-120, deploys/deploy.sh
Deployment Sequence
The following diagram shows the sequence of actions a developer or operator takes on a production host to deploy a new version of a service.
This process is repeated for each microservice that needs to be updated.
Sources: deploys/deploy.sh, deploys/README.md:108-120, QUICK_START.md:27-38
The deploy.sh script performs the following actions:
- Validates that an application name is provided.
- Checks for the existence of the corresponding
<app-name>.ymland.envfiles. - Loads environment variables from
.env, includingDOCKER_USERNAMEandVERSION. - Pulls the specified image version from Docker Hub.
- Stops any running container for the service.
- Starts the new container in detached mode.
- Tails the logs for immediate feedback.
Sources: deploys/deploy.sh:4-70
Monitoring and Verification
After a pipeline run or deployment, you can verify the results in several places:
- GitHub Actions UI: Check the status (pass/fail), duration, and logs for each job in the workflow run.
- GitHub Security Tab: View detailed vulnerability reports from the Trivy scan under
Security > Code scanning. - Docker Hub: Confirm that the new images have been pushed with the correct tags and push dates.
Sources: CI_CD_SETUP.md:115-135
Security Considerations
- Secrets Management: Always use GitHub-encrypted secrets for credentials. The
DOCKER_PASSWORDshould be a short-lived access token, not your account password. - Vulnerability Scanning: All images are scanned for vulnerabilities on every build. It is recommended to regularly review the findings in the GitHub Security tab.
- Image Signing: For enhanced security, consider implementing Docker Content Trust (DCT) to sign images before pushing and verify signatures before deployment.
Sources: CI_CD_SETUP.md:183-195, DOCKER_HUB_SETUP.md:220-231
Summary
The CI/CD setup automates the build and test lifecycle, providing a consistent and secure way to produce Docker images. It enforces quality through linting, testing, and security scanning on every code change. While the CI part is fully automated, the deployment (CD) is a deliberate manual step, giving operators control over when new versions are rolled out to production environments using standardized scripts. This combination improves code quality, enhances security posture, and saves developer time.
Sources: CI_CD_SETUP.md:201-216