Troubleshooting & Support

Relevant source files

The following files were used as context for generating this wiki page:

• CONTRIBUTING.md
• DEVELOPMENT.md
• archive/server 4.1/server-admin/modules/operator/pages/troubleshooting-and-support.adoc
• archive/server 4.1/server-admin/modules/installation/pages/phase-2-core-services.adoc
• archive/server 4.1/server-admin/modules/operator/pages/circleci-server-security-features.adoc
• CONTENT_AUTHORING.md

Troubleshooting and support for CircleCI Server v4.x involves a systematic approach to identifying, diagnosing, and resolving issues within a Kubernetes-based environment. The primary focus is on cluster health, pod status, and log analysis to ensure high availability of core services and execution environments.

Support mechanisms include the generation of automated support bundles for engineer review and manual intervention strategies for pod management and environment-specific constraints. For broader context on system design, see the Architecture Overview.

Support Bundle Generation

A support bundle is the primary diagnostic tool used by CircleCI engineers to fix issues. It aggregates critical system information, configuration, and logs into a single package.

Generation Workflow

The generation process requires kubectl access to the cluster and utilizes the support-bundle plugin managed via Krew.

The diagram shows the sequence of steps required to generate a diagnostic package for support.
Sources: archive/server 4.1/server-admin/modules/operator/pages/troubleshooting-and-support.adoc:12-40

Pod Management & Diagnostics

Kubernetes pods are the fundamental units of CircleCI Server. Monitoring their state is the first step in troubleshooting any service disruption.

Verification Commands

Operators should monitor the READY and STATUS columns to ensure pods are capable of serving requests.

Task	Command	Description
List All Pods	kubectl get pods -n <namespace>	Checks status and readiness of all service pods.
Filter Non-Running	kubectl get pods --field-selector status.phase!=Running	Isolates failed or errored pods.
Detailed Status	kubectl describe pods <pod-name>	Shows events, configuration, and lifecycle details.
Access Logs	kubectl logs <pod-name>	Retrieves standard output/error logs for a specific pod.

Sources: archive/server 4.1/server-admin/modules/operator/pages/troubleshooting-and-support.adoc:43-73

Pod Recovery

To restart a specific service, the standard procedure is to delete the pod, allowing the Kubernetes Deployment controller to recreate it automatically.

kubectl delete pod <pod-name> -n <namespace> --now

Sources: archive/server 4.1/server-admin/modules/operator/pages/troubleshooting-and-support.adoc:76-80

Infrastructure & Service Troubleshooting

Issues often arise from configuration limits or environment constraints such as air-gapped networking.

Air-Gapped REPL Access

In environments without public internet, accessing a Read-Eval-Print Loop (REPL) requires manually committing dependencies to a private registry.

The sequence illustrates the workflow for establishing a REPL connection in restricted network environments.
Sources: archive/server 4.1/server-admin/modules/operator/pages/troubleshooting-and-support.adoc:82-108

Concurrency and Scaling

Performance bottlenecks are frequently traced to concurrency limits. Starting from version 4.1.5, concurrency limits are disabled by default.

Configuration Check:
Check distributor-dispatcher logs for "concurrency limits" messages. To manually disable, update values.yaml:

distributor_dispatcher:
  disable_concurrency: true

Sources: archive/server 4.1/server-admin/modules/operator/pages/troubleshooting-and-support.adoc:110-125

Security & Audit Forensics

When troubleshooting unauthorized access or configuration changes, audit logs provide a historical record of system events.

Audit Log Analysis

Audit logs are exported as CSV files and contain JSON blobs for detailed metadata. Key fields for troubleshooting include:

• id: Unique UUID to identify duplicate events.
• success: Boolean flag indicating if the action was successful.
• request: Unique ID to connect multiple events originating from the same request.

Common Logged Events

Event	Context
context.secrets.accessed	Identifies when and by whom secrets were retrieved.
workflow.job.finish	Provides data on job completion states.
project.settings.update	Tracks configuration changes that may lead to regressions.

Sources: archive/server 4.1/server-admin/modules/operator/pages/circleci-server-security-features.adoc:61-115

Documentation Support & Self-Review

Contributors to the documentation site should perform a self-review when content issues are reported.

Troubleshooting Checklist

• Cross-references: Ensure xref: links resolve correctly in the Antora structure.
• UI Bundle: If changes don't appear, clean ui/build and ui/node_modules before rebuilding.
• Navigation: Verify nav.adoc entries exist and follow correct syntax.
• Cache: Clear browser cache if content updates are not visible after a build.

Sources: DEVELOPMENT.md:175-210, CONTENT_AUTHORING.md:225-240, CONTRIBUTING.md:130-145

The "Troubleshooting & Support" framework ensures that CircleCI Server operators have the necessary tools—from automated support bundles to manual pod management—to maintain a stable and secure CI/CD environment. Proper diagnostic practices, combined with detailed audit logs and service-level configurations, facilitate rapid resolution of system failures.