Backend Architecture
Relevant source files
The following files were used as context for generating this wiki page:
The Backend Architecture of the Fleet project is a multi-faceted system designed to support both the core product—a device management platform—and its public-facing web infrastructure. Built primarily on the Sails.js framework, the backend handles complex data orchestration including osquery schema management, static content generation from Markdown/YAML, and integration with external cloud providers like Google Cloud Platform (GCP). It serves as the primary engine for processing endpoint telemetry, managing GitOps-driven configurations, and exposing RESTful APIs for the frontend application.
Sources: website/scripts/build-static-content.js:10-15, ee/vulnerability-dashboard/README.md:65-70, frontend/README.md:105-110, articles/deploy-fleet-on-gcp.md:1-5
Core Application Framework
The Fleet backend utilizes the Sails.js framework (v1), which follows a Model-View-Controller (MVC) pattern. The system is designed to be "source-available" and "open-core," supporting high-scale enrollment of millions of hosts. The architecture is primarily monolithic but integrates with various microservices and external tools like Redis for caching and MySQL for persistent storage.
Sources: ee/vulnerability-dashboard/README.md:65-75, handbook/company/README.md:5-15
Request Lifecycle
Requests to the backend are handled by Sails controllers, which process inputs, interact with services, and return views or JSON data. For instance, the view-articles controller filters static content built from markdown based on URL categories before passing it to the frontend.
The diagram shows the general flow of a request through the Sails-based backend.
Sources: website/api/controllers/articles/view-articles.js:25-50, website/api/controllers/download-sitemap.js:30-40
Data Orchestration and Static Content
A significant portion of the backend logic is dedicated to the build-static-content script. This script transforms YAML and Markdown files from the repository into structured JSON data and HTML partials, which are then injected into the Sails application configuration (sails.config.builtStaticContent).
Content Processing Pipeline
| Component | Source Format | Description |
|---|---|---|
| Standard Query Library | YAML | Parses docs/queries.yml to generate unique slugs and validate contributor GitHub profiles. |
| Markdown Pages | Markdown (.md) | Converts handbook, docs, and articles into HTML partials with syntax highlighting and relative link transformation. |
| Osquery Schema | JSON/YAML | Merges the osquery base schema with Fleet overrides to generate the interactive table documentation. |
| Open Positions | YAML | Generates handbook pages for hiring based on handbook/company/open-positions.yml. |
Sources: website/scripts/build-static-content.js:39-100, website/scripts/build-static-content.js:200-250, website/scripts/build-static-content.js:510-550
Schema Generation Logic
The backend dynamically extends the osquery schema by merging standard osquery tables with Fleet-specific metadata. This is used by the frontend and the query console.
// Example of how tables are processed into HTML partials for the website
let tableMdString = '\n## '+table.name;
if(table.evented){
tableMdString += ' <a purpose="evented-table-label" href="...">EVENTED TABLE</a>\n';
}
// ... Logic continues to iterate through columns and types
Sources: website/scripts/build-static-content.js:525-545
GitOps and Configuration Management
The backend is designed to support a GitOps-native workflow. Configurations for fleets, policies, and queries are stored in YAML files within an it-and-security/ directory structure. The backend (and associated tools like fleet-slackbot) reads these files to propose changes via GitHub pull requests.
Configuration Hierarchy
- Global Settings (
default.yml): Organization-wide features, integrations (Jira, Zendesk), and SSO settings. - Fleet-specific Configs (
fleets/): Configurations for specific groups like "workstations" or "servers." - Library (
lib/): Reusable snippets for scripts, policies, and software packages.
Sources: tools/fleet-slackbot/system-prompt.js:100-140, website/config/custom.js:100-120
Integration and External Services
The backend relies on several third-party services and cloud native resources, especially when deployed in environments like Google Cloud Platform (GCP).
| Service | Purpose |
|---|---|
| MySQL (Cloud SQL) | Primary database for storing host data, query results, and user information. |
| Redis (Memorystore) | Used for caching and session management to improve performance. |
| GCS / S3 | Storage for software installers and large binary assets. |
| Secret Manager | Secure storage for database passwords, API tokens, and private keys. |
| Algolia | External search engine for indexing website and documentation content. |
Sources: articles/deploy-fleet-on-gcp.md:5-15, website/config/custom.js:330-340, ee/vulnerability-dashboard/README.md:45-55
Security and Change Control
Fleet implements a "DRI" (Directly Responsible Individual) model for backend and codebase management. The website/config/custom.js file defines DRIs for specific file paths, ensuring that changes to critical backend components are reviewed by the correct stakeholder.
DRI Mapping Examples
- Database Schema: Rachael Shaw
- Website Backend/API: Eric Shaw
- REST API Docs: Rachael Shaw / Luke Heath
- Vulnerability Dashboard: Eric Shaw
Sources: website/config/custom.js:105-180
Deployment Architecture (GCP Example)
When deployed on GCP, the backend is orchestrated using Cloud Run for the main application and Cloud Run Jobs for database migrations. An External HTTP(S) Load Balancer handles ingress and SSL termination.
The diagram illustrates the infrastructure components in a typical GCP deployment.
Sources: articles/deploy-fleet-on-gcp.md:145-170
Conclusion
The Fleet backend architecture is a robust hybrid of a standard MVC web application and a sophisticated content generation engine. By leveraging Sails.js for API and request handling, and custom Node.js scripts for GitOps and documentation orchestration, it provides a flexible environment for managing enterprise-scale device fleets while maintaining a high-performance public presence.
Sources: ee/vulnerability-dashboard/README.md:65-75, handbook/company/README.md:120-130