Telemetry & Cost Tracking
Relevant source files
The following files were used as context for generating this wiki page:
Telemetry and Cost Tracking within the bl1nk platform encompasses the systematic monitoring of AI skill performance, infrastructure health, and associated operational expenses. The system is designed to provide visibility into API usage, performance metrics such as response times, and the financial impact of AI model integrations, particularly through providers like AWS Bedrock and OpenRouter.
This system serves as a critical layer for maintaining project health, which is targeted for improvement from 8.5/10 to 9.5/10 through enhanced observability and automated monitoring workflows. By integrating telemetry directly into the development lifecycle—from initial skill creation to production deployment—developers can optimize both the technical performance and the cost-efficiency of their AI agents.
Sources: specs/main/plan.md, Comprehensive.md:7-10
Monitoring Architecture & Infrastructure
The monitoring infrastructure is built upon a multi-layered approach that includes continuous integration (CI) pipelines, performance monitoring workflows, and security analysis. This architecture ensures that telemetry data is captured at every stage of the software development life cycle.
Performance Monitoring Workflow
The platform utilizes a dedicated performance-test.yml GitHub Actions workflow to monitor system performance continuously. This workflow is responsible for tracking key performance indicators (KPIs) such as API response times and page load speeds.
Sources: specs/main/plan.md:77, Comprehensive.md:120
Technical Constraints and Goals
The telemetry system is bound by specific performance targets to ensure a high-quality user experience:
- API Response Time: Must remain under 200ms.
- Page Load Time: Targeted at less than 3 seconds.
- CI Pipeline Execution: Must complete in under 15 minutes.
Sources: specs/main/plan.md:21-25, specs/main/plan.md:167-170
The following diagram illustrates the flow of telemetry data from the application layers to the monitoring infrastructure:
This diagram shows the relationship between client-side interactions, server-side metric collection, and infrastructure-level monitoring.
Cost Tracking & AI Integration
Cost tracking is a specialized subset of the telemetry system focused on the financial overhead of AI model interactions. The platform integrates with high-scale AI providers and requires precise tracking of credential usage and model consumption.
AI Integration Flow
The claude-service.ts and associated tRPC routers manage the connection to AI providers like AWS Bedrock and OpenRouter. Cost tracking logic is embedded within these services to monitor token usage and associated billing metrics.
Sources: Comprehensive.md:41-43, Comprehensive.md:275-277
Credential and API Monitoring
API credentials are encrypted and stored, with their usage tracked through specific routers. This allows the system to attribute costs to specific skills or users.
| Component | Responsibility | Relevant Data |
|---|---|---|
credential.router.ts | Manages provider credentials and encryption. | Provider Type, API Keys |
skill.router.ts | Tracks CRUD operations and version restoration. | Usage Frequency, Skill Versions |
claude-service.ts | Core AI integration logic. | Token Usage, Model ID |
Sources: Comprehensive.md:55-58, Comprehensive.md:274-278
Operational Telemetry & Health Metrics
The platform tracks operational health through automated workflows that scan for vulnerabilities, validate database schemas, and ensure code quality.
Security and Observability Workflows
A suite of GitHub Actions workflows provides the backbone for operational telemetry:
- Security Scanning (
security-scan.yml): Performs weekly and on-demand analysis of dependencies, code, and secrets. - Database Checks (
database-check.yml): Validates Prisma schema integrity to prevent data-related failures. - Build Verification: Ensures multi-platform Electron releases meet quality standards.
Sources: Comprehensive.md:21-30, specs/main/plan.md:73-80
Insight Documentation
When a technical breakthrough or optimization is identified through telemetry, the insight-documenter agent is used to capture the metadata and metrics associated with the improvement.
Sequence of events when capturing a performance breakthrough derived from telemetry data.
Success Criteria for Monitoring
The implementation of Telemetry and Cost Tracking is evaluated against a set of qualitative and quantitative metrics defined in the project's comprehensive plan.
| Metric | Target State | Source of Truth |
|---|---|---|
| Project Health Score | 9.5 / 10 | Comprehensive Health Audit |
| Documentation Coverage | 50+ Files | docs/ directory |
| Workflow Functionality | 6+ Production-grade workflows | .github/workflows/ |
| Security | CodeQL + Dependabot active | GitHub Security Tab |
Sources: Comprehensive.md:323-345, specs/main/plan.md:11-15
Conclusion
The Telemetry & Cost Tracking system is integral to the bl1nk skill platform's goal of achieving enterprise-grade reliability. By combining real-time performance monitoring with granular AI cost analysis and automated infrastructure checks, the platform provides developers with the necessary visibility to build, scale, and optimize AI-driven applications efficiently.