Package Scoring and Analytics

Relevant source files

The following files were used as context for generating this wiki page:

• packages/registry/src/scoring/quality-scorer.ts
• packages/registry/src/scoring/ai-evaluator.ts
• packages/registry/src/routes/analytics.ts
• packages/registry/migrations/002_add_quality_scoring.sql
• packages/registry/src/scoring/tests/quality-scorer.test.ts
• packages/registry/src/validation/package-validator.ts
• packages/webapp/public/blog/prpm-complete-feature-guide.md

The Package Scoring and Analytics system is a core feature of the PRPM registry, designed to quantitatively and qualitatively assess packages to help users discover high-quality, relevant content. It combines a rule-based scoring algorithm with AI-powered evaluation to generate a comprehensive quality score for each package. This score is used to rank packages in search results and provide authors with feedback.

The system is composed of several key components: a data model for storing scores, a multi-faceted scoring algorithm, an AI evaluation module, and API endpoints for exposing analytics data. The initial quality gate for packages is the PackageValidator, which ensures metadata and content meet baseline standards before scoring is even considered.

Data Model

The package scoring data is persisted in the PostgreSQL database. The 002_add_quality_scoring.sql migration introduces several fields to the packages and package_versions tables to support this feature.

Sources: packages/registry/migrations/002_add_quality_scoring.sql

Schema Additions

The following diagram illustrates the new fields added to the packages table.

New Database Fields

These fields are added to the packages table to store scoring and rating information.

Field	Type	Description
quality_score	FLOAT	The overall quality score of the package, ranging from 0 to 5. Defaults to 0.
quality_score_breakdown	JSONB	A JSON object storing the individual components of the quality score.
last_quality_scan_at	TIMESTAMPTZ	Timestamp of the last time the quality score was calculated.
rating_average	FLOAT	The average user-provided rating for the package. Defaults to 0.
rating_count	INTEGER	The total number of user ratings received. Defaults to 0.
version_count	INTEGER	The total number of published versions for the package. Defaults to 1.

Sources: packages/registry/migrations/002_add_quality_scoring.sql:7-13

Quality Scoring Algorithm

The quality score is a numeric value from 0 to 5 that represents the overall quality of a package. It is calculated by the calculateQualityScore function, which aggregates scores from four main categories: Content Quality, Author Credibility, Engagement, and Maintenance.

Sources: packages/registry/src/scoring/quality-scorer.ts, packages/webapp/public/blog/prpm-complete-feature-guide.md

Scoring Flow

The diagram below shows the high-level flow for calculating a package's quality score.

Scoring Factors

The total score is a sum of points from various factors, each with a maximum possible value.

Category	Factor	Max Points	Description
Content Quality	AI Prompt Quality	1.0	Score from AI evaluation of the prompt's effectiveness.
	Prompt Length	0.3	Longer, more substantial content scores higher.
	Has Examples	0.2	Presence of code blocks or examples.
	Has Documentation	0.2	A linked external documentation URL is present.
	Has Description	0.1	A description longer than 20 characters exists.
	Description Quality	0.1	Optimal length is between 100-300 characters.
	Has Repository	0.05	A source code repository URL is provided.
	Metadata Quality	0.05	Presence of tags, keywords, and a homepage.
Author Credibility	Verified Author	0.5	The author's identity is verified.
	Official Package	0.7	The package is published by the official framework/tool author.
	Author Package Count	0.3	Bonus for authors with 5 or more packages.
Engagement	Downloads	0.4	Score based on total downloads, scaled logarithmically.
	Stars	0.3	Based on the number of stars/favorites.
	Ratings	0.3	Based on the average user rating, requiring a minimum number of ratings.
Maintenance	Recency	0.3	Higher score for packages updated in the last 30 days.
	Version Count	0.2	Higher score for packages with more than one version, indicating active maintenance.

Sources: packages/webapp/public/blog/prpm-complete-feature-guide.md, packages/registry/src/scoring/quality-scorer.ts:104-207

AI-Enhanced Scoring

To provide a more objective measure of prompt effectiveness, the system uses an AI model to evaluate the package content. This is handled by the calculateQualityScoreWithAI function, which orchestrates the AI evaluation and integrates its output into the final score.

Sources: packages/registry/src/scoring/quality-scorer.ts, packages/registry/src/scoring/ai-evaluator.ts

AI Evaluation Process

The sequence diagram below illustrates the process of updating a package's quality score with AI evaluation.

The evaluatePromptWithAI function in ai-evaluator.ts constructs a specific prompt to send to the language model. This prompt asks the AI to act as an expert prompt engineer and rate the package's content on a scale of 0 to 1 based on clarity, specificity, and effectiveness.

Sources: packages/registry/src/scoring/ai-evaluator.ts:22-49

Package Validation

Before a package can be scored, it must pass validation. The package-validator.ts module is responsible for ensuring that package metadata and files adhere to the registry's standards. This serves as the first line of defense for quality control.

Sources: packages/registry/src/validation/package-validator.ts

Validation Checks

The validator performs numerous checks, including:

• Required Fields: Ensures name, description, version, and type are present.
• Name Format: Checks for valid characters and length constraints.
• Description Length: Warns if the description is too short and errors if it's too long.
• Version Format: Validates against semantic versioning (semver).
• Category: Checks if the category is one of the predefined valid categories.
• File Size & Extension: Enforces limits on file size and allowed extensions.
• Content Security: Scans for malicious patterns, potential secrets, and prompt injection/jailbreak attempts.

If validation fails, the package cannot be published. Warnings are also generated to suggest improvements, such as adding a license or tags to improve discoverability.

Sources: packages/registry/src/validation/package-validator.ts:49-291

Analytics API

The results of the scoring and other package metrics are exposed through a set of analytics API endpoints defined in routes/analytics.ts. These endpoints provide data for both package authors and the public PRPM web interface.

Sources: packages/registry/src/routes/analytics.ts

Endpoints

Method	Path	Description
GET	/api/v1/analytics/quality-breakdown	Retrieves the detailed quality score breakdown for a given package. Requires packageId query parameter.
GET	/api/v1/analytics/author/:authorId	Fetches aggregate analytics for a specific author, including total downloads and average quality score.
GET	/api/v1/analytics/trending	Returns a list of trending packages based on recent download velocity and score.
POST	/api/v1/analytics/events	An internal endpoint for tracking events like downloads and views.

Sources: packages/registry/src/routes/analytics.ts:22-105

Summary

The Package Scoring and Analytics system is a sophisticated feature that underpins the PRPM registry's commitment to quality. By combining automated, rule-based validation and scoring with nuanced AI-powered evaluations, it creates a robust framework for ranking packages. The resulting scores and analytics are made accessible via a dedicated API, empowering both users to find the best tools and authors to understand their impact. This system is fundamental to creating a trustworthy and effective ecosystem for AI development artifacts.