Data Normalization & Intent Ladder
Relevant source files
The following files were used as context for generating this wiki page:
The Data Normalization and Intent Ladder systems process raw collected content into structured, scored leads. The normalization layer strips noise and HTML from incoming documents to ensure text consistency. The Intent Ladder then evaluates these documents to determine the strength of the user's "buying signal" or intent.
Data Normalization
Normalization transforms a raw DocumentRecord into a NormalizedDocument. This process prepares text for feature extraction and lead scoring by removing artifacts from web scraping.
Normalization Logic
The normalizeDocument function executes several cleanup steps:
- Combines the document title and Markdown content into a single string.
- Removes all HTML tags using regular expressions.
- Converts HTML entities (e.g.,
,&,<) into their plain-text equivalents. - Collapses multiple whitespace characters into single spaces.
- Trims leading and trailing whitespace.
- Discards documents that result in empty text strings.
Sources: lib/pipeline/normalize.ts:16-36
Deduplication
The normalizeDocuments function processes arrays of records. It uses a Set of contentHash values to ensure only unique documents proceed through the pipeline.
Sources: lib/pipeline/normalize.ts:48-60
Normalization Flow
The following diagram illustrates the transformation from a raw database record to the normalized structure used by the analysis pipeline.
The diagram shows the sequential text processing steps and the final deduplication check. Sources: lib/pipeline/normalize.ts:16-60
Intent Ladder and Feature Scoring
The system classifies leads using an "Intent Ladder" and a multi-factor feature vector. These scores determine the relevance and urgency of an opportunity.
Intent Ladder Rungs
Leads are assigned an intentRung (stored as an integer). Higher rungs represent stronger signals, such as direct requests for tool recommendations. The intentTermsForDocuments function extracts these rungs and formats them as tokens (e.g., rung_1) for use in preference models.
Sources: lib/db/schema.ts:166, lib/learning/signals.ts:121-127
Feature Vector
The system computes a deterministic feature vector for every opportunity. This vector includes five primary scores ranging from 0 to 1:
| Feature | Description |
|---|---|
| Fit | Matches the poster's problem against the product description. |
| Intent | Determines if the user is seeking a solution or just venting. |
| Evidence | Measures the volume of posts and sources backing the lead. |
| Momentum | Tracks if the conversation is actively growing. |
| Timing | Evaluates how recently the content was posted. |
Sources: app/page.tsx:84-93, test/opportunity-queue.test.ts:91-98
Feature Scoring Relationships
The diagram below shows how features are aggregated to decide the status of an opportunity.
The diagram represents the transition from normalized documents to a final status based on weighted feature scores. Sources: test/opportunity-queue.test.ts:25-80
Data Models and Storage
The results of normalization and intent classification are stored in the Postgres database via Drizzle ORM.
Document and Lead Schema
Normalized data is stored in the document table, while the results of the Intent Ladder analysis are stored in the lead table.
| Field | Table | Type | Description |
|---|---|---|---|
content_hash | document | text | Unique hash for deduplication. |
intent_rung | lead | integer | The classified level of intent (0-n). |
confidence | lead | real | The model's confidence in the scoring. |
score | lead | real | The final calculated rank. |
factors | lead | jsonb | Explanatory metadata for the score. |
Sources: lib/db/schema.ts:143-176
Pipeline Constraints
To ensure system stability, pipeline options are clamped to safe ranges. For example, the limit for document processing is capped at 1,000, and the scoring threshold is clamped between 0 and 4.
Sources: test/pipeline-options.test.ts:8-16
Summary of Integration
Data normalization acts as the entry gate for all scraped content, ensuring that the Intent Ladder receives clean text. By stripping HTML and collapsing whitespace, the system prevents technical noise from skewing intent scores. The resulting feature vector allows the project to programmatically route leads to different status queues—opportunity, review, or ignore—based on the strength of the detected intent and product fit.
Sources: lib/pipeline/normalize.ts, test/opportunity-queue.test.ts