Knowledge Distillation Pipeline (hx-note)
Relevant source files
The following files were used as context for generating this wiki page:
The Knowledge Distillation Pipeline, primarily implemented through the hx-note and hx-docs-sediment skills, is a system designed to transform ephemeral agent-user dialogue into structured, persistent knowledge. It operates on the principle that "Dialogue is water, knowledge is sediment," ensuring that valuable technical insights, decisions, and error corrections are incrementally refined and stored within the ai-docs knowledge base.
This pipeline mitigates the limitations of standard RAG systems by replacing vague vector retrieval with a multi-layered memory architecture that prioritizes "AI-First" documentation and single sources of truth.
Sources: ai-docs/003-AI/004-记忆/001-对话记忆与知识库增量沉淀/index.md:14-22, ai-docs/003-AI/005-Skill/002-Agent-Skill编写最佳实践/index.md:1-5
0x01 Core Architecture: The Three-Component System
The pipeline architecture consists of a "Minimum Viable Trio" that manages how information flows from raw logs to actionable documentation.
1.1 Components of the Trio
| Component | Function | Data Characteristic |
|---|---|---|
| Ledger | Raw "Account Book" of events. | Append-only; never modified; used for audit/traceability. |
| Views | Derived projections for the LLM. | Overwrite-enabled; structured JSON or Markdown summaries. |
| Policy | The Control Logic. | Decides when to read, write, or prune (active management). |
Sources: ai-docs/003-AI/004-记忆/001-对话记忆与知识库增量沉淀/index.md:39-50
1.2 Data Flow Logic
The diagram below illustrates the flow from raw agent interaction to structured knowledge views.
The diagram shows how raw logs are committed to an immutable ledger before being filtered by a policy to update specific derived views.
Sources: ai-docs/003-AI/004-记忆/001-对话记忆与知识库增量沉淀/index.md:24-38
0x02 The Sedimentation Pipeline
The pipeline follows a "Two Artifacts + Nine Stages" process to ensure single-responsibility and reliability in knowledge creation.
2.1 Pipeline Stages
The process is divided into nine specific stages to handle the transformation of raw dialogue into AI-first documentation:
- Ingestion: Capture URL, text, or dialogue logs.
- Validation: Check against AI-First rules (frontmatter, context).
- Deduplication: Search existing vault nodes to prevent "vault rot."
- Propagation: Distribute changes to related notes (Daily notes, Indices).
- Reconciliation: Automatically resolve contradictions between old and new assertions.
- Synthesis: Discover patterns across years of data.
- Sentinel Injection: Separate machine-generated areas from human-edited areas using markers.
- Bitemporal Tracking: Record when a fact was true (Event Time) vs. when the system learned it (Transaction Time).
- Pruning: Archive stale documents to the
archive/directory.
Sources: scripts/generateAiDocsSidebar.js:107-109, ai-docs/003-AI/004-记忆/003-obsidian-second-brain/index.md:65-75, ai-docs/003-AI/004-记忆/001-对话记忆与知识库增量沉淀/index.md:95-102
0x03 AI-First Documentation Standards
Notes generated by the hx-note pipeline must adhere to "AI-First" standards to optimize for future LLM retrieval rather than immediate human readability.
3.1 Metadata Requirements
Every note processed by the pipeline must include a specific frontmatter structure and a For future Claude preamble.
---
hxid: "unique-uuid"
title: "Technical Topic"
created_at: "YYYY-MM-DD"
model: "Processing-Model-Name"
skill: ["relevant-skill"]
tags: ["tag1", "tag2"]
---
## For future Claude
This note explains [topic] recorded on [date]. It defines the core logic for [subsystem].
Sources: ai-docs/003-AI/004-记忆/003-obsidian-second-brain/index.md:42-55
3.2 Anti-Hallucination Guardrails
The pipeline enforces three strict rules to prevent knowledge base degradation:
- No "False Non-existence": The agent must exhaustively search all aliases and indices before claiming a note does not exist.
- Search Completeness: Scans must be comprehensive; partial scans reported as complete are treated as failures.
- Truthful Ignorance: Unknown values must be marked as
TBDor left as empty## Decisionssections rather than fabricated.
Sources: ai-docs/003-AI/004-记忆/003-obsidian-second-brain/index.md:60-70
0x04 Procedural Memory Distillation
One of the most critical aspects of the hx-note pipeline is the conversion of "Successful Interaction Trajectories" into skills.
4.1 Trajectory Solidification
When an agent encounters a bug, attempts multiple fixes, and eventually succeeds, the pipeline extracts the "Successful Trace" and compresses it into a reusable macro or skill. This prevents the agent from repeating the same trial-and-error process in future sessions.
The sequence diagram illustrates how a user correction triggers the distillation of a successful trajectory into a persistent skill.
Sources: ai-docs/003-AI/004-记忆/001-对话记忆与知识库增量沉淀/index.md:118-135
0x05 Integration with HXLoLi Environment
The distilled knowledge is served through a Docusaurus-based knowledge base (ai-docs), featuring specific UI components for AI-generated content.
5.1 UI Layout and Components
- AIDocHeader: Renders metadata (model, hxid, skill) for notes in the
knowledge-basepath. - MitemitePanel: Provides a specialized panel for knowledge verification files (
.hx-mitemite.md). - Docusaurus Config: Uses a dedicated
plugin-content-docsinstance forai-docsto separate it from standard documentation.
Sources: src/theme/DocItem/Layout/index.tsx:32-35, src/theme/DocItem/Layout/index.tsx:75-78, docusaurus.config.ts:326-340
The Knowledge Distillation Pipeline serves as the bridge between ephemeral interaction and institutional memory, ensuring that every session increases the intelligence and reliability of the software project.