Prompt Engineering & Templates
Relevant source files
The following files were used as context for generating this wiki page:
The SecondLayer project utilizes a sophisticated, self-learning Dynamic Template System designed to manage legal knowledge through question classification, template matching, and automated prompt generation. This system replaces distributed, hardcoded prompt patterns with centralized, versioned architectures that allow for continuous improvement through feedback loops and automated validation.
The primary scope of this system is to optimize LLM orchestration by mapping user intents to specific legal categories and generating structured Mustache-based templates when existing matches are insufficient. This ensures high-quality legal document analysis and response generation while maintaining cost efficiency.
Dynamic Template System Architecture
The template system operates as a 5-stage pipeline, transitioning from raw user input to a verified, rolled-out legal template.
The architecture ensures that every user query is either served by a high-confidence existing template or triggers the creation of a new, validated one.
Sources: mcp_backend/src/services/template-system/README.md:9-41
Core Pipeline Stages
| Stage | Component | Purpose | Key Metrics |
|---|---|---|---|
| 1 | Classification | Determines intent/category and extracts entities (dates, names, etc.) | Confidence Score |
| 2 | Matching | Uses semantic search (Qdrant) to find top-K templates | Similarity Score (>0.65) |
| 3 | Generation | Auto-creates new templates via LLM (gpt-4o) if matching fails | Syntax/Schema Validity |
| 4 | Storage | Manages versioning (MAJOR/MINOR/PATCH) and registry | Version History |
| 5 | Recommendations | Suggests templates based on frequency, trends, or cost | ROI per template |
Sources: mcp_backend/src/services/template-system/README.md:11-39, mcp_backend/src/services/template-system/README.md:374-381
Template Generation and Validation
When a query lacks a matching template, the TemplateGenerator service initiates an automated creation process. This process involves LLM-based construction followed by a rigorous validation pipeline.
Generation Workflow
The generator uses the gpt-4o model to create a reusable template structure, including Mustache syntax placeholders and JSON schemas for inputs and outputs.
Validation Pipeline Requirements
Templates must pass four primary checks before being submitted for admin review:
- Mustache Syntax: Balanced
{{variable}}brackets must be present. - Schema Consistency: Input and output JSON schemas must be valid and of type 'object'.
- Variable Extraction: All variables used in the prompt template must be defined in the input schema properties.
- Sampling Tests: The generator runs 5 sampling tests with generated test inputs to verify rendering and output validity.
Sources: mcp_backend/src/services/template-system/TemplateGenerator.ts:145-179, mcp_backend/src/services/template-system/TemplateGenerator.ts:184-219
Prompt Life-cycle Data Structures
The system uses specific TypeScript interfaces to manage the assembly and execution of prompts. This architecture ensures that prompt context is separated from system instructions.
Core Data Models
| Interface | Description | Key Fields |
|---|---|---|
PromptIntent | Describes the goal and budget | name, reasoning_budget, output_schema |
PromptTemplate | The reusable building block | id, template, variables, version |
PromptContext | Data context for assembly | sources, user_query, metadata |
PromptPolicy | Safety and execution constraints | max_tokens, temperature, hallucination_guard_level |
SourceBlock | Typed context injection | type (raw_text/legal_article), payload, trust_level |
Sources: mcp_backend/src/types/prompt.ts:14-72
Source Injection Pattern
A critical design rule in the project is that SourceBlock data (legal articles, raw text, or citations) is never injected into system messages. Instead, it is treated as a separate context layer within the PromptInstance.
Sources: mcp_backend/src/types/prompt.ts:63-64
Template API Integration
The system exposes REST endpoints for managing the template lifecycle, requiring different levels of authorization.
Implementation Example: Generation Request
The generateTemplate method in TemplateGenerator.ts handles the creation of a generation record.
async generateTemplate(request: TemplateGenerationRequest): Promise<any> {
const generationId = this.generateGenerationId();
// 1. Generate template with LLM
const template = await this.createTemplateWithLLM(request.questionText, request.classification);
// 2. Validate syntax and schema
const validationResult = await this.validateTemplate(template);
// 3. Run sampling tests
const samplingResults = await this.runSamplingTests(template);
// 4. Store generation record
await this.storeGeneration(generationId, request, template, validationResult, samplingResults, true);
return { generationId, status: 'pending', template };
}
Sources: mcp_backend/src/services/template-system/TemplateGenerator.ts:68-109
Performance and Economic Targets
The system is designed with specific performance benchmarks and a "break-even" economic model to justify the cost of template generation.
- Classification Latency: < 50ms (cached via Redis), ~450ms (LLM call).
- Match Rate Target: > 83% within one month of operation.
- Generation Cost: ~$0.08 per template using
gpt-4o. - Economic Break-even: A template becomes cost-effective after 20 executions ($0.004 per use vs $0.08 generation cost).
- Success Rate Requirement: Templates with a success rate < 50% are flagged for auto-deprecation.
Sources: mcp_backend/src/services/template-system/README.md:65-71, mcp_backend/src/services/template-system/README.md:374-381, mcp_backend/src/services/template-system/README.md:387-393
The "Prompt Engineering & Templates" module provides a structured, automated framework for legal AI interactions. By combining semantic matching with automated LLM template generation and rigorous multi-stage validation, the system ensures that legal queries are handled with high precision while maintaining strict audit trails and cost controls.