Logging & Dialogue History
Relevant source files
The following files were used as context for generating this wiki page:
Logging and Dialogue History in the DeepCode project encompass the systematic tracking of interactions between users and AI agents, specifically during the code implementation workflow. The system employs a dual-layered approach: comprehensive dialogue logging for auditing and debugging, and concise memory management to maintain LLM context efficiency. These mechanisms ensure that every step of the research-to-code transformation is recorded while preventing token overflow during long-running implementation sessions.
Sources: utils/dialogue_logger.py, workflows/agents/memory_agent_concise.py:12-25
Dialogue Logging System
The DialogueLogger class provides a structured way to capture complete conversation rounds, including system prompts, user feedback, and assistant tool calls. It organizes logs by paper_id, ensuring that all data related to a specific research paper reproduction is centralized in a dedicated directory.
Sources: utils/dialogue_logger.py:16-35
Data Structure and Log Format
Logs are generated in Markdown format with detailed headers and session overviews. Each "round" of dialogue captures a specific state in the implementation workflow.
| Field | Description |
|---|---|
| Round Number | Syncs with workflow iterations to track implementation progress. |
| Messages | Detailed capture of System, User, and Assistant roles with timestamps. |
| Tool Calls | Specific Model Context Protocol (MCP) tools invoked by the AI. |
| Tool Results | The raw and processed output from tool executions (e.g., shell output). |
| Memory Optimization | Records of when and how conversation history was compressed. |
Sources: utils/dialogue_logger.py:58-75, utils/dialogue_logger.py:202-270
Dialogue Flow Architecture
The following diagram illustrates how the DialogueLogger interacts with the workflow to record implementation steps.
The workflow engine initiates a round, logs individual message components, and ensures tool execution data is persisted to the Markdown log.
Sources: utils/dialogue_logger.py:58-150
Memory Management and History Compression
To handle the extensive context required for code implementation, the project utilizes a ConciseMemoryAgent. This agent monitors dialogue history and applies "clean slate" logic to prevent token limits from being reached while preserving critical implementation state.
Sources: workflows/agents/memory_agent_concise.py:28-50
Concise Memory Logic
The system maintains a specific hierarchy of information that is never discarded, while pruning transient conversational data.
- Permanent Context: The
system_promptand theinitial_plan(reproduction roadmap) are always preserved. - Triggered Compression: Memory optimization is triggered immediately after a
write_filetool call is detected. - Knowledge Base: Instead of full history, the agent reads from
implement_code_summary.md, which contains LLM-generated summaries of previously implemented files.
Sources: workflows/agents/memory_agent_concise.py:535-565, workflows/agents/memory_agent_concise_index.py:560-580
Memory Optimization Workflow
This sequence diagram shows the transition from a full conversation to a concise state following a file implementation.
After a file is written, the memory agent flags the session for optimization, ensuring the next LLM call uses only the essential "Knowledge Base" and the initial plan.
Sources: workflows/agents/memory_agent_concise.py:587-610, workflows/code_implementation_workflow.py:255-275
Implementation Tracking and Summarization
The implement_code_summary.md file serves as the long-term memory for the implementation state. Every time a file is created, an LLM generates a structured summary of that file's logic and dependencies.
Sources: workflows/agents/memory_agent_concise.py:214-230
Code Summary Structure
The summary generated by the AI includes specific metadata used by subsequent agents to understand the existing codebase without reading every file.
| Section | Content |
|---|---|
| Implementation Progress | The file path and core implementation ideas for the current round. |
| Dependencies | Functions likely to be called by other files, including signatures and required packages. |
| Next Steps | A single, non-implemented file suggested for the next iteration. |
Sources: workflows/agents/memory_agent_concise.py:270-300, prompts/code_prompts.py:500-530
Logic for "Knowledge Base" Retrieval
When implementing a new component, the agent uses the read_code_mem tool. This retrieves the latest entry from the summary log, providing the assistant with a focused "Knowledge Base" of the most recently implemented logic.
# workflows/agents/memory_agent_concise.py:465-475
def _read_code_knowledge_base(self) -> Optional[str]:
try:
if os.path.exists(self.code_summary_path):
with open(self.code_summary_path, "r", encoding="utf-8") as f:
content = f.read().strip()
return self._extract_latest_implementation_entry(content)
except Exception as e:
self.logger.error(f"Failed to read code knowledge base: {e}")
Sources: workflows/agents/memory_agent_concise.py:465-485
Prompt Templates for Summarization
The CONVERSATION_SUMMARY_PROMPT is specifically designed for role-aware summarization. It distinguishes between User instructions (requirements) and Assistant decisions (architectural choices).
Summary Extraction Targets
- Completed Files: Status of successfully implemented files.
- Technical Decisions: Choices made regarding system architecture.
- Key Constraints: Limitations identified during implementation.
- Role Context: Preservation of who provided specific guidance to avoid role confusion.
Sources: prompts/code_prompts.py:465-495
Summary
The Logging and Dialogue History system is a critical component of the DeepCode infrastructure, enabling reliable, long-running AI implementation cycles. By combining detailed Markdown logging via DialogueLogger with aggressive context management via ConciseMemoryAgent, the project ensures that implementation progress is both auditable and contextually efficient. The use of a persistent code summary file allows the agents to maintain a technical "Knowledge Base" that scales beyond the token limits of individual LLM sessions.