Code Evaluation Backends
Relevant source files
The following files were used as context for generating this wiki page:
Code Evaluation Backends provide the infrastructure for executing dynamic code cells within the Oh My Pi environment. These backends are utilized by the eval tool to perform computation, data processing, and stateful script execution across different programming languages.
The system is designed to handle complex logic that exceeds the capabilities of simple shell scripts. While the Bash Tool is reserved for single binary calls or short pipelines, the evaluation backends are the primary choice for any task requiring control flow, state management, or complex shell quoting.
Sources: packages/coding-agent/src/prompts/system/system-prompt.md, packages/coding-agent/DEVELOPMENT.md
Architecture Overview
The evaluation system is structured around language-specific kernels (primarily JavaScript and Python) that interface with a central execution registry. This allows the agent to execute code in "cells," similar to a notebook environment, where state can be maintained across different execution turns.
High-Level Execution Flow
The following diagram illustrates the flow from a tool call to the language-specific backend.
The system distinguishes between "single binary calls" (handled via bash) and "programs" involving loops, conditionals, or state, which must be routed to the eval backends.
Sources: packages/coding-agent/src/prompts/system/system-prompt.md, packages/coding-agent/DEVELOPMENT.md
JavaScript Execution Backend
The JavaScript backend provides an environment for executing Node.js or Bun-compatible scripts. It is typically used for internal tool logic, workspace automation, and tasks requiring integration with the project's TypeScript/JavaScript ecosystem.
| Component | Description |
|---|---|
js/executor.ts | Orchestrates the lifecycle of the JavaScript execution context. |
| REPL Integration | Supports stateful execution where variables defined in one cell persist to the next. |
| Tool Bridge | Allows the evaluated code to interact with internal agent capabilities. |
Sources: packages/coding-agent/src/eval/js/executor.ts, packages/coding-agent/DEVELOPMENT.md
Python Execution Backend
The Python backend is optimized for data analysis, scripting, and leveraging external Python libraries. It often operates as a REPL (Read-Eval-Print Loop) kernel, enabling the agent to iteratively build and test logic.
Process Lifecycle
- Initialization: The executor starts a Python process or connects to an existing kernel.
- Input Sanitization: Code blocks are processed to handle multi-line strings and indentation.
- Execution: The code is piped to the Python interpreter.
- Result Capture: Standard output, standard error, and execution results are captured and returned to the agent.
Evaluation Constraints and Rules
The usage of evaluation backends is governed by strict engineering principles to ensure system stability and predictability.
- Default for Compute: Evaluated cells are the "default for any compute."
- Bash vs. Eval: Bash is restricted to "litmus" tests: single external CLI calls returning simple values (counts, differences, checksums). Any command with a loop, conditional, heredoc, or more than two pipe stages MUST be implemented as a program in an evaluation backend.
- No Inline Scripts: Writing multiline or complex inline-script bash is prohibited; these should be converted to JavaScript or Python evaluation cells.
- Stateful Memory: Backends are expected to handle state across the session, allowing the agent to define functions or variables that remain available for future steps.
Sources: packages/coding-agent/src/prompts/system/system-prompt.md, packages/natives/test/native.test.ts
Core Functions and Classes
The following table summarizes key technical elements identified in the source files.
| Element | Location | Purpose |
|---|---|---|
executeShell | packages/natives/native/index.js | Low-level shell execution used as a fallback or for environment setup. |
PtySession | packages/natives/native/index.js | Manages pseudo-terminal sessions for interactive or complex shell tasks. |
js/executor.ts | packages/coding-agent/src/eval/ | Main entry point for JavaScript code cell evaluation. |
py/executor.ts | packages/coding-agent/src/eval/ | Main entry point for Python code cell evaluation. |
Sources: packages/natives/test/native.test.ts, packages/coding-agent/DEVELOPMENT.md
Summary
The Code Evaluation Backends provide a robust, multi-language environment for executing complex logic within the Oh My Pi harness. By separating simple shell commands from stateful program execution, the system ensures that the agent can perform sophisticated computations in JavaScript and Python while maintaining high standards for code quality and maintainability.