Relevant source files

The following files were used as context for generating this wiki page: - [README.md](https://github.com/Willis0826/cubic-wiki/blob/main/README.md) - [package.json](https://github.com/Willis0826/cubic-wiki/blob/main/package.json) - [tsconfig.json](https://github.com/Willis0826/cubic-wiki/blob/main/tsconfig.json) - [src/app/api/generate-v2/route.ts](https://github.com/Willis0826/cubic-wiki/blob/main/src/app/api/generate-v2/route.ts) - [src/app/api/generate/route.ts](https://github.com/Willis0826/cubic-wiki/blob/main/src/app/api/generate/route.ts) - [src/lib/llm.ts](https://github.com/Willis0826/cubic-wiki/blob/main/src/lib/llm.ts) - [prisma/schema.prisma](https://github.com/Willis0826/cubic-wiki/blob/main/prisma/schema.prisma) - [src/app/api/chat/route.ts](https://github.com/Willis0826/cubic-wiki/blob/main/src/app/api/chat/route.ts) - [src/lib/github-utils.ts](https://github.com/Willis0826/cubic-wiki/blob/main/src/lib/github-utils.ts) - [src/lib/wiki-utils.ts](https://github.com/Willis0826/cubic-wiki/blob/main/src/lib/wiki-utils.ts) - [src/app/api/generate-subsystem-summary/route.ts](https://github.com/Willis0826/cubic-wiki/blob/main/src/app/api/generate-subsystem-summary/route.ts)

# System Architecture Cubic Wiki is a platform designed to analyze GitHub repositories and automatically generate documentation using Large Language Models (LLMs). The system fetches repository data, processes it through different analysis pipelines, and presents the results in a structured wiki format, complete with an AI-powered chat assistant for interactive exploration. The architecture is built on a modern web stack, leveraging serverless functions for backend logic, a relational database for persistence, and external AI services for intelligence. The core of the system lies in its ability to intelligently group files into logical "subsystems" and generate summaries. It offers two distinct analysis modes: a fast, path-based analysis (V1) and a more in-depth, content-based analysis (V2) that uses embeddings and clustering. This dual-mode approach provides flexibility, allowing users to choose between speed and accuracy depending on the repository's complexity. Sources: [README.md:3-4]() ## Core Components The system is composed of a Next.js frontend, a set of backend API routes for processing, a PostgreSQL database managed via Prisma, and the OpenAI API for all AI-related tasks. ```mermaid graph TD subgraph User Interface A[Next.js / React 19] end subgraph Backend (Vercel Serverless) B[API Routes] B -- "Interacts with" --> C{GitHub API} B -- "Interacts with" --> D{OpenAI API} B -- "Interacts with" --> E[Supabase (PostgreSQL)] end subgraph External Services C D end subgraph Database E -- "Managed by" --> F[Prisma ORM] end A -- "HTTP Requests" --> B ``` This diagram shows the high-level interaction between the frontend, backend API routes, and external services like GitHub and OpenAI. Sources: [README.md:7-13](), [package.json:11-29]() ### Technology Stack The project utilizes a modern tech stack focused on TypeScript, React, and serverless technologies. | Category | Technology | Purpose | |---------------|------------------------------------------|----------------------------------------------------------------------| | **Frontend** | Next.js 15, React 19 | Application framework and UI rendering. | | **UI** | Mantine UI, Tailwind CSS | Component library and styling. | | **Backend** | Next.js API Routes (on Vercel) | Serverless functions for handling API requests. | | **Database** | Supabase (PostgreSQL), Prisma ORM | Data storage and object-relational mapping. | | **AI** | OpenAI GPT-4o-mini, `text-embedding-3-small` | Summary generation, file clustering, and chat functionality. | | **DevOps** | Vercel, pnpm | Deployment and package management. | Sources: [README.md:7-13](), [package.json:11-29](), [src/lib/llm.ts:275-279]() ### Environment Configuration The system requires several environment variables for connecting to external services. | Variable | Description | |------------------|-------------------------------------------| | `OPENAI_API_KEY` | API key for OpenAI services. | | `OPENAI_MODEL` | Specifies the GPT model to use (e.g., `gpt-4o-mini`). | | `GITHUB_TOKEN` | Personal access token for GitHub API. | | `DATABASE_URL` | Connection string for the PostgreSQL database. | Sources: [README.md:25-32]() ## Data Model The application's data is structured around three main entities: `WikiPage`, `Subsystem`, and `File`. These are managed using Prisma and stored in a PostgreSQL database. Sources: [prisma/schema.prisma]() ```mermaid erDiagram WikiPage { Int id PK String repoUrl "unique" String branch String title String summary "Text" String shortSummary "Text, nullable" DateTime createdAt DateTime updatedAt } Subsystem { Int id PK String title String summary "Text, nullable" String shortSummary "Text" Json files Int wikiPageId FK } File { Int id PK String path String summary "Text" Float[] embedding "vector" Int wikiPageId FK } WikiPage ||--o{ Subsystem : "has" WikiPage ||--o{ File : "has" ``` This ER diagram illustrates the relationships between the `WikiPage`, `Subsystem`, and `File` tables in the database. Sources: [prisma/schema.prisma:11-57]() ### Table Definitions | Table | Field | Type | Description | |------------|----------------|------------|--------------------------------------------------------------| | **WikiPage** | `id` | `Int` | Primary key. | | | `repoUrl` | `String` | Unique URL of the analyzed GitHub repository. | | | `branch` | `String` | The repository branch that was analyzed. | | | `title` | `String` | Title of the wiki page (e.g., `owner/repo`). | | | `summary` | `String` | Detailed, Markdown-formatted summary of the repository. | | | `shortSummary` | `String?` | A plain text summary, max 100 words. | | **Subsystem**| `id` | `Int` | Primary key. | | | `title` | `String` | Name of the subsystem. | | | `summary` | `String?` | Detailed, Markdown-formatted summary of the subsystem. | | | `shortSummary` | `String` | A short, 1-2 sentence description of the subsystem. | | | `files` | `Json` | A JSON array of file paths belonging to this subsystem. | | | `wikiPageId` | `Int?` | Foreign key linking to `WikiPage`. | | **File** | `id` | `Int` | Primary key. | | | `path` | `String` | The path of the file within the repository. | | | `summary` | `String` | A short, plain text summary of the file's content. | | | `embedding` | `Float[]` | Vector embedding generated from the file summary. | | | `wikiPageId` | `Int?` | Foreign key linking to `WikiPage`. | Sources: [prisma/schema.prisma:11-57]() ## Repository Analysis Flow The platform provides two methods for analyzing a repository, each with its own API endpoint and logic. ### 🔍 Basic Analysis (V1) This mode is fast and cost-effective, relying on file paths and the README for analysis. It is handled by the `/api/generate` endpoint. ```mermaid flowchart TD A[User submits Repo URL] --> B{/api/generate}; B --> C[Parse URL & Get Metadata]; C --> D[Fetch README content]; D --> E[LLM: Generate Repo Summary]; C --> F[Fetch Repo File Tree]; F --> G[Group Files by Top-Level Dir]; G & E --> H[LLM: Group Dirs into Subsystems]; H --> I[Find or Create WikiPage in DB]; I --> J[Save Summary & Subsystems]; J --> K[Return WikiPage ID]; ``` This flowchart shows the V1 analysis process, from user submission to saving the generated wiki data. Sources: [src/app/api/generate/route.ts](), [README.md:41-47]() **Process Steps:** 1. The user submits a repository URL to the frontend, which calls the `/api/generate` endpoint. 2. The server parses the URL and fetches repository metadata, including the default branch. 3. The README content is fetched and sent to an LLM to generate a repository summary and a short summary. 4. The entire file tree of the repository is fetched. 5. Files are grouped into buckets based on their top-level directory. 6. These file buckets, along with the README summary, are sent to an LLM (`generateSubsystemsFromFilePaths`) to be organized into logical, feature-oriented subsystems. 7. The system checks if a `WikiPage` for this repository already exists. It either updates the existing page or creates a new one, saving the summary and the newly generated subsystems. Sources: [src/app/api/generate/route.ts:14-72](), [src/lib/llm.ts:68-137]() ### ✨ Advanced Analysis (V2) This mode performs a deeper analysis by examining file contents. It uses embeddings and k-means clustering to identify subsystems. This flow is handled by the `/api/generate-v2` endpoint. ```mermaid flowchart TD A[User submits Repo URL] --> B{/api/generate-v2}; B --> C[Download Repo as ZIP]; C --> D[Extract & Filter Files]; D --> E{File count > 50?}; E -- Yes --> F[LLM: Select Important Files]; E -- No --> G[Use All Filtered Files]; F --> H[Process Important Files]; G --> H; subgraph "Concurrent Processing (p-limit)" H --> I[For each file: LLM Generate Summary]; I --> J[For each summary: Generate Embedding]; end J --> K[K-Means Clustering on Embeddings]; K --> L[Group Files into Clusters]; L --> M[For each cluster: LLM Generate Subsystem Title/Summary]; M --> N[Fetch README & Generate Repo Summary]; N --> O[Find or Create WikiPage in DB]; O --> P[Save Summaries, Subsystems, & Files with Embeddings]; P --> Q[Return WikiPage ID]; ``` This flowchart details the V2 analysis process, which involves file content analysis, embedding, and clustering. Sources: [src/app/api/generate-v2/route.ts](), [README.md:50-62]() **Process Steps:** 1. The user submits a repository URL, triggering a POST request to `/api/generate-v2`. 2. The entire repository is downloaded as a ZIP archive for performance and to avoid rate limits. 3. Files are extracted. Common non-source files (images, lock files, build artifacts) are filtered out. 4. If more than 50 files remain, an LLM (`generateImportantFiles`) is used to select the most relevant ones for analysis. 5. For each selected file, the system concurrently generates a content summary (`generateFileSummary`) and a vector embedding (`generateFileEmbedding`). Concurrency is managed using `p-limit`. 6. The file embeddings are grouped using a k-means clustering algorithm. The number of clusters (K) is determined heuristically, aiming for one cluster per five files, with a minimum of 2 and a maximum of 8. 7. For each resulting cluster, an LLM (`generateSubsystemsFromFileSummary`) generates a subsystem title and short summary based on the summaries of the files within that cluster. 8. The system then saves all generated data—the `WikiPage`, `Subsystems`, and individual `File` records with their summaries and embeddings—to the database. Sources: [src/app/api/generate-v2/route.ts:22-162](), [src/lib/llm.ts:139-279](), [src/lib/github-utils.ts:80-117]() ## Deep Dive Analysis For any given subsystem, users can request a more detailed analysis via the "Get More Insights" button. This triggers a request to the `/api/generate-subsystem-summary` endpoint. ```mermaid flowchart TD A[User clicks "Get More Insights"] --> B{/api/generate-subsystem-summary}; B --> C[Fetch Subsystem & related WikiPage from DB]; C --> D[Get list of file paths for Subsystem]; D --> E[Concurrently fetch content of each file from GitHub]; E --> F[Filter out any un-fetched files]; F --> G[LLM: Generate Detailed Summary & Mermaid Diagram]; G --> H[Update Subsystem record in DB with new summary]; H --> I[Return summary to Frontend]; I --> J[Render Markdown & Mermaid diagram]; ``` This flowchart shows the process for generating a detailed subsystem summary on demand. Sources: [src/app/api/generate-subsystem-summary/route.ts](), [src/components/subsystem-card/subsystem-card.tsx:10-14](), [src/lib/llm.ts:282-323]() This feature fetches the full content of every file within a subsystem and sends it to an LLM to produce a comprehensive summary, which may include a Mermaid diagram to visualize the architecture. The generated summary is then saved back to the `Subsystem` record in the database for future viewing. Sources: [src/app/api/generate-subsystem-summary/route.ts:14-77]() ## 🤖 AI Chat Assistant Each wiki page features an AI chat assistant that provides an interactive way to explore the repository's documentation. It is powered by the `/api/chat` endpoint. ```mermaid sequenceDiagram participant User participant ChatWidget as "Chat Widget (UI)" participant API as "/api/chat" participant LLM as "OpenAI (gpt-4o-mini)" participant DB as "Database" User->>ChatWidget: Enters question ChatWidget->>API: POST { messages, wikiId } API->>DB: Fetch WikiPage, Subsystems, Files API->>LLM: streamText({ system_prompt, context, messages, tools }) Note over API,LLM: Context includes repo summary, subsystems, and file list. LLM-->>API: Tool Call: searchCodebase({query}) API->>API: Executes tool: searches context data API-->>LLM: Tool Result: "..." LLM-->>API: Streaming response text API-->>ChatWidget: Streams response ChatWidget-->>User: Displays streaming answer ``` This sequence diagram illustrates how a user's question is processed by the chat assistant, including context retrieval and tool usage. Sources: [src/app/api/chat/route.ts](), [src/components/chat/chat-widget.tsx]() When a user asks a question, the frontend sends the message history and the current `wikiId` to the `/api/chat` endpoint. The backend retrieves all associated data for that wiki (summary, subsystems, files) and constructs a detailed context. This context is passed to the OpenAI model as part of a system prompt. The model can answer questions directly or use a set of defined tools to find specific information: - `searchCodebase`: Performs a text-based search across all context data. - `getSubsystemDetails`: Retrieves detailed information for a specific subsystem. - `getFileDetails`: Retrieves the summary for a specific file. The model can perform up to 3 tool calls in a sequence to answer complex queries. Responses are streamed back to the user in real-time. Sources: [src/app/api/chat/route.ts:10-149]() ## Summary The architecture of Cubic Wiki is designed to be a scalable, AI-driven documentation platform. It effectively decouples the frontend presentation layer from the backend processing logic, which is implemented as a series of serverless API routes. The system's key strength is its flexible, dual-mode analysis engine that leverages LLMs for both high-level structural analysis (V1) and deep, content-aware clustering (V2). By storing generated data, including vector embeddings, in a structured database, the platform enables powerful features like the context-aware AI chat assistant, providing an interactive and insightful experience for developers looking to understand a new codebase. Sources: [README.md](), [src/app/api/generate-v2/route.ts](), [src/app/api/chat/route.ts]()## FULL FILE CONTEXT SUMMARY Query: You are an expert technical writer and software architect. Your task is to generate a comprehensive and accurate technical wiki page in Markdown format about a specific feature, system, or module within a given software project. You will be given: 1. The "[WIKI_PAGE_TOPIC]" for the page you need to create. 2. A list of "[RELEVANT_SOURCE_FILES]" from the project that you MUST use as the sole basis for the content. You MUST use AT LEAST 5 relevant source files for comprehensive coverage - if fewer are provided, search for additional related files in the codebase. CRITICAL STARTING INSTRUCTION: The very first thing on the page MUST be a

block listing ALL the [RELEVANT_SOURCE_FILES] you used to generate the content. There MUST be AT LEAST 5 source files listed - if fewer were provided, you MUST find additional related files to include. Format it exactly like this:

Relevant source files

Remember, do not provide any acknowledgements, disclaimers, apologies, or any other preface before the `

` block. JUST START with the `

` block. The following files were used as context for generating this wiki page: - [README.md](https://github.com/Willis0826/cubic-wiki/blob/main/README.md) - [package.json](https://github.com/Willis0826/cubic-wiki/blob/main/package.json) - [tsconfig.json](https://github.com/Willis0826/cubic-wiki/blob/main/tsconfig.json)

Immediately after the

block, the main title of the page should be a H1 Markdown heading: # System Architecture. Based ONLY on the content of the [RELEVANT_SOURCE_FILES]: 1. **Introduction:** Start with a concise introduction (1-2 paragraphs) explaining the purpose, scope, and high-level overview of "System Architecture" within the context of the overall project. If relevant, and if information is available, link to other potential wiki pages using the format [Link Text](#page-anchor-or-id). 2. **Detailed Sections:** Break down "System Architecture" into logical sections using H2 (##) and H3 (###) Markdown headings. For each section: * Explain the architecture, components, data flow, or logic relevant to the section's focus, as evidenced in the source files. * Identify key functions, classes, data structures, API endpoints, or configuration elements pertinent to that section. 3. **Mermaid Diagrams:** * EXTENSIVELY use Mermaid diagrams (e.g., flowchart TD, sequenceDiagram, classDiagram, erDiagram, graph TD) to visually represent architectures, flows, relationships, and schemas found in the source files. * Ensure diagrams are accurate and directly derived information from the [RELEVANT_SOURCE_FILES]. * Provide a brief explanation before or after each diagram to give context. * CRITICAL diagram requirements for optimal rendering: - **Flow diagrams**: Use "graph TD" (top-down) or "flowchart TD" directives for better vertical layout - **Node labels**: Keep node labels concise but descriptive (under 25 characters for readability) - **Node IDs**: Use alphanumeric characters and underscores only (avoid spaces, special characters) - **Connections**: Use standard arrow syntax (-->, ---, -.->) for reliable rendering - **Sequence diagrams**: - Start with "sequenceDiagram" directive on its own line - Use participant names without spaces (use underscores if needed) - Define participants explicitly when needed: `participant User as "User Interface"` - Use correct arrow types: - ->> for synchronous calls/requests - -->> for responses - -x for failed/error messages - --x for failed responses - Add activation boxes sparingly with +/- notation - Use "Note over" or "Note right of" for clarifications - **Class diagrams**: Use standard UML notation with proper relationship syntax - **ER diagrams**: Use entity relationship syntax with proper cardinality notations * **Quality guidelines**: - Test diagram syntax mentally before including - If a diagram becomes too complex, split into multiple focused diagrams - Ensure proper closing of all brackets and quotes - Use consistent naming conventions within each diagram - Add line breaks for readability in complex diagrams * **Example patterns (GOOD vs BAD)**: ``` GOOD Flow Diagram: ```mermaid flowchart TD A[User Request] --> B[Validate Input] B --> C[Process Data] C --> D[Return Response] ``` BAD Flow Diagram: ```mermaid graph LR A[User Request with Very Long Description] --> B ``` GOOD Sequence Diagram: ```mermaid sequenceDiagram participant User as "User Interface" participant API as "API Server" participant DB as "Database" User->>API: POST /api/data API->>DB: SELECT * FROM table DB-->>API: Query results API-->>User: JSON response ``` ``` 4. **Tables:** * Use Markdown tables to summarise information such as: * Key features or components and their descriptions. * API endpoint parameters, types, and descriptions. * Configuration options, their types, and default values. * Data model fields, types, constraints, and descriptions. 5. **Code Snippets (ENTIRELY OPTIONAL):** * Include short, relevant code snippets (e.g., Python, Java, JavaScript, SQL, JSON, YAML) directly from the [RELEVANT_SOURCE_FILES] to illustrate key implementation details, data structures, or configurations. * Ensure snippets are well-formatted within Markdown code blocks with appropriate language identifiers. 6. **Source Citations (EXTREMELY IMPORTANT):** * For EVERY piece of significant information, explanation, diagram, table entry, or code snippet, you MUST cite the specific source file(s) and relevant line numbers from which the information was derived. * Place citations at the end of the paragraph, under the diagram/table, or after the code snippet. * Use the exact format: `Sources: [filename.ext:start_line-end_line]()` for a range, or `Sources: [filename.ext:line_number]()` for a single line. Multiple files can be cited: `Sources: [file1.ext:1-10](), [file2.ext:5](), [dir/file3.ext]()` (if the whole file is relevant and line numbers are not applicable or too broad). * If an entire section is overwhelmingly based on one or two files, you can cite them under the section heading in addition to more specific citations within the section. * You MUST cite AT LEAST 5 different source files throughout the wiki page to ensure comprehensive coverage. 7. **Technical Accuracy:** All information must be derived SOLELY from the [RELEVANT_SOURCE_FILES]. Do not infer, invent, or use external knowledge about similar systems or common practices unless it's directly supported by the provided code. If information is not present in the provided files, do not include it or explicitly state its absence if crucial to the topic. 8. **Clarity and Conciseness:** Use clear, professional, and concise technical language suitable for other developers working on or learning about the project. Avoid unnecessary jargon, but use correct technical terms where appropriate. 9. **Conclusion/Summary:** End with a brief summary paragraph if appropriate for "System Architecture", reiterating the key aspects covered and their significance within the project. IMPORTANT: Generate the content in English language. Remember: - Ground every claim in the provided source files. - Prioritize accuracy and direct representation of the code's functionality and structure. - Structure the document logically for easy understanding by other developers. Total Files Included: 24 Total Files Skipped: 0 Total Context Tokens: 15,835 / 800,000 (2.0%) Remaining Token Budget: 784,165 ### Included Files: 1. README.md (968 tokens) 2. src/app/api/generate-v2/route.ts (1,441 tokens) 3. src/lib/llm.ts (2,579 tokens) 4. src/app/api/chat/route.ts (1,095 tokens) 5. src/lib/github-utils.ts (969 tokens) 6. src/app/api/generate/route.ts (660 tokens) 7. src/lib/wiki-utils.ts (762 tokens) 8. package.json (531 tokens) 9. src/app/wiki/page.tsx (502 tokens) 10. src/components/repo-input/repo-input.tsx (1,133 tokens) 11. src/app/wiki/[id]/page.tsx (619 tokens) 12. prisma/schema.prisma (450 tokens) 13. src/components/subsystem-card/subsystem-card.tsx (1,064 tokens) 14. src/app/api/generate-subsystem-summary/route.ts (581 tokens) 15. src/app/layout.tsx (274 tokens) 16. src/components/chat/chat-widget.tsx (1,115 tokens) 17. src/components/navigation/navigation.tsx (212 tokens) 18. src/components/subsystem-card/components/get-more-insights-button/get-more-insights-button.tsx (395 tokens) 19. src/components/mantine-theme/mantine-theme.tsx (32 tokens) 20. tsconfig.json (184 tokens) ... and 4 more files ================================================================================ FILE: README.md Relevance Score: 0.990 | Chunks Retrieved: 1 | Tokens: 968 ================================================================================ # Cubic Wiki A GitHub repository analysis platform that uses LLMs to intelligently group files into subsystems and generate comprehensive summaries. ## Tech Stack - **Frontend**: Next.js 15, React 19, TypeScript - **UI Components**: Mantine UI, Tailwind CSS - **Backend**: Supabase, Prisma ORM - **AI**: OpenAI GPT-4o-mini - **Deployment**: Vercel - **Package Manager**: pnpm ## Getting Started ### Prerequisites 1. **Database Setup**: Create a database on Supabase or run PostgreSQL with Docker locally 2. **OpenAI API Key**: Get an OpenAI API key with sufficient credits 3. **GitHub Token**: Create a GitHub personal access token 4. **Environment Variables**: Create a `.env` file in the root directory ### Environment Configuration ```env # OpenAI OPENAI_API_KEY= OPENAI_MODEL=gpt-4o-mini # GitHub GITHUB_TOKEN= # Database DATABASE_URL= ``` ### Installation & Development 1. Install dependencies: `pnpm install` 2. Init database schemas: `pnpm prisma:push` 3. Start the development server: `pnpm dev` ## How It Works ### Repository Analysis Cubic Wiki offers two analysis modes to generate comprehensive documentation for GitHub repositories: #### 🔍 Basic Analysis (V1) - **Method**: File path and README-based analysis - **Process**: - Retrieves repository file structure and README content - Groups files by top-level directories - Uses LLM to intelligently organize files into logical subsystems - Generates comprehensive overview of repository structure - **Advantages**: Fast, cost-effective, suitable for most repositories - **Best for**: Quick documentation generation and repositories with clear file organization #### ✨ Advanced Analysis (V2) - Beta - **Method**: Full file content analysis with AI clustering - **Process**: - Downloads the entire codebase content - Uses AI to identify and select the most important files (up to 50) - Generates individual file summaries and embeddings - Applies K-means clustering to group related files into subsystems - Creates intelligent file groupings based on semantic similarity - Provides deeper insights into code architecture and patterns - **Advantages**: More accurate subsystem identification, better understanding of code relationships - **Best for**: Complex repositories, detailed architectural analysis - **Note**: Request timeout is 60 seconds, larger repositories may fail to generate ### Deep Dive Analysis For specific subsystems, users can request "Get More Insights" to: - Send all relevant files to the LLM for detailed analysis - Generate in-depth insights about the subsystem's architecture and functionality - Provide recommendations and understanding of the codebase - Create Mermaid diagrams to visualize system architecture ### 🤖 AI Chat Assistant Every wiki page includes an intelligent chat widget that allows users to: - **Ask Questions**: Get instant answers about the repository structure, subsystems, and files - **Search Codebase**: Find specific information across all analyzed files and subsystems - **Get Details**: Request detailed information about specific subsystems or files - **Context-Aware**: The AI assistant has full context of the wiki page content and can provide relevant, accurate responses - **Multi-Step Reasoning**: Supports complex queries with up to 3-step tool calls for comprehensive answers - **Real-time Streaming**: Responses are streamed in real-time for a smooth user experience The chat assistant uses the same AI model and has access to all the analyzed data, making it a powerful tool for understanding and exploring codebases. ## Features - **Dual Analysis Modes**: Choose between fast path-based analysis (V1) or comprehensive content-based analysis (V2) - **Intelligent File Grouping**: LLM-powered subsystem identification with semantic clustering - **Comprehensive Summaries**: Detailed analysis of repository structure and architecture - **Deep Dive Capabilities**: Subsystem-specific insights with Mermaid diagram generation - **AI Chat Assistant**: Interactive chat widget for asking questions about the codebase - **Smart Search**: AI-powered search across subsystems, files, and summaries - **File Content Analysis**: Advanced V2 mode analyzes actual code content for better accuracy - **AI-Powered Clustering**: K-means clustering groups related files based on semantic similarity - **Smart File Selection**: Automatically identifies and prioritizes the most important files - **Modern Tech Stack**: Built with Next.js 15, TypeScript, and Mantine UI - **Scalable Architecture**: Prisma ORM with Supabase backend and vector embeddings ================================================================================ ================================================================================ FILE: src/app/api/generate-v2/route.ts Relevance Score: 0.980 | Chunks Retrieved: 1 | Tokens: 1,441 ================================================================================ import { NextRequest, NextResponse } from "next/server"; import { generateShortSummary, generateSubsystemsFromFileSummary, generateSummaryFromReadme, generateFileSummary, generateFileEmbedding, generateImportantFiles, } from "@/lib/llm"; import pLimit from "p-limit"; import { kmeans } from "ml-kmeans"; import { repoUrlSchema, parseGitHubUrl, getRepoMetadata, getReadmeContent, createWikiPageData, getFileFromRepoZip, } from "@/lib/github-utils"; import { findExistingWikiPage, updateWikiPageWithFiles, createWikiPageWithFiles, } from "@/lib/wiki-utils"; // Limit the number of concurrent requests to 10 const limit = pLimit(10); // File content based generation // This is the new generation method, it's more accurate but more expensive // Pro feature export async function POST(req: NextRequest) { try { const json = await req.json(); const { repoUrl } = repoUrlSchema.parse(json); const { owner, repo } = parseGitHubUrl(repoUrl); // Get default branch const { branch } = await getRepoMetadata(owner, repo); // Get file content from repo zip const fileContents = await getFileFromRepoZip(repoUrl); // Filter out the files by the file name const filteredFileContents = fileContents.filter((file) => { if ( file.path.startsWith("dist/") || file.path.endsWith(".lock") || file.path.includes("package-lock.json") || file.path.includes("pnpm-lock.yaml") || file.path.includes("node_modules/") || file.path.endsWith(".exe") || file.path.endsWith(".png") || file.path.endsWith(".jpg") || file.path.endsWith(".jpeg") || file.path.endsWith(".gif") || file.path.endsWith(".svg") || file.path.endsWith(".ico") || file.path.endsWith(".webp") || file.path.endsWith(".mp4") || file.path.endsWith(".mp3") || file.path.endsWith(".wav") || file.path.endsWith(".ogg") || file.path.endsWith(".flac") || file.path.endsWith(".webm") || file.path.endsWith(".mov") || file.path.endsWith(".zip") || file.path.endsWith(".tar") || file.path.endsWith(".gz") || file.path.endsWith(".bz2") || file.path.endsWith(".xz") || file.path.endsWith(".7z") || file.path.endsWith(".rar") || file.path.endsWith(".iso") ) { return false; } return true; }); // Ask LLM to only keep the important files for inferring the subsystems if the file count is more than 50 let importantFiles: { path: string; content: string }[] = []; if (filteredFileContents.length > 50) { importantFiles = await generateImportantFiles(filteredFileContents); } else { importantFiles = filteredFileContents; } // Generate summary, embedding, and save to database for each file // Using plimit to limit the number of concurrent requests to 10 const fileSummaries = await Promise.all( importantFiles.map((file) => limit(() => generateFileSummary(file.content).catch((err) => { // skip the error, it could be token limit error console.error(err); return ""; }) ) ) ); const fileEmbedding = await Promise.all( fileSummaries.map((summary) => generateFileEmbedding(summary)) ); // Clustering the files // Try to create 1 cluster per 5 files // Try to create at least 2 clusters and at most 8 clusters const K = Math.min(8, Math.max(2, Math.floor(fileEmbedding.length / 5))); // heuristic const clustering = kmeans(fileEmbedding, K, {}); // Prepare empty clusters to fill in with files const clusters: { files: { path: string; summary: string; embedding: number[] }[]; }[] = Array.from({ length: K }, (_, i) => ({ files: [], // initially empty centroid: clustering.centroids[i], // the center of the cluster })); // Assign each file to the closest cluster for (let i = 0; i < fileEmbedding.length; i++) { const embedding = fileEmbedding[i]; const path = importantFiles[i].path; const summary = fileSummaries[i]; // Which cluster this file belongs to const clusterIndex = clustering.clusters[i]; // Add the file to the right cluster clusters[clusterIndex].files.push({ path, summary, embedding, }); } // Generate subsystems for each cluster const subsystems = ( await Promise.all( clusters.map((cluster) => generateSubsystemsFromFileSummary(cluster.files) ) ) ).filter((subsystem) => subsystem !== null); // Fetch README, readme is always an important part to understand the codebase const decoded = await getReadmeContent(owner, repo); let summary = ""; try { summary = await generateSummaryFromReadme(decoded); } catch { summary = ""; } // Generate short summary for the wiki page list let shortSummary = ""; try { shortSummary = await generateShortSummary(summary); } catch { shortSummary = ""; } // Check if a wiki page already exists for this repository const existingWikiPage = await findExistingWikiPage(repoUrl); const wikiPageData = createWikiPageData( repoUrl, branch, owner, repo, summary, shortSummary ); if (existingWikiPage) { // Update the existing wiki page and replace all subsystems const updatedWikiPage = await updateWikiPageWithFiles( existingWikiPage.id, wikiPageData, subsystems, importantFiles, fileSummaries, fileEmbedding ); return NextResponse.json({ id: updatedWikiPage.id, }); } // Create new wiki page with subsystems const wikiPage = await createWikiPageWithFiles( wikiPageData, subsystems, importantFiles, fileSummaries, fileEmbedding ); return NextResponse.json({ id: wikiPage.id, }); } catch (err: unknown) { console.error(err); // Type-safe error handling const errorMessage = err instanceof Error ? err.message : "An unknown error occurred"; return NextResponse.json({ error: errorMessage }, { status: 500 }); } } ================================================================================ ================================================================================ FILE: src/lib/llm.ts Relevance Score: 0.970 | Chunks Retrieved: 2 | Tokens: 2,579 ================================================================================ import OpenAI from "openai"; const TEMPERATURE = 0.2; // more deterministic const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY, }); export const generateSummaryFromReadme = async (readme: string) => { const systemPrompt = ` You are a helpful assistant that generates a summary for a given README. The summary should be a short description of the repo, but also include the main features and the main purpose of the repo. The summary should be in markdown format. The summary should be in English. `; const completion = await openai.chat.completions.create({ model: process.env.OPENAI_MODEL || "gpt-4o-mini", messages: [ { role: "system", content: systemPrompt, }, { role: "user", content: readme, }, ], temperature: TEMPERATURE, }); return completion.choices[0].message.content || ""; }; export const generateShortSummary = async (summary: string) => { const systemPrompt = ` You are a helpful assistant that generates a short summary for a given summary. The short summary should be a short description of the summary. The short summary should be in plain text. The short summary should be no more than 100 words. The short summary should be in English. `; const completion = await openai.chat.completions.create({ model: process.env.OPENAI_MODEL || "gpt-4o-mini", messages: [ { role: "system", content: systemPrompt }, { role: "user", content: summary }, ], temperature: TEMPERATURE, }); return completion.choices[0].message.content || ""; }; export const generateSubsystemsFromFilePaths = async ( fileMap: Record, readmeSummary: string ) => { const systemPrompt = ` You are a senior software architect documenting a code base. **Goal:** Group files into HIGH-LEVEL, FEATURE-ORIENTED subsystems that an engineer would look for: Authentication, Billing, CLI, Data-Layer, etc. Avoid buckets that are purely technical (e.g. "components", "utils") unless they *are* a standalone feature. **Output rules** • Return ONLY valid JSON – array of objects { "title", "shortSummary", "files" }. • Don't include any code blocks \`\`\` in the output. • 3-8 subsystems total. • Every file path must appear in exactly one subsystem. • Think step-by-step internally, but do NOT include that reasoning in the reply. **Example** README summary: "A wiki page generator that lets developers understand any repo quickly." Files: { "src": ["cli.ts", "auth/email.ts", "auth/password.ts", "db/index.ts"], "prisma": ["schema.prisma"] } → [ { "title": "CLI Tool", "shortSummary": "Entry point users run to generate docs", "files": ["src/cli.ts"] }, { "title": "Authentication", "shortSummary": "Email + password login for the web UI", "files": ["src/auth/email.ts", "src/auth/password.ts"] }, { "title": "Database Layer", "shortSummary": "Prisma schema and helpers", "files": ["src/db/index.ts", "prisma/schema.prisma"] } ]`.trim(); const completion = await openai.chat.completions.create({ model: process.env.OPENAI_MODEL || "gpt-4o-mini", messages: [ { role: "system", content: systemPrompt, }, { role: "user", content: `README summary: ${readmeSummary} Files: ${JSON.stringify(fileMap)}`, }, ], temperature: TEMPERATURE, }); try { const subsystems = JSON.parse( completion.choices[0].message.content || "[]" ) as { title: string; shortSummary: string; files: string[]; }[]; return subsystems; } catch (error) { console.error(error); return []; } }; export const generateImportantFiles = async ( fileContents: { path: string; content: string; }[] ) => { const systemPrompt = ` You are a software engineer analysing a codebase. Your job is to select the most important files from a given list of file paths. These files should help identify the major subsystems / features of the repository. **Constraints:** - ONLY choose file paths from the list provided by the user. - DO NOT invent or guess file paths that are not in the list. - If you don't see a README or index file in the list, don't include one. - Only return files that exist in the list. - The file paths are case-sensitive. - DO NOT include any binary or asset files such as images, audio, video, font files, or minified builds. - Exclude files with extensions like: .png, .jpg, .jpeg, .svg, .gif, .webp, .ico, .mp3, .mp4, .wav, .ogg, .ttf, .woff, .woff2, .eot, .zip, .min.js, etc. **Output:** - A raw JSON array of strings (e.g., ["src/index.ts", "README.md"]). - NO markdown, no commentary, no explanations. - Limit to 50 files max. Only output valid JSON. `; const completion = await openai.chat.completions.create({ model: process.env.OPENAI_MODEL || "gpt-4o-mini", messages: [ { role: "system", content: systemPrompt }, { role: "user", content: JSON.stringify(fileContents.map((file) => file.path)), }, ], temperature: TEMPERATURE, }); try { const importantFiles = JSON.parse( completion.choices[0].message.content || "[]" ) as string[]; // Filter out the files that are not in the important files // Take care the case sensitivity of the file paths const fileMap = new Map(fileContents.map((f) => [f.path.toLowerCase(), f])); const filteredImportantFiles = importantFiles .map((p) => fileMap.get(p.toLowerCase())) .filter((f): f is { path: string; content: string } => !!f); console.log(filteredImportantFiles.map((file) => file.path)); console.log(filteredImportantFiles.length); return filteredImportantFiles; } catch (error) { console.error(error); return []; } }; export const generateSubsystemsFromFileSummary = async ( fileContents: { path: string; // The content of the file summary: string; }[] ) => { const systemPrompt = ` You are a software engineer generating a subsystem from a given files. The most important part is to understand the main features and the main purpose of the provided files array. The should focus on the main features and the main purpose of the provided files array. **Goal:** Analysis files by their summary and file paths and generate a concise for - Title for the subsystem - Short summary (1–2 sentences) for the subsystem **Output rules** • Return ONLY valid JSON object { "title", "shortSummary" }. • Don't include any code blocks \`\`\` in the output. • There should be only one subsystem given all the summaries of the files are highly related to each other. • Think step-by-step internally, but do NOT include that reasoning in the reply. **Example** Files: [ { "path": "src/cli.ts", "summary": "A useful CLI tool to help user login to the web UI" }, { "path": "src/auth/email.ts", "summary": "Email login for the web UI" }, { "path": "src/auth/password.ts", "summary": "Password login for the web UI" }, { "path": "src/db/index.ts", "summary": "User and Email database operations" }, { "path": "prisma/schema.prisma", "summary": "Prisma schema for the database" } ] → { "title": "Authentication", "shortSummary": "Email + password login for the web UI" } `; const completion = await openai.chat.completions.create({ model: process.env.OPENAI_MODEL || "gpt-4o-mini", messages: [ { role: "system", content: systemPrompt }, { role: "user", content: `Files: ${JSON.stringify( fileContents.map((file) => ({ path: file.path, summary: file.summary, })) )}`, }, ], }); try { const subsystem = JSON.parse( completion.choices[0].message.content || "{}" ) as { title: string; shortSummary: string; }; return { ...subsystem, files: fileContents.map((file) => file.path) }; } catch (error) { console.error(error); return null; } }; export const generateSubsystemSummaryFromFiles = async ( title: string, files: { path: string; content: string; }[] ) => { const systemPrompt = ` You are a helpful assistant that generates a summary for a given files. The files are related to a specific subsystem with a title. The summary should provide a clear and concise description that can help the user to understand the subsystem. The summary should be in markdown format. The summary should be in English. If you have enough information, you can also generate a diagram to help the user to understand the subsystem. The diagram should be in mermaid format. Example: Title: Authentication Files: - src/auth/index.ts import { User } from "@/auth/user"; ... - src/auth/user.ts import { crypto } from "@/auth/crypto"; ... Summary: This subsystem handles authentication and password management. It includes the following files: - src/auth/index.ts: Main authentication module - src/auth/user.ts: User management functionality - src/auth/crypto.ts: Cryptography functionality Diagram: \`\`\`mermaid graph TD \`\`\` `; const completion = await openai.chat.completions.create({ model: process.env.OPENAI_MODEL || "gpt-4o-mini", messages: [ { role: "system", content: systemPrompt }, { role: "user", content: JSON.stringify({ title, files }) }, ], temperature: TEMPERATURE, }); return completion.choices[0].message.content || ""; }; export const generateFileSummary = async (fileContent: string) => { const systemPrompt = ` You are a helpful assistant that generates a summary for a given file. The summary should be a short description of the file. The summary should be in plain text. The summary should be in English. The summary should be no more than 100 words. `; const completion = await openai.chat.completions.create({ model: process.env.OPENAI_MODEL || "gpt-4o-mini", messages: [ { role: "system", content: systemPrompt }, { role: "user", content: fileContent }, ], temperature: TEMPERATURE, }); return completion.choices[0].message.content || ""; }; export const generateFileEmbedding = async (summary: string) => { const embedRes = await openai.embeddings.create({ model: "text-embedding-3-small", input: summary, }); const embedding = embedRes.data[0].embedding; return embedding; }; ================================================================================ ================================================================================ FILE: src/app/api/chat/route.ts Relevance Score: 0.960 | Chunks Retrieved: 1 | Tokens: 1,095 ================================================================================ import { streamText, tool } from "ai"; import { openai } from "@ai-sdk/openai"; import { z } from "zod"; import { prisma } from "@/lib/prisma"; // Allow streaming responses up to 30 seconds export const maxDuration = 30; export async function POST(req: Request) { const { messages, wikiId } = await req.json(); if (!wikiId) { return Response.json({ error: "Wiki ID is required" }, { status: 400 }); } // Get the wiki page and its related data const wiki = await prisma.wikiPage.findUnique({ where: { id: parseInt(wikiId) }, include: { Subsystems: { orderBy: { id: "asc" }, }, Files: { orderBy: { path: "asc" }, }, }, }); if (!wiki) { return Response.json({ error: "Wiki page not found" }, { status: 404 }); } // Create context from wiki data const context = ` Wiki Page: ${wiki.title} Repository: ${wiki.repoUrl} (branch: ${wiki.branch}) Summary: ${wiki.summary} Subsystems: ${wiki.Subsystems.map( (subsystem) => ` - ${subsystem.title}: ${subsystem.shortSummary} Files: ${Array.isArray(subsystem.files) ? subsystem.files.join(", ") : "N/A"} ` ).join("\n")} Files: ${wiki.Files.map( (file) => ` - ${file.path}: ${file.summary} ` ).join("\n")} `.trim(); const result = streamText({ model: openai("gpt-4o-mini"), messages, system: `You are a helpful AI assistant for a wiki page. You have access to the following information about this specific wiki page: ${context} Please answer questions based on this information. If the question is not related to this wiki page or you don't have enough information, politely let the user know. Keep your answers concise and relevant to the wiki page content. You can use tools to get more specific information about the codebase.`, tools: { searchCodebase: tool({ description: "Search for specific information in the codebase, subsystems, or files", parameters: z.object({ query: z .string() .describe("The search query to find relevant information"), }), execute: async ({ query }) => { // Simple text-based search through the wiki data const searchResults = []; // Search in summary if (wiki.summary.toLowerCase().includes(query.toLowerCase())) { searchResults.push(`Summary: ${wiki.summary}`); } // Search in subsystems const matchingSubsystems = wiki.Subsystems.filter( (subsystem) => subsystem.title.toLowerCase().includes(query.toLowerCase()) || subsystem.shortSummary.toLowerCase().includes(query.toLowerCase()) ); matchingSubsystems.forEach((subsystem) => { searchResults.push( `Subsystem - ${subsystem.title}: ${subsystem.shortSummary}` ); }); // Search in files const matchingFiles = wiki.Files.filter( (file) => file.path.toLowerCase().includes(query.toLowerCase()) || file.summary.toLowerCase().includes(query.toLowerCase()) ); matchingFiles.forEach((file) => { searchResults.push(`File - ${file.path}: ${file.summary}`); }); return searchResults.length > 0 ? searchResults.join("\n\n") : "No relevant information found for this query."; }, }), getSubsystemDetails: tool({ description: "Get detailed information about a specific subsystem", parameters: z.object({ subsystemName: z .string() .describe("The name of the subsystem to get details for"), }), execute: async ({ subsystemName }) => { const subsystem = wiki.Subsystems.find((s) => s.title.toLowerCase().includes(subsystemName.toLowerCase()) ); if (!subsystem) { return `Subsystem "${subsystemName}" not found. Available subsystems: ${wiki.Subsystems.map( (s) => s.title ).join(", ")}`; } return `Subsystem: ${subsystem.title} Description: ${subsystem.shortSummary} Files: ${Array.isArray(subsystem.files) ? subsystem.files.join(", ") : "N/A"}`; }, }), getFileDetails: tool({ description: "Get detailed information about a specific file", parameters: z.object({ fileName: z .string() .describe("The name or path of the file to get details for"), }), execute: async ({ fileName }) => { const file = wiki.Files.find((f) => f.path.toLowerCase().includes(fileName.toLowerCase()) ); if (!file) { return `File "${fileName}" not found. Available files: ${wiki.Files.map( (f) => f.path ).join(", ")}`; } return `File: ${file.path} Summary: ${file.summary}`; }, }), }, maxSteps: 3, // Allow multi-step tool calls }); return result.toDataStreamResponse(); } ================================================================================ ================================================================================ FILE: src/lib/github-utils.ts Relevance Score: 0.950 | Chunks Retrieved: 1 | Tokens: 969 ================================================================================ import { Octokit } from "octokit"; import { z } from "zod"; import JSZip from "jszip"; // Zod schema for safety export const repoUrlSchema = z.object({ repoUrl: z.string().url(), }); // Create Octokit client export const octokit = new Octokit({ auth: process.env.GITHUB_TOKEN, // Disable throttling, otherwise the request will hang until the rate limit is reset throttle: { onRateLimit: () => false, onSecondaryRateLimit: () => false }, }); // GitHub URL parsing utility export function parseGitHubUrl(repoUrl: string) { const match = repoUrl.match( /github\.com\/([^\/]+)\/([^\/]+?)(?:\.git)?(?:\/|$)/ ); if (!match) { throw new Error("Invalid GitHub URL"); } return { owner: match[1], repo: match[2], }; } // Get repository metadata export async function getRepoMetadata(owner: string, repo: string) { const repoMeta = await octokit.rest.repos.get({ owner, repo }); return { branch: repoMeta.data.default_branch, }; } // Get repository tree export async function getRepoTree(owner: string, repo: string, branch: string) { const treeRes = await octokit.rest.git.getTree({ owner, repo, tree_sha: branch, recursive: "true", }); return treeRes.data.tree.filter((item) => item.type === "blob"); } // Get README content export async function getReadmeContent(owner: string, repo: string) { const readme = await octokit.rest.repos.getReadme({ owner, repo }); return Buffer.from(readme.data.content, "base64").toString("utf-8"); } // Get file content utility (for generate-v2) export async function getFileContent(repoUrl: string, filePath: string) { try { const { owner, repo } = parseGitHubUrl(repoUrl); const response = await octokit.rest.repos.getContent({ owner, repo, path: filePath, }); // Handle the case where response.data might be an array if (Array.isArray(response.data)) { return null; } if (response.data.type !== "file") { return null; } return Buffer.from(response.data.content || "", "base64").toString("utf-8"); } catch (error) { console.error(`Failed to fetch file ${filePath}:`, error); return null; } } // Get file content from repo zip, for better performance and handling rate limit export async function getFileFromRepoZip( repoUrl: string ): Promise<{ path: string; content: string }[]> { const { owner, repo } = parseGitHubUrl(repoUrl); // Get default branch const repoMeta = await octokit.rest.repos.get({ owner, repo }); const branch = repoMeta.data.default_branch; // Download the ZIP archive const response = await octokit.request( `GET /repos/${owner}/${repo}/zipball/${branch}`, { owner, repo, ref: branch, headers: { Accept: "application/vnd.github.v3.raw" }, } ); // Extract ZIP contents const zip = await JSZip.loadAsync(Buffer.from(response.data)); const files: { path: string; content: string }[] = []; for (const fullPath in zip.files) { const entry = zip.files[fullPath]; if (entry.dir) continue; // Strip GitHub-added root directory (e.g., "owner-repo-sha/") const relativePath = fullPath.substring(fullPath.indexOf("/") + 1); try { const content = await entry.async("string"); files.push({ path: relativePath, content }); } catch { // Skip unreadable (likely binary) files } } return files; } // Common wiki page data structure export interface WikiPageData { repoUrl: string; branch: string; title: string; summary: string; shortSummary: string; } // Create wiki page data export function createWikiPageData( repoUrl: string, branch: string, owner: string, repo: string, summary: string, shortSummary: string ): WikiPageData { return { repoUrl, branch, title: `${owner}/${repo}`, summary, shortSummary, }; } ================================================================================ ================================================================================ FILE: src/app/api/generate/route.ts Relevance Score: 0.940 | Chunks Retrieved: 1 | Tokens: 660 ================================================================================ import { NextRequest, NextResponse } from "next/server"; import { generateShortSummary, generateSubsystemsFromFilePaths, generateSummaryFromReadme, } from "@/lib/llm"; import { repoUrlSchema, parseGitHubUrl, getRepoMetadata, getRepoTree, getReadmeContent, createWikiPageData, } from "@/lib/github-utils"; import { findExistingWikiPage, updateWikiPage, createWikiPage, } from "@/lib/wiki-utils"; // File path based generation // This is the old generation method, it's cheaper but less accurate export async function POST(req: NextRequest) { try { const json = await req.json(); const { repoUrl } = repoUrlSchema.parse(json); const { owner, repo } = parseGitHubUrl(repoUrl); // Get default branch const { branch } = await getRepoMetadata(owner, repo); // Fetch README, readme is always an important part to understand the codebase const decoded = await getReadmeContent(owner, repo); let summary = "";