Repository Analysis and Cloning
Relevant source files
The following files were used as context for generating this wiki page:
The Repository Analysis and Cloning feature is a core component of TestGen AI, responsible for fetching source code from a given GitHub repository. The primary goal is to gather relevant files and consolidate their content into a single "context" string. This context is then provided to the Google Gemini AI to generate relevant test cases.
The project contains two distinct implementations for this functionality: a server-side Go backend that performs a git clone, and a client-side TypeScript version that uses the GitHub REST API to fetch file contents directly in the browser. The client-side approach is the more recent implementation, designed to simplify deployment by removing the need for a separate backend server.
Sources: README.md:20-28, testgen/CLIENT_BACKEND.md:5-11
System Architecture
The system supports two architectural patterns for repository analysis. The initial design used a Go backend, while the current approach favors a client-side implementation that communicates directly with the GitHub API.
This diagram illustrates the two possible data flows for cloning a repository: the client-side path (blue) and the alternative Go backend path (green).
Sources: README.md:20-28, testgen/CLIENT_BACKEND.md:5-11, testgen-backend/main.go, testgen/src/lib/backend.ts
Go Backend Implementation
Sources: testgen-backend/main.go
The Go backend provides a traditional server-side approach. It exposes an API endpoint that, when called, clones the specified GitHub repository to the server's local filesystem, processes the files, and returns the result.
API Endpoint: POST /api/clone-repo
This endpoint orchestrates the entire cloning and analysis process on the server.
| Parameter | Type | Description |
|---|---|---|
repoUrl | string | The full URL of the GitHub repository to clone. |
Sources: testgen-backend/main.go:343-346, README.md:104-107
Response Body (RepoResponse)
{
"success": true,
"message": "Repository cloned successfully",
"filesCount": 25,
"contextPath": "repos/username-repo-context.txt",
"files": [...]
}
Sources: testgen-backend/main.go:23-29, README.md:109-116
Sequence of Operations
The following diagram shows the sequence of events when the /api/clone-repo endpoint is called.
*Sources: testgen-backend/main.go:341-404
Key Functions
| Function | Description | File:Line |
|---|---|---|
cloneRepoHandler | The main HTTP handler for the /api/clone-repo endpoint. | testgen-backend/main.go:341-404 |
parseGitHubURL | Cleans and parses a GitHub URL string to extract the owner and repository name. | testgen-backend/main.go:121-147 |
cloneRepository | Executes the git clone --depth 1 command to download the repository into a temporary local directory. | testgen-backend/main.go:149-163 |
readRepositoryFiles | Walks the cloned directory, filters files based on exclusion patterns and size, and reads their content. | testgen-backend/main.go:165-224 |
shouldExcludeFile | Checks if a given file path matches any of the defined exclusion patterns. | testgen-backend/main.go:95-112 |
generatePromptContext | Aggregates the content of all filtered files into a single string, formatted for the AI model. | testgen-backend/main.go:226-279 |
Client-Side Implementation
Sources: testgen/src/lib/backend.ts, testgen/src/lib/github.ts, testgen/CLIENT_BACKEND.md
The client-side implementation moves the repository analysis logic entirely into the browser, eliminating the need for a Go backend. It leverages the GitHub REST API to fetch repository contents without performing a git clone.
Class: ClientBackend
This class encapsulates all the logic for interacting with the GitHub API, filtering files, and generating the AI prompt context.
This diagram shows that the Dashboard.tsx component calls ApiService, which in turn uses the static methods of the ClientBackend class to perform the cloning operation.
Sources: testgen/src/lib/backend.ts:70, testgen/src/lib/api.ts:19, testgen/src/pages/Dashboard.tsx:81
Sequence of Operations
The following diagram shows the client-side sequence for fetching repository files.
Key Functions
| Function | Description | File:Line |
|---|---|---|
cloneRepository | The main entry point. It parses the URL and initiates the recursive file fetching process using the GitHub Contents API. | testgen/src/lib/backend.ts:223-279 |
fetchFilesRecursively | Recursively traverses directories. For each item, it checks if it's a file to be fetched or a directory to recurse into, respecting maxFiles and exclusion rules. | testgen/src/lib/backend.ts:149-221 |
fetchRepositoryFiles | An alternative implementation that uses the GitHub Git Trees API (/git/trees/{branch}?recursive=1) to get a flat list of all files, then fetches content for each. | testgen/src/lib/github.ts:98-175 |
shouldExcludeFile | Checks if a file path matches any of the defined exclusion patterns. | testgen/src/lib/backend.ts:130-142 |
createFallbackResponse | If no source files are found in a repository, this function generates a simple JavaScript example to ensure the user can still proceed with test generation. | testgen/src/lib/backend.ts:360-401 |
Core Logic and Components
File Filtering
Both implementations use a predefined list of patterns to exclude irrelevant files and directories from the analysis. This is crucial for keeping the AI context concise and focused on source code.
Exclusion Patterns (EXCLUDE_PATTERNS)
| Category | Examples |
|---|---|
| Dependencies | node_modules, package-lock.json, yarn.lock, go.mod, go.sum |
| Build Output | dist, build, coverage, .next, .cache |
| Git Internals | .git, .gitignore |
| Config/Docs | .vscode, *.md, LICENSE, tsconfig.json, webpack.config.* |
| Test/Assets | __tests__, test, docs, assets, public |
| Minified/Temp | *.min.js, *.log, *.tmp |
Sources: testgen-backend/main.go:68-85, testgen/src/lib/backend.ts:41-68, testgen/src/lib/github.ts:15-43
The filtering logic checks if any part of a file's path matches one of these patterns.
Context Generation
After all relevant files have been read, their contents are concatenated into a single string. This string is formatted with clear separators and file path headers to provide structured context to the Gemini AI. The Go implementation further organizes files by type (.go, config, other).
Example Generated Context Structure:
=== REPOSITORY CODE CONTEXT FOR TEST GENERATION ===
This context contains all source code files from the cloned repository.
Generate comprehensive test cases based on the functions, methods, and logic found in these files.
=== FILES ===
=== GO SOURCE FILES ===
// File: main.go
package main
...
---
=== OTHER FILES ===
// File: utils/helpers.js
function helper() { ... }
---
=== END OF CONTEXT ===
Sources: testgen-backend/main.go:226-279, testgen/src/lib/backend.ts:303-358
Frontend Integration
The process is initiated from the Dashboard.tsx component. The handleCloneRepository function is triggered when the user clicks the "fetch content" button.
// testgen/src/pages/Dashboard.tsx
const handleCloneRepository = async () => {
// ... validation ...
setIsCloning(true);
try {
// Clone repository using backend API
const response = await ApiService.cloneRepository(repoUrl);
// Generate prompt context from files using the backend
const promptContext = ClientBackend.generatePromptContext(response.files);
// Update state
setClonedFiles(response.files.map(f => f.path));
setCurrentRepo(`${repoInfo.owner}/${repoInfo.repo}`);
setCodeInput(promptContext);
// ... toast notification ...
} catch (error) {
// ... error handling ...
} finally {
setIsCloning(false);
}
};
Sources: testgen/src/pages/Dashboard.tsx:81-119
Summary
The Repository Analysis and Cloning feature is a versatile system designed to extract source code for AI processing. It has evolved from a Go-based server-side solution using git clone to a more streamlined, serverless client-side approach that leverages the GitHub API. Both implementations share core logic for URL parsing, file filtering, and context generation, ensuring that a clean, relevant, and well-structured code context is prepared for the subsequent test generation phase.