Embedding Generation
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
Embedding Generation in CodeScholar is the process of transforming academic paper abstracts and search queries into high-dimensional numerical vectors. This system enables semantic search capabilities, allowing AI agents and researchers to find relevant research based on the conceptual meaning of text rather than simple keyword matching. The primary implementation utilizes OpenAI's large-scale embedding models to populate a unified library of research data.
Sources: scripts/generate_embeddings.py:1-10, server.js:14-25
The generated embeddings serve as the foundation for the project's Hybrid Search architecture, which combines vector similarity with traditional text matching. This ensures that the Model Context Protocol (MCP) server can provide context-aware research recommendations to developers based on their code context or natural language queries.
Sources: server.js:115-130, scripts/generate_embeddings.py:192-195
Architecture and Workflow
The embedding generation process is structured as a pipeline that interacts with external AI services while enforcing strict rate limits to ensure stability and cost-effectiveness.
Embedding Generation Pipeline
The pipeline follows a sequential flow from raw JSON data to an enriched library containing 3072-dimensional vectors.
The diagram shows the logic flow from loading the library to saving the final embedded results.
Sources: scripts/generate_embeddings.py:127-148
Key Components
The system relies on several core components to manage the embedding lifecycle:
| Component | Responsibility | Technical Details |
|---|---|---|
| OpenAIEmbedder | Manages API communication and rate limiting. | Uses text-embedding-3-large. |
| AsyncOpenAI | Asynchronous client for non-blocking API calls. | Rate limited to 50 req/sec. |
| Batch Processor | Handles progress tracking and bulk updates. | Processes in batches of 50. |
| LocalIndex (Vectra) | Stores and queries vectors locally for the server. | Uses cosine similarity. |
Sources: scripts/generate_embeddings.py:27-45, server.js:37-45
Implementation Details
Model Configuration
The project standardizes on specific parameters for embedding generation to ensure high retrieval accuracy.
- Model:
text-embedding-3-large - Dimensions: 3072
- Rate Limit Delay: 0.02s (3000 RPM)
- Encoding Format: float
Sources: scripts/generate_embeddings.py:12-16
Core Embedding Logic
The OpenAIEmbedder class encapsulates the logic for converting text to vectors. It includes a specific rate_limit method to stay within the 50 requests per second tier.
async def generate_embedding(self, text: str) -> Optional[List[float]]:
if not text or not text.strip():
return None
await self.rate_limit()
try:
response = await self.client.embeddings.create(
model=EMBEDDING_MODEL,
input=text,
encoding_format="float"
)
return response.data[0].embedding
except Exception as e:
print(f"[ERROR] Error: {e}")
return None
Sources: scripts/generate_embeddings.py:53-75
Data Usage and Storage
Embeddings are stored within the analysis.embedding field of the paper objects in the unified_library.json file.
Metadata Schema with Embeddings
When papers are processed, they are updated with specific analysis metadata used by the search engine.
| Field | Type | Description |
|---|---|---|
abstract | string | The source text used for embedding. |
analysis.embedding | array[float] | The 3072-dimensional vector. |
id | UUID | Unique identifier used to link vectors to metadata. |
Sources: scripts/generate_embeddings.py:92-96, server.js:48-55
Usage in Semantic Search
The server utilizes these embeddings to perform "Hybrid Search," which merges results from keyword matching and vector similarity.
The sequence shows how a user query is converted into an embedding in real-time to perform a semantic lookup.
Sources: server.js:115-135, server.js:630-650
Alternative Embedding Representations
While the primary pipeline uses OpenAI, the research documentation mentions alternative text representation techniques used in specific academic papers within the library, such as:
- GloWord_biGRU: A fusion model combining GloVe and Word2Vec embeddings for sentiment classification.
- DeepWalk: Graph embeddings used to quantify distances in knowledge spaces (e.g., Wikipedia).
Sources: knowledge/papers/fusion-text-representations-to-enhance-contextual-meaning-in-sentiment-classific.json:15-25, knowledge/papers/milgrams-experiment-in-the-knowledge-space-individual-navigation-strategies.json:18-24
Conclusion
Embedding generation is a critical preprocessing step that enables the CodeScholar platform to transition from basic keyword search to advanced semantic retrieval. By utilizing OpenAI's text-embedding-3-large model and a robust asynchronous pipeline, the system builds a searchable knowledge base capable of linking complex software engineering concepts to their theoretical research foundations.