Utility Functions

Relevant source files

The following files were used as context for generating this wiki page:

• LLM.py
• searcher/sementic_search.py
• agents.py
• main.py
• prompts/juder_prompts.py

Utility functions in the CoI-Agent project provide essential support for string manipulation, data extraction, network communication, and mathematical operations. These functions serve as the foundational building blocks for higher-level agents like the DeepResearchAgent and ReviewAgent, enabling them to process unstructured LLM outputs, perform semantic calculations, and manage file downloads.

These utilities are distributed across core modules to handle specific tasks such as parsing XML-style tags from text, calculating vector similarities for paper re-ranking, and managing asynchronous HTTP requests for PDF retrieval.
Sources: LLM.py, searcher/sementic_search.py, agents.py

Text Parsing and Extraction

The system relies heavily on structured output from Large Language Models, often formatted with custom XML-like tags (e.g., <idea>, <relevant>). Utility functions ensure these tags are correctly parsed to extract the underlying data.

Tag-Based Extraction

The get_content_between_a_b function is a core utility used to locate and return text residing between a start tag and an end tag. This is wrapped by the extract function, which provides a more convenient interface for common extraction tasks across the codebase.

def extract(text, type):
    if text:
        target_str = get_content_between_a_b(f"<{type}>", f"</{type}>", text)
        if target_str:
            return target_str
        else:
            return text
    else:
        return ""

Sources: searcher/sementic_search.py:22-31, LLM.py:19-30

Logic Flow for Extraction

The following diagram illustrates how the extract utility processes LLM responses:

This logic ensures that if a specific tag (like novelty or feasibility) is found, only that portion is returned; otherwise, the function falls back to the full text to avoid data loss.
Sources: searcher/sementic_search.py:8-31, agents.py:38-46

Mathematical and Semantic Utilities

To facilitate research paper ranking and relevance assessment, the project includes utilities for vector mathematics and semantic similarity.

Cosine Similarity

Calculations for cosine similarity are implemented to compare embeddings (vectors) generated by LLMs. This is used specifically in re-ranking search results to ensure the most relevant papers are prioritized based on the user's research topic.

Function	Input Types	Description
cal_cosine_similarity	vec1, vec2 (list or np.array)	Calculates the cosine similarity between two individual vectors.
cal_cosine_similarity_matric	matric1, matric2 (list or np.array)	Performs batch similarity calculations between a query vector and a matrix of document embeddings.

Sources: searcher/sementic_search.py:102-120, LLM.py:104-108

Network and Data Retrieval

Utility functions manage the complexities of downloading scientific papers and interacting with external APIs like Semantic Scholar.

Asynchronous Fetching

The fetch utility provides a robust wrapper around aiohttp to download binary content (PDFs) with customized headers and timeout management. It mimics a standard browser User-Agent to prevent request blocking.

PDF and File Management

The download_pdf_async function leverages the fetching utility to save papers locally. It includes checks to prevent redundant downloads if a file already exists on the disk.
Sources: searcher/sementic_search.py:220-234

Environment and LLM Configuration

The project utilizes utility patterns for initializing Large Language Models based on configuration files or environment variables.

• get_llms(): Reads MAIN_LLM_MODEL and CHEAP_LLM_MODEL from environment variables, ensuring the system has both a high-capability model for reasoning and a cost-effective model for extraction tasks.
• Config Loading: In main.py, a utility loop iterates through config.yaml to populate os.environ, making settings available globally.

Sources: agents.py:15-26, main.py:11-17

Summary of Key Utility Components

Component	Location	Purpose
extract	utils.py, sementic_search.py	Parsers XML-like tags from LLM responses.
fetch	sementic_search.py	Asynchronous HTTP client for downloading paper PDFs.
get_llm	agents.py	Factory function for instantiating LLM clients.
cal_cosine_similarity	LLM.py	Mathematical utility for semantic relevance scoring.
parse_pdf_to_dict	sementic_search.py (via scipdf)	Utility to convert PDF files into structured JSON-like dictionaries.

Sources: agents.py:12-26, searcher/sementic_search.py:34-53, 212-218, LLM.py:104-108

Utility functions within the CoI-Agent framework transform raw, unstructured data from the web and LLMs into the structured formats required for autonomous research. By centralizing text extraction, mathematical comparisons, and network handling, the project maintains a clear separation between low-level data processing and high-level agent logic.