Data Ingestion & Caching

Relevant source files

The following files were used as context for generating this wiki page:

• src/data/real_data_fetcher.py
• src/data/sample_data.py
• src/config/settings.py
• ARCHITECTURE.md
• AGENTS.md
• IMPROVEMENTS.md

Data Ingestion and Caching manages the retrieval of financial data from multiple sources and provides local storage mechanisms to optimize performance. The system supports both static sample data for demonstration purposes and live market data via external APIs.

The core objective is to populate the AssetRelationshipGraph with structured financial assets (Equities, Bonds, Commodities, Currencies) and their metadata. Caching ensures that expensive API calls to providers like Yahoo Finance are minimized by persisting graph data to local JSON or SQLite storage.

Sources: ARCHITECTURE.md, AGENTS.md, IMPROVEMENTS.md

Data Ingestion Sources

The project utilizes two primary ingestion paths to populate the financial asset database.

Real Data Fetcher

The RealDataFetcher class uses the yfinance library to retrieve live market data, company information, and financial metrics. It transforms raw API responses into structured domain objects such as Equity or Bond.

Sample Data

The sample_data.py module contains create_sample_database(), which constructs a predefined in-memory dataset. This is used for demos, testing, and internal development where external API access is not required.

Sources: src/data/real_data_fetcher.py, src/data/sample_data.py, AGENTS.md

The diagram shows the logic path for selecting between live market data and static sample data.
Sources: src/config/settings.py, AGENTS.md

Configuration & Environment Settings

Data ingestion behavior is controlled by centralized settings. The system uses pydantic-settings to manage environment variables that determine the data source and cache locations.

Option	Type	Default	Description
USE_REAL_DATA_FETCHER	bool	False	Enables live data fetching via Yahoo Finance.
GRAPH_CACHE_PATH	str	None	File path for the serialized graph cache.
REAL_DATA_CACHE_PATH	str	None	File path for raw data cache from APIs.
DATABASE_URL	str	"sqlite:dev.db"	Path to the SQLite database for persistent storage.

Sources: src/config/settings.py, AGENTS.md

Caching Strategy

The system implements a tiered caching strategy to improve response times and handle rate limiting from external providers.

In-Memory Cache

The AssetRelationshipGraph maintains assets and relationships in-memory during application runtime. This allows for rapid relationship discovery and metric calculation without repeated disk or network I/O.
Sources: ARCHITECTURE.md, AGENTS.md

Persistent Storage

The application supports persisting the entire graph state to a local file. This is configured via the GRAPH_CACHE_PATH. Additionally, for enterprise-readiness, the system preserves SQLite compatibility for local development and disaster recovery.
Sources: AGENTS.md, GEMINI.md

Data Flow for Caching

The sequence diagram illustrates the prioritization of local cache over network requests.
Sources: ARCHITECTURE.md, AGENTS.md

Data Transformation Logic

When ingesting real data, the RealDataFetcher performs specific mapping from API fields to the internal data model.

1. Ingestion: RealDataFetcher queries yfinance for a specific ticker.
2. Mapping:
   • Stock prices and market caps are mapped to the price and market_cap fields.
   • Sector info is mapped to the sector field for relationship discovery.
   • Dividend data is extracted for income comparison metrics.
3. Validation: Dataclass __post_init__ methods validate inputs, such as ensuring non-negative prices and 3-letter ISO currency codes.

Sources: src/data/real_data_fetcher.py, AUDIT_REPORT.md, AGENTS.md

Conclusion

Data Ingestion & Caching provides the foundational data layer for the Relationship Database. By supporting both sample and real data sources, the system facilitates flexible development while the caching mechanisms ensure performance and stability. The transition toward durable graph persistence using SQLite marks a shift toward production-ready disaster recovery and state management.

Sources: ARCHITECTURE.md, GEMINI.md