Tijori Export & Insights
Relevant source files
The following files were used as context for generating this wiki page:
Tijori Export & Insights is a data collection and processing module that extracts company overviews and narrative insights from Tijori Finance. It maps public company snapshots to a pinned National Stock Exchange (NSE) universe, providing high-level business descriptions and key monitoring points for the scanner.
This system operates as part of the weekly adjusted-history workflow within the EDL Pipeline. It collects narrative sections such as "The Read" and "What to Watch," verified against official NSE symbols to ensure data integrity across the platform.
Sources: tools/TIJORI.md:1-25, tools/export_tijori_overviews.py:1-15
System Architecture
The exporter uses an asynchronous HTTP client to discover company URLs via sitemaps and search APIs. It processes HTML snapshots to extract structured JSON data, maintaining a local cache with HTTP validators to minimize redundant network requests.
Data Flow Overview
The following diagram illustrates the lifecycle of a Tijori data export, from discovery to public dataset publication.
The flow shows how company data is fetched conditionally and verified against the NSE universe before being committed to the public dataset.
Sources: tools/TIJORI.md:10-45, tools/export_tijori_overviews.py:175-230
Core Components
The Fetcher Engine
The Fetcher class manages a shared asynchronous HTTP connection pool. It implements gradual growth and error backoff to handle rate limits (HTTP 429) and server errors.
- Initial State: Starts at two requests in flight.
- Scaling: Increases concurrency up to a ceiling (default 8) upon successful requests.
- Throttling: Reduces limits and pauses requests when encountering 429s or 5xx errors.
- Persistence: Persistent 429 errors trigger a hard stop to the collection run.
Sources: tools/export_tijori_overviews.py:112-167, tools/test_export_tijori_overviews.py:28-60
Extraction Logic
The parse_company function processes the retrieved HTML. It attempts to extract data from embedded JSON scripts first, falling back to CSS selector-based parsing of the visible HTML snapshot.
| Data Point | Extraction Source |
|---|---|
| Symbol | p.symbol text or company_details_data JSON |
| Narrative | .memory_the_read__list > li |
| Watch Points | .memory_what_to_watch__list > li |
| Key Events | script#memoryOverviewKeyEvents |
| Update Date | .memory_card__updated_date (parsed to ISO) |
Sources: tools/export_tijori_overviews.py:38-84
Data Artifacts and Storage
The system maintains several intermediate and final artifacts within the reference/tijori/ directory.
Intermediate Files (Git Ignored)
sitemap.xml: The latest discovered URL list from Tijori.universe.json: The pinned NSE stock list used for the current run.responses.jsonl: A line-delimited JSON journal of all successful and failed fetches.exceptions.json: Diagnostics for symbols that failed extraction or verification.
Committed Artifacts
tijori-overviews.json.gz: The primary compressed cache containing all reports and HTTP validators.summary.json: High-level metadata including counts for available, unmapped, and error states.
Sources: tools/TIJORI.md:25-38, reference/tijori/summary.json:1-15
Technical Specifications
CLI Configuration
The export_tijori_overviews.py script accepts several parameters to control the export behavior.
| Argument | Type | Default | Description |
|---|---|---|---|
--refresh | Flag | N/A | Rechecks cached reports using ETags/Last-Modified headers. |
--concurrency | Int | 8 | Maximum simultaneous async HTTP connections. |
--rate | Float | 8 | Maximum requests per second. |
--retry-unmapped | Flag | N/A | Retries symbols that previously failed URL discovery. |
--finalize-only | Flag | N/A | Rebuilds public artifacts from the local journal without network calls. |
Sources: tools/export_tijori_overviews.py:173-186, tools/TIJORI.md:5-20
Public Dataset Schema
The exporter publishes to frontend/public/data/tijori/. The current.json file serves as the manifest for consumers.
{
"schemaVersion": 1,
"revision": "sha256-hash",
"updatedAt": "ISO-TIMESTAMP",
"reportCount": 2425,
"universeSessionDate": "YYYY-MM-DD",
"encoding": "gzip",
"datasetUrl": "/data/tijori/revisions/<hash>/overviews.json.gz"
}
Sources: tools/export_tijori_overviews.py:92-110, tools/TIJORI.md:54-65
Error Handling and Integrity
The system prioritizes existing data over new failures. If a refresh fails for a specific company (e.g., HTTP 403 or extraction error), the system retains the previously saved report and records the error in the summary diagnostics.
- Symbol Verification: The visible symbol in the page data MUST match the NSE symbol before saving.
- Conditional Requests: Uses
If-None-Match(ETag) andIf-Modified-Since(Last-Modified) to avoid re-downloading unchanged content. - Atomic Writes: All JSON and Gzip artifacts are written to
.tmpfiles and moved into place to prevent corruption during crashes.
Sources: tools/TIJORI.md:67-85, tools/export_tijori_overviews.py:86-90, tools/test_export_tijori_overviews.py:90-110
Conclusion
Tijori Export & Insights provides a robust narrative enrichment layer to the Nexus Scanner. By combining resilient asynchronous fetching with strict NSE symbol validation, it ensures that company insights remain accurate and up-to-date while minimizing bandwidth usage through intelligent caching and conditional HTTP requests.