Background Jobs & Scripts
Relevant source files
The following files were used as context for generating this wiki page:
Background jobs and scripts in the SecondLayer project encompass automated data synchronization, batch document processing, database migrations, and testing utilities. These components are essential for maintaining the platform's legal data accuracy, including parliament bills, court decisions, and state registry information. They operate outside the primary request-response cycle of the web application, often running as scheduled tasks or manual maintenance operations.
The system utilizes a variety of languages including TypeScript for complex logic and Shell scripts for orchestration and deployment. These scripts bridge the gap between external data sources (like Verkhovna Rada APIs) and the internal storage systems (PostgreSQL, Qdrant).
Data Synchronization Scripts
The project maintains specific scripts for synchronizing data from Ukrainian legal and state sources. These are primarily located within the specific MCP (Model Context Protocol) server directories.
Parliament (RADA) Sync
The mcp_rada server uses scripts to bulk-fetch data from the official Verkhovna Rada API. This includes information on deputies, bills, and legislation texts.
This diagram illustrates the flow of data from the external RADA API into the internal database and cache via specialized adapters and services.
- sync-deputies.ts: Bulk syncs all deputies from the RADA API and upserts them into the database with a 7-day TTL.
- sync-laws.ts: Fetches and caches common laws (e.g., Constitution, Codes) from zakon.rada.gov.ua.
- cleanup:cache: A maintenance script designed to remove expired cache entries from the system.
Sources: mcp_rada/IMPLEMENTATION_STATUS.md:188-212, CLAUDE.md:129-135
Registry Imports
The mcp_openreyestr server includes scripts for importing large datasets from the State Register, including legal entities and the debtors registry.
| Script | Purpose | Source Data |
|---|---|---|
import:entities | Imports legal entities | XML Data from data.gov.ua |
import:debtors | Imports state debtors registry | Open Data API |
Sources: CLAUDE.md:138-143
Batch Document Processing
The system supports high-throughput document processing for legal research and analysis. This is orchestrated through specific scripts that handle multiple files concurrently.
Workflow Orchestration
The batch-process-documents.ts script allows for the mass processing of documents through various operations such as parsing and summarization.
The flow shows how a directory of input documents is fed into the batch processor, which executes multiple legal analysis operations.
- Concurrency Control: The processor supports a
--concurrencyflag to manage the number of simultaneous processing tasks. - Operation Mapping: Supports operations like
parse,summarize, andextract_key_clauses.
Sources: scripts/testing/test-batch-processing.sh:58-96
Database & Infrastructure Maintenance
Automated scripts are used to manage the lifecycle of the database schema and the overall containerized environment.
Automated Migrations
Database migrations are handled by a dedicated migrate.ts runner in each service. In the local development environment, a migrate-local service runs once on startup to ensure the schema is current.
| Command | File Path | Description |
|---|---|---|
npm run db:setup | mcp_backend/scripts/db-setup.sh | Initializes DB and applies initial schema. |
npm run migrate | mcp_backend/src/migrations/migrate.ts | Runs pending SQL migrations. |
cleanup_stale_timers | SQL Trigger/Function | Automatically deletes timers without a ping for >24h. |
Sources: mcp_rada/IMPLEMENTATION_STATUS.md:218-223, deployment/LOCAL_DEVELOPMENT.md:18-22, IMPLEMENTATION_PHASES.md:391
Infrastructure Management
The manage-gateway.sh script serves as the primary entry point for managing the containerized environments (local, dev, stage, prod).
This sequence shows the deployment orchestration handled by the management script.
Sources: deployment/LOCAL_DEVELOPMENT.md:92-115, CLAUDE.md:204-213
Verification & Testing Scripts
A suite of shell scripts is used to verify the integrity of the system after updates or deployments.
- lexwebapp/verify-setup.sh: A comprehensive verification tool that checks for the existence of core files, validates environment variables (e.g.,
VITE_API_URL), and verifies that the package configuration is correct. - run-tests.sh: An orchestration script that triggers unit tests across the service, including SSE Client, MCP Service, and React hook tests.
- test-batch-processing.sh: Validates that the backend is healthy and the
batch_process_documentstool is properly registered before starting mass operations.
Sources: lexwebapp/verify-setup.sh:10-140, scripts/testing/test-batch-processing.sh:12-50
Conclusion
Background jobs and scripts are the connective tissue of the SecondLayer platform, ensuring that data from external legal sources is synchronized, processed, and maintained without user intervention. By utilizing specialized TypeScript scripts for data logic and Shell scripts for infrastructure orchestration, the system maintains high data integrity and deployment reliability across local and production environments.