Data Import & Export Pipelines

The Data Import & Export Pipelines manage the movement of evaluation data between external storage formats and the platform's internal database. This system supports bulk operations for datasets, evaluation results, and configuration metadata to ensure interoperability with external LLM evaluation tools.

The pipeline automates data transformation and validation during the ingestion process. It ensures that imported content adheres to the platform's schema while providing flexible export options for reporting and external analysis.

Sources: apps/server/src/modules/data-pipeline/data-pipeline.service.ts, README.md

Pipeline Architecture

The platform uses a service-oriented architecture to handle data streams. The DataPipelineService coordinates between the storage layer and the data processing logic.

The diagram shows the bidirectional flow of information through the evaluation platform's pipeline.

Sources: apps/server/src/modules/data-pipeline/data-pipeline.controller.ts:1-30, apps/server/src/modules/data-pipeline/data-pipeline.service.ts:10-50

Import Process

The import pipeline processes uploaded files and converts them into internal dataset entities. You must provide a valid datasetId and a supported file type (JSON or CSV) to initiate an import.

Import Workflow

1. The DataPipelineController receives a POST request with a file buffer.
2. DataPipelineService validates the file format and schema.
3. The service maps columns or keys to the internal DatasetItem model.
4. The system performs a bulk insert into the database.

Sources: apps/server/src/modules/data-pipeline/dto/import-data.dto.ts:5-15, apps/server/src/modules/data-pipeline/data-pipeline.service.ts:65-80

Export Process

The export pipeline extracts evaluation results and dataset content into portable formats.

The sequence diagram illustrates how the system fetches data from the database and streams it to the user.

Supported Export Formats

The platform supports several output formats defined in the ExportDataDto:

• JSON: Full fidelity export including metadata and nested evaluation metrics.
• CSV: Flattened structure suitable for spreadsheet analysis.
• Excel: Multi-sheet export separating prompts, responses, and scores.

Sources: apps/server/src/modules/data-pipeline/dto/export-data.dto.ts:2-20, apps/server/src/modules/data-pipeline/data-pipeline.service.ts:100-125

Data Validation and Mapping

Before persistence, the pipeline validates every record against the DatasetService constraints. If a record fails validation, the system logs the error and proceeds based on the configured error policy.

// Example of data mapping logic in the service
async mapToInternal(row: any): Promise<DatasetItemDto> {
  return {
    input: row.input || row.prompt,
    expectedOutput: row.expected || row.target,
    metadata: row.metadata || {},
  };
}

Sources: apps/server/src/modules/data-pipeline/data-pipeline.service.ts:140-155, apps/server/src/modules/dataset/dataset.service.ts:40-55

Summary

The Data Import & Export Pipelines provide the critical link between the evaluation platform and external data sources. By supporting flexible mapping and multiple file formats, the system enables users to easily migrate evaluation benchmarks and extract insights for external reporting. The pipeline ensures data integrity through rigorous validation steps before any information enters the primary database.

Sources: README.md, apps/server/src/modules/data-pipeline/data-pipeline.service.ts