Environment Setup
Relevant source files
The following files were used as context for generating this wiki page:
The Environment Setup for the work-validity-eval project involves configuring a Python-based research pipeline designed to rethink AI agent evaluation. The environment supports data ingestion from O*NET and ESCO, task screening using LLMs, embedding generation, and the compilation of a NeurIPS position-track paper.
Proper setup ensures that reproduction scripts can access necessary OpenAI API models for operation normalization and hierarchy building, and that the LaTeX environment is prepared for document generation.
Sources: README.md:1-25, README.md:201-210
Python Environment and Dependencies
The project requires a Python virtual environment to manage dependencies specified in the requirements.txt file. These dependencies include data analysis libraries like pandas, machine learning tools such as scikit-learn, and the openai SDK for interacting with Large Language Models.
Installation Steps
- Virtual Environment Creation: Use the standard Python
venvmodule. - Activation: Activate the environment based on the operating system.
- Dependency Installation: Install required packages via
pip.
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
Sources: README.md:201-205, requirements.txt
API Configuration and External Services
The reproduction pipeline heavily relies on OpenAI services for embedding and classification tasks. Users must provide an OPENAI_API_KEY to the shell environment for the scripts to function correctly.
Environment Variables for Models
Several scripts allow overriding the default models via environment variables. If these variables are not set, the system defaults to specific versions of GPT or OpenAI embedding models.
| Variable | Description | Default Value |
|---|---|---|
OPENAI_API_KEY | Required for all LLM and Embedding calls. | N/A |
NORMALIZE_TASK_MODEL | Model used for domain-stripping O*NET tasks. | gpt-5.4-mini |
OPERATION_HIERARCHY_MODEL | Model used for describing top-level operations. | gpt-5.4-mini |
OPERATION_HIERARCHY_EMBED_MODEL | Model used for cluster theme embeddings. | text-embedding-3-small |
ROUND2_KSCREEN_MODEL | Model used for task-level knowledge screening. | gpt-5.4-mini |
Sources: scripts/normalize_task_operations.py:40, scripts/normalize_task_operations.py:192-194, scripts/build_operation_hierarchy.py:33-36, scripts/round2_knowledge_screen.py:42, README.md:212-214
Model Logic Flow
The following diagram illustrates how environment variables are checked during script initialization to configure the LLM clients.
The logic ensures that the script halts immediately if authentication is missing while providing flexibility for model selection.
Sources: scripts/normalize_task_operations.py:192-197, scripts/round2_knowledge_screen.py:169-173
Data Infrastructure
The environment expects a specific directory structure under data/ for raw inputs and derived artifacts. Raw data is retrieved using shell scripts from O*NET and ESCO sources.
Directory Structure Requirements
data/raw/onet/: O*NET 30.2 task statements and job zone references.data/raw/esco/: ESCO v1.1.1 competence catalogs.data/derived/: Output location for Parquet files and JSONL caches.
Sources: README.md:162-170, README.md:139-158, scripts/build_benchmark_inventory.py:23-25
Manuscript Build Environment
To reproduce the position-track paper (main.pdf), a LaTeX environment is required. The build process uses pdflatex and bibtex along with standard TeX Live packages.
Build Commands
pdflatex main
bibtex main
pdflatex main
pdflatex main
Required LaTeX Packages
graphicxbooktabstabularxamsfontsmicrotypehyperrefnatbib(numeric)
Sources: README.md:129-137
Reproduction Pipeline Sequence
Setting up the environment includes executing the reproduction pipeline in a specific order to ensure data dependencies are met.
This diagram shows the initial stages of the reproduction pipeline where the local environment interacts with external data sources and APIs to build the analysis artifacts.
Sources: README.md:172-187, scripts/round2_knowledge_screen.py:25-35
Summary
The environment setup for work-validity-eval establishes a controlled pipeline for evaluating AI agent validity through a combination of Python dependencies, OpenAI API integration, and LaTeX document generation. By configuring model environment variables and following the sequenced reproduction scripts, researchers can regenerate the 18-operation taxonomy and the paper's diagnostic audit of frontier agent suites.
Sources: README.md:201-214, README.md:172-198