Data Cleaning & Validation Flow

Relevant source files

The following files were used as context for generating this wiki page:

• hw1/p1.py
• hw1/CHOLESTEROL.txt
• hw1/WEIGHT.txt
• hw2/p1.py
• hw2/p2.py

The Data Cleaning & Validation Flow in this project is designed to ingest raw numerical data from external text files, filter out non-numeric noise, and prepare a structured dataset for statistical analysis. This process ensures that mathematical computations—such as calculating the mean, variance, and coefficients of variation—are performed on a sanitized list of floating-point numbers, preventing runtime errors during array processing.

The system utilizes standard Python file I/O operations combined with numpy and statistics libraries to transform unstructured text content into validated numerical arrays.

Sources: hw1/p1.py:1-12, hw1/CHOLESTEROL.txt

Data Ingestion and Sanitization

The primary entry point for data handling is the open_file function. This module implements a robust reading mechanism that iterates through line-delimited text files.

File Parsing Logic

1. Line Stripping: Each line is stripped of leading and trailing whitespace, which is critical for handling files with trailing tabs or empty lines.
2. Type Validation: The system attempts to cast each string to a float.
3. Error Handling: A try-except block captures ValueError exceptions. If a line contains non-numeric data, the system logs the occurrence and skips the entry rather than terminating the execution.

def open_file(file_name):
    with open(file_name, 'r') as file:
        for line in file:
            line = line.strip()
            if line:
                try:
                    data.append(float(line))
                except ValueError:
                    print(f"Skipping non-numeric value: {line}")

Sources: hw1/p1.py:7-16, hw1/CHOLESTEROL.txt:29

Validation Flow Diagram

The following diagram illustrates the logic used to clean and validate input data from raw text files.

This flow ensures that empty lines and invalid characters do not corrupt the downstream statistical calculations.
Sources: hw1/p1.py:7-16

Processing and Statistical Validation

Once data is cleaned and stored in the data list, the compute_file function orchestrates the calculation of several descriptive statistics. This stage serves as a secondary validation of the dataset's distribution.

Statistical Components

The following table summarizes the metrics calculated during the post-validation phase:

Metric	Library/Method	Formula/Logic
Mean	np.mean(data)	Arithmetic average of validated entries
Median	np.median(data)	Middle value of sorted entries
Range	np.ptp(data)	Peak-to-peak (Max - Min)
Variance	np.var(data)	Average of squared deviations from mean
CV (1st)	Manual	$3 \times (mean - median) / std$
CV (2nd)	List Comp	$\sum(x - mean)^3 / (N \times std^3)$

Sources: hw1/p1.py:22-52

Transformation Sequence

The sequence below shows how data moves from the raw storage files through the validation function into the computation engine.

Data Structures

The project relies on a global data list to accumulate processed values. While simple, it requires manual clearing if compute_file is called multiple times on different datasets in the same session, as open_file uses data.append() without resetting the list.

• Input Format: Newline-delimited text (e.g., CHOLESTEROL.txt contains values like 205, 166, 236).
• Internal Storage: Python list of float objects.
• Output: Standard output stream (print statements).

Sources: hw1/p1.py:5-10, hw1/CHOLESTEROL.txt:1-5, hw1/WEIGHT.txt:1-5

Conclusion

The Data Cleaning & Validation Flow provides a reliable method for converting messy external text inputs into high-precision numerical data. By implementing per-line validation and exception handling, the system maintains stability even when encountering non-numeric artifacts or empty lines in the source files.