Integrating Custom Datasets

Relevant source files

The following files were used as context for generating this wiki page:

• hw1/CHOLESTEROL.txt
• hw1/WEIGHT.txt
• hw1/p1.py
• hw2/heart_failure_clinical_records_dataset.csv
• hw2/p1.py

Introduction

The integration of custom datasets within this project involves a streamlined workflow for ingesting raw biometric and clinical data from external text and CSV files. This process focuses on transforming unstructured or semi-structured data into a format suitable for statistical analysis and machine learning.

The system is designed to handle various data formats, including newline-delimited numerical values used for physical metrics and structured CSV files for complex clinical records. Once integrated, these datasets serve as the primary input for descriptive statistics and predictive modeling.

Sources: hw1/p1.py:1-10, hw2/p1.py:1-5

Data Ingestion Architecture

The project utilizes a modular approach to data ingestion, separating the file reading logic from the analysis logic. This allows for the integration of diverse data sources by simply pointing the ingestion functions to the relevant file paths.

File Loading and Parsing

For simple datasets, the system uses a custom open_file function. This function reads a file line-by-line, strips whitespace, and converts valid numeric strings into floats while providing error handling for non-numeric content.

def open_file(file_name):
    with open(file_name, 'r') as file:
        for line in file:
            line = line.strip()
            if line:
                try:
                    data.append(float(line))
                except ValueError:
                    print(f"Skipping non-numeric value: {line}")

Sources: hw1/p1.py:6-15

Structured Data Handling

For more complex datasets, such as the clinical records stored in CSV format, the system leverages standard data manipulation libraries (implied by the file structure in hw2). These files contain multiple columns of patient data, including age, high blood pressure status, and serum creatinine levels.

Sources: hw2/heart_failure_clinical_records_dataset.csv

Data Processing Pipeline

The following diagram illustrates the flow of data from raw external files through the processing and analysis stages.

The data processing pipeline ensures that raw inputs are cleaned and converted into numerical types before any mathematical operations are performed.
Sources: hw1/p1.py:17-19, hw2/p1.py

Supported Dataset Formats

The system currently supports two primary data formats for integration:

Format	Extension	Typical Content	Loading Method
Text	.txt	Single-column numerical metrics (e.g., Weight, Cholesterol)	open_file()
CSV	.csv	Multi-column clinical records and patient data	Data Library Loaders

Sources: hw1/CHOLESTEROL.txt, hw1/WEIGHT.txt, hw2/heart_failure_clinical_records_dataset.csv

Integrated Statistical Metrics

Once a custom dataset is integrated, the system automatically computes a suite of statistical indicators to characterize the data.

Descriptive Statistics

The system calculates central tendency and dispersion metrics using numpy and statistics libraries.

• Mean and Median: Calculated via np.mean and np.median.
• Mode: Determined using statistics.mode.
• Dispersion: Includes Range (np.ptp), Standard Deviation (np.std), and Variance (np.var).

Sources: hw1/p1.py:23-42

Derived Coefficients

The project implements specific formulas for the Coefficient of Variation to assess data distribution:

1. First Coefficient of Variation: Based on the difference between mean and median relative to standard deviation.
   • Formula: 3 * (mean - median) / std
2. Second Coefficient of Variation: A skewness-based calculation.
   • Formula: sum([(x - mean)**3 for x in data]) / (len(data) * std**3)

Sources: hw1/p1.py:44-51

Conclusion

Integrating custom datasets into the project provides a foundational capability for biometric and clinical analysis. By supporting both simple text-based metrics and complex clinical records, the architecture allows for flexible data exploration and serves as the precursor for advanced statistical modeling.

Sources: hw1/p1.py, hw2/p1.py