Biomedical Dataset Management

Relevant source files

The following files were used as context for generating this wiki page:

• hw1/p1.py
• hw1/CHOLESTEROL.txt
• hw1/WEIGHT.txt
• hw2/p1.py
• hw2/p2.py

The Biomedical Dataset Management system provides a structured approach for ingesting, processing, and analyzing medical data points such as cholesterol levels and body weight. The system focuses on reading raw textual data, handling potential input errors, and performing statistical analysis to derive insights into data distribution and central tendency.

This module serves as the primary interface for handling datasets stored in flat-file formats. It utilizes standard Python libraries alongside numerical processing tools like numpy to compute complex metrics, including coefficients of variation and standard deviations.
Sources: hw1/p1.py:1-5, hw1/CHOLESTEROL.txt

Data Ingestion and Validation

The system employs a robust ingestion mechanism designed to transform raw text files into structured numerical arrays. The ingestion process utilizes a global list structure to accumulate data points before processing.

File Parsing Logic

The open_file function serves as the primary entry point for data ingestion. It implements a line-by-line reading strategy that includes:

• Trimming: Leading and trailing whitespace is removed to ensure clean numeric conversion.
• Validation: A try-except block captures ValueError exceptions, allowing the system to skip non-numeric or malformed data entries while logging the error to the console.
• Conversion: Validated strings are cast to floating-point numbers and appended to the internal data buffer.

Sources: hw1/p1.py:7-17

Data Flow Diagram

The following diagram illustrates the lifecycle of a data point from a flat file to a processed metric.

Statistical Analysis Framework

Once data is ingested, the system executes a suite of statistical functions to characterize the dataset. These computations utilize both the numpy library and the statistics module.

Core Metrics

The analysis pipeline calculates several standard and custom metrics as detailed below:

Metric	Implementation Method	Description
Mean	np.mean()	Average value of the dataset.
Median	np.median()	Middle value in the sorted dataset.
Mode	statistics.mode()	Most frequent value in the dataset.
Range	np.ptp()	Peak-to-peak (Maximum - Minimum).
Std Deviation	np.std()	Measure of data dispersion.
Variance	np.var()	Squared deviation from the mean.

Sources: hw1/p1.py:24-43

Advanced Statistical Computations

The system implements two specific types of "Coefficient of Variation" to analyze skewness and distribution characteristics:

1. First Coefficient of Variation: Calculated as $3 \times (\text{mean} - \text{median}) / \text{std}$.
2. Second Coefficient of Variation: Calculated using the third moment (sum of cubed deviations from the mean) divided by $(n \times \text{std}^3)$.

# 7. Calculate 1st Coefficient of Variation
cv1 = 3*(mean - median) / std

# 8. Calculate 2nd Coefficient of Variation
cv2 = sum([(x - mean) ** 3 for x in data]) / (len(data) * std ** 3)

Sources: hw1/p1.py:45-50

Execution Sequence

The module is designed for sequential execution where data loading must precede computation.

Process Interaction

The following sequence diagram outlines the interaction between the file system and the computation functions.

Summary

Biomedical Dataset Management facilitates the transition from raw clinical data (such as those found in CHOLESTEROL.txt and WEIGHT.txt) to actionable statistical insights. By combining robust error handling during file ingestion with advanced mathematical modeling, the project ensures data integrity and provides a comprehensive view of biometric variables.
Sources: hw1/CHOLESTEROL.txt, hw1/WEIGHT.txt, hw1/p1.py:53