Model Validation & Metrics
Relevant source files
The following files were used as context for generating this wiki page:
The Model Validation & Metrics system in the MBBank CAPTCHA OCR project provides the infrastructure for assessing the accuracy and performance of the trained OCR model. This system ensures that the model can reliably recognize alphanumeric characters from CAPTCHA images by partitioning datasets and calculating performance indicators like loss and exact-match accuracy.
Validation occurs in two primary stages: real-time loss monitoring during the training loop and a post-training evaluation phase on a dedicated test dataset. The results are used to visualize the convergence of the model and to provide a final quantitative measure of its predictive reliability.
Dataset Partitioning and Preparation
Validation begins with the logic defined in the DatasetLoader class, which handles the splitting of the source images into training and testing subsets.
Split Logic
The system automatically partitions the available images in the dataset/ directory. By default, the system reserves approximately 10% of the data for testing, while the remaining 90% is used for training. To prevent overfitting, the total number of images processed is capped at 62,500.
The diagram shows the logic for dividing the dataset into training and validation segments.
Sources: train_model/core.py:34-54
Data Structure for Validation
The DatasetLoader ensures that each item returned for validation contains the image tensor, a one-hot encoded label for loss calculation, and the original text string for accuracy verification.
| Component | Type | Purpose |
|---|---|---|
pil_img | Tensor | Transformed image data (50x160 pixels) |
label_oh | np.array | One-hot encoded ground truth for the multi-label loss function |
label | String | Plain text ground truth for final accuracy calculation |
Sources: train_model/core.py:64-71, train_model/train.py:23-28
Training Metrics: Multi-Label Loss
During the training phase, the system monitors the MultiLabelSoftMarginLoss. This metric measures how well the model's output (a reshaped tensor of dimensions [batch, max_capcha, chars_len]) aligns with the one-hot encoded ground truth.
Loss Tracking
The loss is calculated at every step and stored in a list for visualization.
- Loss Function:
torch.nn.MultiLabelSoftMarginLoss() - Optimizer:
torch.optim.Adamwith a learning rate of 0.001 - Visualization: After 100 epochs, the system generates
loss.png, a plot of loss values over iterations to visualize model convergence.
Sources: train_model/train.py:38-54, train_model/train.py:60-64
Post-Training Validation: Exact Match Accuracy
After the training epochs are complete, the model is switched to evaluation mode (model.eval()) to perform a final validation pass.
Validation Workflow
The validation process iterates through the test_dl (test DataLoader). Unlike training, which uses loss values, the validation phase focuses on "Exact Match" accuracy—the model must predict every character in the CAPTCHA correctly for the prediction to be marked as "correct".
This sequence illustrates the conversion of model output tensors back into text for accuracy comparison.
Sources: train_model/train.py:68-84
Metric Calculation
The accuracy is calculated as:
Accuracy = (Total Correct Predictions) / (Total Number of Test Images)
The final output displays the raw counts (e.g., Accuracy: 950 / 1000) alongside the percentage.
Sources: train_model/train.py:86
Prediction Inference Flow
The validation logic used during training is mirrored in the production OcrModel class within the mb_capcha_ocr package, which utilizes onnxruntime for inference.
| Step | Operation | Source File |
|---|---|---|
| 1 | Image Grayscale Conversion (L) | mb_capcha_ocr/predict.py:23 |
| 2 | Resize to 160x50 | mb_capcha_ocr/predict.py:24 |
| 3 | Normalization (pixel / 255.0) | mb_capcha_ocr/predict.py:25 |
| 4 | ONNX Session Execution | mb_capcha_ocr/predict.py:30 |
| 5 | Argmax Label Decoding | mb_capcha_ocr/predict.py:32-33 |
Summary
Model Validation & Metrics in this project rely on a strict 10% split of the input dataset. By combining real-time loss tracking (MultiLabelSoftMarginLoss) with a final exact-match accuracy test on unseen data, the system provides a comprehensive view of the model's ability to generalize character recognition across diverse CAPTCHA samples. The export to ONNX format serves as the final validation that the model is ready for deployment via the mb_capcha_ocr library.
Sources: train_model/train.py:57-58, setup.py:22