Author: Wasel Al-Mahri (waseeld)
Repository: https://github.com/waseeld
- Overview
- System Architecture
- Repository Structure
- Supported Document Formats
- Quick Start & CLI Usage
- JSON Output Structure
- Hardware-Accelerated Training (6GB VRAM + 32GB RAM)
- Privacy & Sensitive Data Isolation
- Benchmark Summary
- Roadmap
MRZ-AI is an end-to-end Machine Learning and deterministic validation pipeline for detecting, extracting, reading, and verifying Machine Readable Zones (MRZ) across Passports, Saudi Umrah eVisas (PDFs & Images), Standard Visas, and National IDs / Iqamas.
It combines three core stages into a clean, modular architecture:
- ML#1 β Lightweight YOLOv11n MRZ Detector (
~10.1 MBONNX):- Trained on a unified dataset of scene-first synthetic documents, real passport scans, and multi-page Umrah eVisa PDFs.
- Runs natively on CPU via ONNX Runtime (
CPUExecutionProvider) with an automatic morphological MRZ band fallback (blackhat + Scharr gradient) for full-page PDFs.
- ML#2 β Multi-Variant Crop Preprocessing + Geometry-Aware OCR:
- Automatic Hough / minAreaRect deskewing,
-7Β°and-10Β°tilt rotation recovery (handling slanted mobile captures), CLAHE contrast enhancement, unsharp masking, and Otsu thresholding. - Geometry-aware OCR line assembly that strips non-MRZ distractor words, cleans
<filler noise, and stitches slanted line fragments.
- Automatic Hough / minAreaRect deskewing,
- MRZ Core β Deterministic ICAO Doc 9303 & KSA Umrah eVisa Parser + Checksum Recovery:
- Strict modulo-10 (
7-3-1weights) check-digit verification. - Position-aware and checksum-constrained minimum-edit OCR ambiguity recovery (
0βO/D/Q,1βI/L,2βZ,5βS,6βG,8βB).
- Strict modulo-10 (
Input (--source: Single Image | Single PDF | Entire Folder of Images/PDFs)
β
[ Document Loader (OpenCV + PyPDFium2 Page Renderer) ]
β
[ ML#1: YOLOv11n ONNX Detector (CPU) + Morphological Band Fallback ]
β
[ Multi-Variant Crop Preprocessor (Deskew, -7Β°/-10Β° Tilt Sweep, CLAHE, Sharpen, Otsu) ]
β
[ ML#2: OCR Reader + Geometry-Aware Line Assembler ]
β
[ MRZ Core: Format Detection + ICAO/eVisa Checksum Validation + Ambiguity Recovery ]
β
Structured JSON Report (scan_results.json + Cropped MRZ Strips)
MRZ/
βββ ml1_detector/
β βββ config/
β β βββ config.yaml # Hardware-accelerated training & model configuration
β βββ data/
β β βββ passport_gt_boxes.json # Verified reference bounding boxes
β β βββ prepare_unified_dataset.py # Multi-document unified dataset builder
β β βββ synthetic_generator.py # Scene-first synthetic generator (TD1/TD2/TD3 + negatives)
β βββ datasets/
β β βββ data.yaml # Portable YOLO dataset configuration
β βββ fonts/
β β βββ OCRB.ttf # Official OCR-B MRZ font
β βββ imaging/
β β βββ line_assembler.py # Geometry-aware OCR line assembly & filler repair
β β βββ mrz_band_locator.py # Morphological blackhat + Scharr MRZ band locator
β β βββ pdf.py # Unified image/PDF/folder loader (pypdfium2)
β β βββ preprocess.py # Deskewing, rotation sweep, CLAHE, sharpen, Otsu
β βββ inference/
β β βββ onnx_export.py # YOLOv11n -> ONNX exporter
β β βββ predict.py # Standalone detector inference CLI
β βββ mrz_core/
β β βββ charset.py # ICAO alphabet & character values
β β βββ checksum.py # Modulo-10 (7-3-1) check-digit calculator
β β βββ corrector.py # Minimum-edit checksum-guided OCR recovery
β β βββ dates.py # YYMMDD & DDMMYYYY date parser
β β βββ formats.py # TD3, TD3_DDMMYYYY, TD2, TD1, MRVA, MRVB, KSA eVisa
β β βββ normalizer.py # Line & filler normalizer
β β βββ parser.py # Deterministic multi-format MRZ parser
β β βββ recovery.py # End-to-end parse + recovery orchestrator
β βββ ocr/
β β βββ baseline.py # Standalone crop-to-text OCR evaluator
β βββ pipeline/
β β βββ scanner.py # Unified end-to-end scanner (Image/PDF/Folder -> JSON)
β βββ training/
β βββ train.py # 6GB VRAM (AMP FP16) + 32GB RAM cached YOLO trainer
βββ tests/
β βββ test_mrz_core.py # Unit tests for all MRZ formats & checksum recovery
βββ AGENTS.md # AI-agent operating model & responsibilities
βββ PROGRESS.md # Milestone & verification history
βββ PROJECT_PLAN.md # Architecture roadmap & phase status
βββ requirements.txt # Python dependencies
βββ setup.py # Package installer
βββ test_setup.py # Environment & GPU hardware verification script
| Format Code | Dimensions | Document Types |
|---|---|---|
TD3 |
2 lines Γ 44 chars | International Passports (ICAO Doc 9303 standard YYMMDD dates) |
TD3_DDMMYYYY |
2 lines Γ 44 chars | Non-standard / Specimen Passports using 8-digit DDMMYYYY dates |
KSA_EVISA_CUSTOM_44X2 |
2 lines Γ 44 chars | Saudi Umrah / Visit eVisas (starting with 1< or I< and 8-digit DDMMYYYY DOB) |
TD1 |
3 lines Γ 30 chars | National ID Cards & Residence Permits (Iqama) |
TD2 |
2 lines Γ 36 chars | Official Travel Documents & ID Cards |
MRV-A |
2 lines Γ 44 chars | Machine Readable Visas (Type A) |
MRV-B |
2 lines Γ 36 chars | Machine Readable Visas (Type B) |
All commands run directly from the repository root:
# 1. Install dependencies
pip install -r requirements.txt
# 2. Verify environment & hardware (CUDA VRAM, ONNX Runtime, PyPDFium2, EasyOCR, MRZ Core)
python test_setup.py
# 3. Run unit tests for MRZ Core (TD3, TD3_DDMMYYYY, KSA Umrah eVisa, TD1 ID/Iqama)
python -m pytest tests/test_mrz_core.py -v# Scan a folder of passport images on CPU
python -m ml1_detector.pipeline.scanner \
--source real_data/passports \
--detector-device cpu \
--ocr-device cpu \
--project pipeline_runs \
--name passports_scan
# Scan a multi-page Umrah eVisa PDF on CPU
python -m ml1_detector.pipeline.scanner \
--source real_data/visas_pdfs \
--detector-device cpu \
--ocr-device cpu \
--project pipeline_runs \
--name umrah_pdf_scan
# Scan an entire directory containing both images and PDFs recursively (fast mode)
python -m ml1_detector.pipeline.scanner \
--source real_data \
--detector-device cpu \
--ocr-device gpu \
--fast \
--project pipeline_runs \
--name full_scanEach scan writes scan_results.json containing a run-level summary and per-page document records:
{
"summary": {
"total_pages_processed": 15,
"yolo_detection_count": 15,
"yolo_detection_rate": 1.0,
"shape_valid_count": 15,
"shape_valid_rate": 1.0,
"parsed_count": 15,
"parsed_rate": 1.0,
"checksum_verified_count": 15,
"checksum_verified_rate": 1.0
},
"documents": [
{
"document_id": "sample_passport_01",
"source_type": "image",
"page_number": 1,
"status": "VERIFIED",
"detector": {
"yolo_detected": true,
"selected_box_source": "yolo11n_detector",
"bbox_xyxy": [21, 513, 938, 634],
"confidence": 0.9036
},
"mrz": {
"format": "TD3",
"document_type": "passport",
"issuing_country": "UTO",
"nationality": "UTO",
"document_number": "L898902C3",
"name": {
"surname": "ERIKSSON",
"given_names": ["ANNA", "MARIA"],
"full_name": "ANNA MARIA ERIKSSON"
},
"date_of_birth": {
"raw": "740812",
"iso": "1974-08-12",
"ambiguous": false
},
"sex": "F",
"expiration_date": {
"raw": "120415",
"iso": "2012-04-15",
"ambiguous": false
},
"checksums_valid": true,
"corrections": []
}
}
]
}The training pipeline (ml1_detector.training.train) is specifically tuned for machines with 6GB GPU VRAM (e.g., NVIDIA RTX 4050 Laptop GPU) and 32GB System RAM:
amp: true(FP16 Mixed Precision) & TF32 Matmul: Cuts GPU memory usage nearly in half while accelerating tensor operations.cache: "ram": Preloads training and validation images into the 32GB System RAM so disk I/O bottlenecks are eliminated.- Safe VRAM Batch Auto-Capping: Automatically caps batch size at
8for640Γ640and2for1280Γ1280on 6GB GPUs to prevent CUDA Out-Of-Memory errors.
# Build the unified multi-document dataset (Passports + Umrah PDFs + Synthetic IDs/Iqamas)
python -m ml1_detector.data.prepare_unified_dataset \
--max-pages-per-pdf 15 \
--num-synthetic-train 120 \
--num-synthetic-val 30
# Fine-tune YOLOv11n with GPU FP16 AMP and RAM caching
python -m ml1_detector.training.train \
--data ml1_detector/datasets/unified_mrz_boost.yaml \
--weights gpu_runss/synthetic_v2_first_ft1_ocr_safe_v2/weights/best.pt \
--device gpu \
--epochs 5 \
--batch-size 8 \
--image-size 640 \
--cache ram \
--amp \
--project gpu_runss \
--name unified_mrz_v1 \
--exist-ok
# Export the fine-tuned model to ONNX for lightweight CPU deployment
python -m ml1_detector.inference.onnx_export \
--weights gpu_runss/unified_mrz_v1/weights/best.pt \
--device gpu \
--image-size 640 \
--output exports/unified_mrz_yolo11n_640.onnxAll sensitive documents and local runtime outputs are strictly isolated and excluded from Git via .gitignore:
real_data/(real passport images and Umrah visa PDFs)progress/(local run state and machine-specific paths)pipeline_runs/,gpu_runss/,exports/(cropped outputs, scan results, and weights)*.pdf,*.pt,*.onnx
| Dataset / Source | Pages | YOLOv11n ONNX (CPU) Detections | Shape-Valid MRZ | Parsed Documents | Checksum Verified |
|---|---|---|---|---|---|
Real Passports Set (real_data/passports) |
15 | 15/15 (100.0%) | 15/15 (100.0%) | 15/15 (100.0%) | 15/15 (100.0%) |
| Multi-Page Passport PDF Batch | 6 | 6/6 (100.0%) | 6/6 (100.0%) | 6/6 (100.0%) | 6/6 (100.0%) |
| Saudi Umrah eVisas PDF Batch A | 10 | 10/10 (100.0%) | 10/10 (100.0%) | 10/10 (100.0%) | 10/10 (100.0%) |
| Saudi Umrah eVisas PDF Batch B | 5 | 5/5 (100.0%) | 5/5 (100.0%) | 5/5 (100.0%) | 5/5 (100.0%) |
| Total Verified Benchmark | 36 | 36/36 (100.0%) | 36/36 (100.0%) | 36/36 (100.0%) | 36/36 (100.0%) |
- β
Phase 1 β Stabilize ML#1 Training & Synthetic Scene Generator (
v2) - β Phase 2 β OCR-Safe Real-Scene Adaptation & Lightweight ONNX CPU Export
- β Phase 3 β Multi-Variant Preprocessing, Tilt Recovery & Geometry-Aware OCR Line Assembly
- β Phase 4 β Unified End-to-End Pipeline (Images, Multi-Page PDFs, Passports, Umrah eVisas, TD1/TD2 IDs) -> JSON
- π Phase 5 β Dedicated Custom CRNN (
CNN + BiLSTM + CTC) OCR Model Trained & Exported to ONNX