Skip to content
waseeldPublic

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

1 watching

Forks

Latest commit

Β 

History

7 Commits

Folders and files

Repository files navigation

πŸ“˜ MRZ-AI

Unified Lightweight CPU-First MRZ Detection, OCR, and ICAO/Umrah eVisa Parsing Pipeline

Author: Wasel Al-Mahri (waseeld)
Repository: https://github.com/waseeld


πŸ“‘ Table of Contents


🧠 Overview

MRZ-AI is an end-to-end Machine Learning and deterministic validation pipeline for detecting, extracting, reading, and verifying Machine Readable Zones (MRZ) across Passports, Saudi Umrah eVisas (PDFs & Images), Standard Visas, and National IDs / Iqamas.

It combines three core stages into a clean, modular architecture:

  1. ML#1 β€” Lightweight YOLOv11n MRZ Detector (~10.1 MB ONNX):
    • Trained on a unified dataset of scene-first synthetic documents, real passport scans, and multi-page Umrah eVisa PDFs.
    • Runs natively on CPU via ONNX Runtime (CPUExecutionProvider) with an automatic morphological MRZ band fallback (blackhat + Scharr gradient) for full-page PDFs.
  2. ML#2 β€” Multi-Variant Crop Preprocessing + Geometry-Aware OCR:
    • Automatic Hough / minAreaRect deskewing, -7Β° and -10Β° tilt rotation recovery (handling slanted mobile captures), CLAHE contrast enhancement, unsharp masking, and Otsu thresholding.
    • Geometry-aware OCR line assembly that strips non-MRZ distractor words, cleans < filler noise, and stitches slanted line fragments.
  3. MRZ Core β€” Deterministic ICAO Doc 9303 & KSA Umrah eVisa Parser + Checksum Recovery:
    • Strict modulo-10 (7-3-1 weights) check-digit verification.
    • Position-aware and checksum-constrained minimum-edit OCR ambiguity recovery (0↔O/D/Q, 1↔I/L, 2↔Z, 5↔S, 6↔G, 8↔B).

🧩 System Architecture

Input (--source: Single Image | Single PDF | Entire Folder of Images/PDFs)
        ↓
[ Document Loader (OpenCV + PyPDFium2 Page Renderer) ]
        ↓
[ ML#1: YOLOv11n ONNX Detector (CPU) + Morphological Band Fallback ]
        ↓
[ Multi-Variant Crop Preprocessor (Deskew, -7Β°/-10Β° Tilt Sweep, CLAHE, Sharpen, Otsu) ]
        ↓
[ ML#2: OCR Reader + Geometry-Aware Line Assembler ]
        ↓
[ MRZ Core: Format Detection + ICAO/eVisa Checksum Validation + Ambiguity Recovery ]
        ↓
Structured JSON Report (scan_results.json + Cropped MRZ Strips)

πŸ“‚ Repository Structure

MRZ/
β”œβ”€β”€ ml1_detector/
β”‚   β”œβ”€β”€ config/
β”‚   β”‚   └── config.yaml                  # Hardware-accelerated training & model configuration
β”‚   β”œβ”€β”€ data/
β”‚   β”‚   β”œβ”€β”€ passport_gt_boxes.json       # Verified reference bounding boxes
β”‚   β”‚   β”œβ”€β”€ prepare_unified_dataset.py   # Multi-document unified dataset builder
β”‚   β”‚   └── synthetic_generator.py       # Scene-first synthetic generator (TD1/TD2/TD3 + negatives)
β”‚   β”œβ”€β”€ datasets/
β”‚   β”‚   └── data.yaml                    # Portable YOLO dataset configuration
β”‚   β”œβ”€β”€ fonts/
β”‚   β”‚   └── OCRB.ttf                     # Official OCR-B MRZ font
β”‚   β”œβ”€β”€ imaging/
β”‚   β”‚   β”œβ”€β”€ line_assembler.py            # Geometry-aware OCR line assembly & filler repair
β”‚   β”‚   β”œβ”€β”€ mrz_band_locator.py          # Morphological blackhat + Scharr MRZ band locator
β”‚   β”‚   β”œβ”€β”€ pdf.py                       # Unified image/PDF/folder loader (pypdfium2)
β”‚   β”‚   └── preprocess.py                # Deskewing, rotation sweep, CLAHE, sharpen, Otsu
β”‚   β”œβ”€β”€ inference/
β”‚   β”‚   β”œβ”€β”€ onnx_export.py               # YOLOv11n -> ONNX exporter
β”‚   β”‚   └── predict.py                   # Standalone detector inference CLI
β”‚   β”œβ”€β”€ mrz_core/
β”‚   β”‚   β”œβ”€β”€ charset.py                   # ICAO alphabet & character values
β”‚   β”‚   β”œβ”€β”€ checksum.py                  # Modulo-10 (7-3-1) check-digit calculator
β”‚   β”‚   β”œβ”€β”€ corrector.py                 # Minimum-edit checksum-guided OCR recovery
β”‚   β”‚   β”œβ”€β”€ dates.py                     # YYMMDD & DDMMYYYY date parser
β”‚   β”‚   β”œβ”€β”€ formats.py                   # TD3, TD3_DDMMYYYY, TD2, TD1, MRVA, MRVB, KSA eVisa
β”‚   β”‚   β”œβ”€β”€ normalizer.py                # Line & filler normalizer
β”‚   β”‚   β”œβ”€β”€ parser.py                    # Deterministic multi-format MRZ parser
β”‚   β”‚   └── recovery.py                  # End-to-end parse + recovery orchestrator
β”‚   β”œβ”€β”€ ocr/
β”‚   β”‚   └── baseline.py                  # Standalone crop-to-text OCR evaluator
β”‚   β”œβ”€β”€ pipeline/
β”‚   β”‚   └── scanner.py                   # Unified end-to-end scanner (Image/PDF/Folder -> JSON)
β”‚   └── training/
β”‚       └── train.py                     # 6GB VRAM (AMP FP16) + 32GB RAM cached YOLO trainer
β”œβ”€β”€ tests/
β”‚   └── test_mrz_core.py                 # Unit tests for all MRZ formats & checksum recovery
β”œβ”€β”€ AGENTS.md                            # AI-agent operating model & responsibilities
β”œβ”€β”€ PROGRESS.md                          # Milestone & verification history
β”œβ”€β”€ PROJECT_PLAN.md                      # Architecture roadmap & phase status
β”œβ”€β”€ requirements.txt                     # Python dependencies
β”œβ”€β”€ setup.py                             # Package installer
└── test_setup.py                        # Environment & GPU hardware verification script

πŸͺͺ Supported Document Formats

Format Code Dimensions Document Types
TD3 2 lines Γ— 44 chars International Passports (ICAO Doc 9303 standard YYMMDD dates)
TD3_DDMMYYYY 2 lines Γ— 44 chars Non-standard / Specimen Passports using 8-digit DDMMYYYY dates
KSA_EVISA_CUSTOM_44X2 2 lines Γ— 44 chars Saudi Umrah / Visit eVisas (starting with 1< or I< and 8-digit DDMMYYYY DOB)
TD1 3 lines Γ— 30 chars National ID Cards & Residence Permits (Iqama)
TD2 2 lines Γ— 36 chars Official Travel Documents & ID Cards
MRV-A 2 lines Γ— 44 chars Machine Readable Visas (Type A)
MRV-B 2 lines Γ— 36 chars Machine Readable Visas (Type B)

πŸš€ Quick Start & CLI Usage

All commands run directly from the repository root:

# 1. Install dependencies
pip install -r requirements.txt

# 2. Verify environment & hardware (CUDA VRAM, ONNX Runtime, PyPDFium2, EasyOCR, MRZ Core)
python test_setup.py

# 3. Run unit tests for MRZ Core (TD3, TD3_DDMMYYYY, KSA Umrah eVisa, TD1 ID/Iqama)
python -m pytest tests/test_mrz_core.py -v

End-to-End Scanning (Image, PDF, or Folder -> JSON)

# Scan a folder of passport images on CPU
python -m ml1_detector.pipeline.scanner \
  --source real_data/passports \
  --detector-device cpu \
  --ocr-device cpu \
  --project pipeline_runs \
  --name passports_scan

# Scan a multi-page Umrah eVisa PDF on CPU
python -m ml1_detector.pipeline.scanner \
  --source real_data/visas_pdfs \
  --detector-device cpu \
  --ocr-device cpu \
  --project pipeline_runs \
  --name umrah_pdf_scan

# Scan an entire directory containing both images and PDFs recursively (fast mode)
python -m ml1_detector.pipeline.scanner \
  --source real_data \
  --detector-device cpu \
  --ocr-device gpu \
  --fast \
  --project pipeline_runs \
  --name full_scan

πŸ“„ JSON Output Structure

Each scan writes scan_results.json containing a run-level summary and per-page document records:

{
  "summary": {
    "total_pages_processed": 15,
    "yolo_detection_count": 15,
    "yolo_detection_rate": 1.0,
    "shape_valid_count": 15,
    "shape_valid_rate": 1.0,
    "parsed_count": 15,
    "parsed_rate": 1.0,
    "checksum_verified_count": 15,
    "checksum_verified_rate": 1.0
  },
  "documents": [
    {
      "document_id": "sample_passport_01",
      "source_type": "image",
      "page_number": 1,
      "status": "VERIFIED",
      "detector": {
        "yolo_detected": true,
        "selected_box_source": "yolo11n_detector",
        "bbox_xyxy": [21, 513, 938, 634],
        "confidence": 0.9036
      },
      "mrz": {
        "format": "TD3",
        "document_type": "passport",
        "issuing_country": "UTO",
        "nationality": "UTO",
        "document_number": "L898902C3",
        "name": {
          "surname": "ERIKSSON",
          "given_names": ["ANNA", "MARIA"],
          "full_name": "ANNA MARIA ERIKSSON"
        },
        "date_of_birth": {
          "raw": "740812",
          "iso": "1974-08-12",
          "ambiguous": false
        },
        "sex": "F",
        "expiration_date": {
          "raw": "120415",
          "iso": "2012-04-15",
          "ambiguous": false
        },
        "checksums_valid": true,
        "corrections": []
      }
    }
  ]
}

⚑ Hardware-Accelerated Training (6GB VRAM + 32GB RAM)

The training pipeline (ml1_detector.training.train) is specifically tuned for machines with 6GB GPU VRAM (e.g., NVIDIA RTX 4050 Laptop GPU) and 32GB System RAM:

  • amp: true (FP16 Mixed Precision) & TF32 Matmul: Cuts GPU memory usage nearly in half while accelerating tensor operations.
  • cache: "ram": Preloads training and validation images into the 32GB System RAM so disk I/O bottlenecks are eliminated.
  • Safe VRAM Batch Auto-Capping: Automatically caps batch size at 8 for 640Γ—640 and 2 for 1280Γ—1280 on 6GB GPUs to prevent CUDA Out-Of-Memory errors.
# Build the unified multi-document dataset (Passports + Umrah PDFs + Synthetic IDs/Iqamas)
python -m ml1_detector.data.prepare_unified_dataset \
  --max-pages-per-pdf 15 \
  --num-synthetic-train 120 \
  --num-synthetic-val 30

# Fine-tune YOLOv11n with GPU FP16 AMP and RAM caching
python -m ml1_detector.training.train \
  --data ml1_detector/datasets/unified_mrz_boost.yaml \
  --weights gpu_runss/synthetic_v2_first_ft1_ocr_safe_v2/weights/best.pt \
  --device gpu \
  --epochs 5 \
  --batch-size 8 \
  --image-size 640 \
  --cache ram \
  --amp \
  --project gpu_runss \
  --name unified_mrz_v1 \
  --exist-ok

# Export the fine-tuned model to ONNX for lightweight CPU deployment
python -m ml1_detector.inference.onnx_export \
  --weights gpu_runss/unified_mrz_v1/weights/best.pt \
  --device gpu \
  --image-size 640 \
  --output exports/unified_mrz_yolo11n_640.onnx

πŸ”’ Privacy & Sensitive Data Isolation

All sensitive documents and local runtime outputs are strictly isolated and excluded from Git via .gitignore:

  • real_data/ (real passport images and Umrah visa PDFs)
  • progress/ (local run state and machine-specific paths)
  • pipeline_runs/, gpu_runss/, exports/ (cropped outputs, scan results, and weights)
  • *.pdf, *.pt, *.onnx

πŸ“Š Benchmark Summary

Dataset / Source Pages YOLOv11n ONNX (CPU) Detections Shape-Valid MRZ Parsed Documents Checksum Verified
Real Passports Set (real_data/passports) 15 15/15 (100.0%) 15/15 (100.0%) 15/15 (100.0%) 15/15 (100.0%)
Multi-Page Passport PDF Batch 6 6/6 (100.0%) 6/6 (100.0%) 6/6 (100.0%) 6/6 (100.0%)
Saudi Umrah eVisas PDF Batch A 10 10/10 (100.0%) 10/10 (100.0%) 10/10 (100.0%) 10/10 (100.0%)
Saudi Umrah eVisas PDF Batch B 5 5/5 (100.0%) 5/5 (100.0%) 5/5 (100.0%) 5/5 (100.0%)
Total Verified Benchmark 36 36/36 (100.0%) 36/36 (100.0%) 36/36 (100.0%) 36/36 (100.0%)

πŸ›£ Roadmap

  • βœ… Phase 1 β€” Stabilize ML#1 Training & Synthetic Scene Generator (v2)
  • βœ… Phase 2 β€” OCR-Safe Real-Scene Adaptation & Lightweight ONNX CPU Export
  • βœ… Phase 3 β€” Multi-Variant Preprocessing, Tilt Recovery & Geometry-Aware OCR Line Assembly
  • βœ… Phase 4 β€” Unified End-to-End Pipeline (Images, Multi-Page PDFs, Passports, Umrah eVisas, TD1/TD2 IDs) -> JSON
  • πŸ”œ Phase 5 β€” Dedicated Custom CRNN (CNN + BiLSTM + CTC) OCR Model Trained & Exported to ONNX

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages