ESP32 Triple OTA with TMR Fault Tolerance

A high-resilience embedded firmware framework for the ESP32 that implements three-way OTA partition management with automatic corruption detection, self-repair, and two-layer Triple Modular Redundancy (TMR) for critical data and arithmetic operations.

This project serves as a framework for industrial-grade firmware resilience. It employs three OTA partitions (ota_0, ota_1, ota_2) with runtime health monitoring. If any partition becomes corrupted, the system automatically repairs it by copying from a known-good source. Additionally, it implements software-based TMR for sensor data and math to tolerate memory bit-flips and transient CPU faults.

🚀 Core Architecture

1. Triple OTA Partition Management

The system treats the flash memory as a redundant array of firmware images.

• Continuous Health Checks: Scans all three partitions at boot and every 60 seconds using esp_partition_get_sha256().
• Autonomous Self-Repair: If a partition is marked INVALID, the system identifies a valid source partition and performs a chunk-based (4KB) destructive copy to restore the corrupted slot.
• Fail-Safe Boot: If the currently running partition is found to be invalid, the system automatically switches the boot partition to a valid slot and restarts.
• Watchdog Integration: All flash operations are wrapped with esp_task_wdt resets to prevent system hangs during long erase/write cycles.

2. Two-Layer Triple Modular Redundancy (TMR)

To protect against RAM corruption (soft errors), the system implements a high-order redundancy pattern.

The var Class (Data Redundancy)

Instead of storing a single float, critical values are stored in a 3×3 matrix (9 copies).

• Write: Values are written to all 9 slots with immediate read-back verification.
• Read (Two-Stage Voting):
  • Layer 1: Computes the majority vote for each of the 3 rows $\rightarrow$ 3 candidate values.
  • Layer 2: Computes the majority vote of those 3 candidates $\rightarrow$ final validated value.
• Auto-Scrubbing: During a read() operation, the system automatically identifies and repairs any mismatched copies in the matrix.

The math Class (Computational Redundancy)

Prevents transient CPU faults during arithmetic.

• Redundant Execution: Performs the operation (add, sub, mul, div) 9 times independently.
• Voting: Applies the same two-stage TMR voting to the 9 results to return the most probable correct value.
• Barriers: Uses volatile keywords and assembly memory barriers to prevent the compiler from optimizing away the redundant calculations.

---

🛠 Project Structure

tmr-esp32/
├── firmware_01/          # Primary firmware source (.ino)
├── firmware_02/          # Duplicate for flashing to ota_1
├── firmware_03/          # Duplicate for flashing to ota_2
├── scripts/
│   ├── main.py           # Build & Flash automation (Arduino-CLI + esptool)
│   ├── test.py           # Automated Test Suite (Corruption & Recovery)
│   └── partitions.csv     # Custom partition table defining ota_0, 1, 2
├── LOG.md                # Execution logs and test results
└── README.md             # Project documentation

💻 Setup & Usage

Prerequisites

• Arduino CLI installed and configured.
• Python 3.8+ with pyserial library.
• ESP32 with $\ge 4\text{MB}$ flash.

Build & Flash

1. Open scripts/main.py and configure your PORT (e.g., "COM3") and paths to arduino-cli.exe and esptool.exe.
2. Run the automation script:

   python scripts/main.py

   This will compile the code three times and flash all three OTA slots and the partition table in one pass.

Verification & Testing

The scripts/test.py script provides a rigorous validation pipeline:

1. Normal Operation Test: Verifies TMR math and variable stability.
2. Corruption Test: Systematically corrupts ota_0, ota_1, and ota_2 one by one via esptool and monitors the serial output to verify that the firmware detects the failure and successfully repairs the partition.

---

📊 Technical Specifications

Flash Layout

Partition	Offset	Size	Purpose
ota_0	0x10000	0x140000	Primary App Slot
ota_1	0x150000	0x140000	Redundant App Slot 1
ota_2	0x290000	0x140000	Redundant App Slot 2

Task Scheduling

Task	Priority	Frequency	Responsibility
firmwareTask	5	60s	Partition scanning, SHA-256 validation, and repair
baseOperation	4	5s	TMR data processing and sensor simulation

⚠️ Development Roadmap (Production To-Dos)

• Memory Optimization: Replace malloc in repair functions with static buffers.
• Compliance: Transition from C++ templates to pure C for NASA/JPL Power of Ten safety-critical compliance.
• Performance: Implement CRC32 as a primary fast-check before running full SHA-256.
• Anti-Brick: Implement non-volatile rollback counters to prevent boot-looping on corrupted updates.