Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 28 additions & 5 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,24 @@ which keep the sections as `children` and each get a detail column. Zooms go one
when W110 puts the regions in zooms of their own, their groups get a note instead.
`--min-size 0` turns folding off.

## Typical task: map an ESP-IDF partition table
1. Find the table: the CSV that `CONFIG_PARTITION_TABLE_FILENAME` in the project's `sdkconfig`
names, or the `build/partition_table/partition-table.bin` the build writes from it.
2. `memmap from-partitions partitions.csv -o layout.yaml --sdkconfig sdkconfig --json --strict`.
The table does not record the flash size, so give `--sdkconfig`, which also gives the
partition table and bootloader offsets. Without one, give `--flash-size 4M`,
`--bootloader-offset` (0x1000 on the ESP32), and `--table-offset` when
`CONFIG_PARTITION_TABLE_OFFSET` is not 0x8000.
3. Fix W113 (a rule current ESP-IDF rejects, checked for a CSV only) by its hint. W100 marks a
partition edge inside a 4K erase sector.
4. Continue from step 4 of the first task.

Partitions are placed as ESP-IDF places them: a blank offset follows the partition above it,
aligned to 64K for an app and 4K for the rest, starting after the partition table. Offsets are
positions in the flash chip, drawn from 0, not CPU addresses. The column also holds the
bootloader, the partition table and `Free` blocks; the header comments say where each setting
came from. https://github.com/iharhl/memmap/blob/main/docs/from-partitions.md has the details.

## Rules
- Never guess keys; run `memmap schema` when unsure.
- Ranges are inclusive: end = start + size - 1.
Expand Down Expand Up @@ -83,11 +101,12 @@ when W110 puts the regions in zooms of their own, their groups get a note instea
Quote `--set` values that contain `[` or `{`, since shells expand them.

## JSON output
With `--json`, `check`, `render` and `from-elf` always print one object on stdout, for
validation errors and for usage and IO failures alike: `{"ok": ..., "diagnostics": [...]}`.
`render` adds `output` and `format` on success, `from-elf` adds `output` once it has written the
config. `describe --json` prints the same object on failure. Only a mistake in the command line
itself, such as an unknown flag, is reported as plain text on stderr.
With `--json`, `check`, `render`, `from-elf` and `from-partitions` always print one object on
stdout, for validation errors and for usage and IO failures alike:
`{"ok": ..., "diagnostics": [...]}`. `render` adds `output` and `format` on success, and the two
importers add `output` once they have written the config. `describe --json` prints the same
object on failure. Only a mistake in the command line itself, such as an unknown flag, is
reported as plain text on stderr.

## Checking the result
`memmap describe layout.yaml --json` returns:
Expand Down Expand Up @@ -147,6 +166,7 @@ Every diagnostic has a stable code, a YAML path, a line and usually a `hint`. In
| W110 | `from-elf`: section outside every memory region, placed at the top level |
| W111 | `from-elf`: section overlaps another one, left out |
| W112 | `from-elf`: map region left out (no size, past the address space, overlap, too small) |
| W113 | `from-partitions`: the CSV breaks a rule of ESP-IDF's partition table check |
| E090 | Usage or IO failure, such as a missing file or bad `--set` (`--json` only) |
| E091 | Render or internal failure, such as missing cairo (`--json` only) |

Expand All @@ -161,5 +181,8 @@ Exit codes: `0` ok, `1` validation errors, `2` usage or IO error, `3` render err
real-firmware goldens: `MEMMAP_UPDATE_GOLDEN=1 uv run pytest tests/test_elf_snapshots.py
tests/test_linker.py`. docs/from-elf.md describes them and how to add an image or a
map.
- After an intended change to `from-partitions` or the drawing, regenerate its goldens:
`MEMMAP_UPDATE_GOLDEN=1 uv run pytest tests/test_from_partitions.py`. docs/from-partitions.md
describes the real ESP-IDF builds they come from.
- A new diagnostic code or recipe goes in this file too. `memmap guide` ships it to users, and
`tests/test_cli.py` covers the recipes.
41 changes: 38 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,16 @@
# memmap

Draw datasheet-style memory maps of MCU memory. You describe the layout in a YAML file, or
have `memmap` write that file from a linked firmware image. Either way you get the same config,
which `memmap` checks and renders to SVG, PNG or PDF.
have `memmap` write that file from a linked firmware image or an ESP-IDF partition table. Either
way you get the same config, which `memmap` checks and renders to SVG, PNG or PDF.

- **Write it by hand.** One YAML file holds the memory (`memory`) and how to draw it (`view`),
kept separate. Start from one of the [examples](#examples).
- **Generate it from firmware.** `from-elf` reads a linked ELF and the linker's map file (GNU ld,
Keil armlink or IAR), and places every section at its real address and size in the chip's flash
and RAM regions.
- **Generate it from a partition table.** `from-partitions` reads an ESP-IDF partition table, CSV
or binary, and places every partition where the ESP-IDF build would.
- **Check before drawing.** Validation catches overlaps, overflow and flash sector misalignment,
with the YAML path, line and a hint for each.
- **Verify as text.** Output is deterministic, and `describe` lists everything drawn, so agents and
Expand Down Expand Up @@ -45,7 +47,7 @@ export DYLD_FALLBACK_LIBRARY_PATH=/opt/homebrew/lib

## Quick start

Choose a starting point. Both give you a `layout.yaml`.
Choose a starting point. Each gives you a `layout.yaml`.

**A layout you are planning.** Copy the closest example:

Expand All @@ -59,6 +61,12 @@ cp examples/custom_bootloader.yaml layout.yaml
memmap from-elf build/firmware.elf -o layout.yaml --map build/firmware.map
```

**An ESP-IDF project.** Pass its partition table and its `sdkconfig`:

```bash
memmap from-partitions partitions.csv -o layout.yaml --sdkconfig sdkconfig
```

Then check the config, see what will be drawn, and render it:

```bash
Expand Down Expand Up @@ -130,6 +138,24 @@ ordinary config: rename or merge blocks, add `view` options, then `render` it.
[docs/from-elf.md](docs/from-elf.md) covers what is read from each map, what happens when it is
not enough, and how sections become blocks.

## Generating a config from an ESP-IDF partition table

`from-partitions` draws the flash of an ESP-IDF project: the bootloader, the partition table,
each partition and the free space between them. It reads the partition table CSV the project
names in `CONFIG_PARTITION_TABLE_FILENAME`, or the `build/partition_table/partition-table.bin`
the build writes. Partitions with a blank offset are placed as the build places them, so the
drawing matches the binary table.

The table does not record the flash size, so pass the project's `sdkconfig`, which also gives
the partition table and bootloader offsets. Without one, give the flash size:

```bash
memmap from-partitions partitions.csv -o layout.yaml --flash-size 4M
```

A CSV that breaks a rule of ESP-IDF's own table check gets W113, with a hint.
[docs/from-partitions.md](docs/from-partitions.md) covers what is read and drawn, and the limits.

## Commands

| Command | Purpose |
Expand All @@ -138,6 +164,7 @@ not enough, and how sections become blocks.
| `memmap check CONFIG [--json] [--strict]` | Validate only |
| `memmap describe CONFIG [--json]` | Resolved regions, blocks, labels and option origins |
| `memmap from-elf ELF [-o OUT] [--map FILE] [--region id=start:size[:name]] [--min-size N] [--force] [--json] [--strict]` | Write a config from a linked ELF, then check it |
| `memmap from-partitions TABLE [-o OUT] [--sdkconfig FILE] [--flash-size SIZE] [--table-offset OFFSET] [--bootloader-offset OFFSET] [--force] [--json] [--strict]` | Write a config from an ESP-IDF partition table, then check it |
| `memmap schema` | JSON Schema of the config |
| `memmap guide` | Print the agent guide, for agents using an installed copy |

Expand All @@ -148,6 +175,7 @@ scripts and agents.

- [AGENTS.md](AGENTS.md): task guide, recipes and the full diagnostic list. `memmap guide` prints it.
- [docs/from-elf.md](docs/from-elf.md): drawing a linked firmware image, its map readers and limits.
- [docs/from-partitions.md](docs/from-partitions.md): drawing an ESP-IDF partition table.
- [Examples](#examples): one config per feature set, below.

## Examples
Expand Down Expand Up @@ -204,6 +232,13 @@ regenerate those too:
MEMMAP_UPDATE_GOLDEN=1 uv run pytest tests/test_elf_snapshots.py tests/test_linker.py
```

`from-partitions` is tested on real ESP-IDF builds, where the `partition-table.bin` the build
wrote checks how memmap places the CSV. After an intended change to it, review and regenerate:

```bash
MEMMAP_UPDATE_GOLDEN=1 uv run pytest tests/test_from_partitions.py
```

[docs/from-elf.md](docs/from-elf.md#testing) explains the approach and shows the drawings.

Text is measured with a built-in Helvetica/Arial width table, so wrapping is slightly approximate
Expand Down
166 changes: 166 additions & 0 deletions docs/from-partitions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
# Drawing an ESP-IDF partition table

`memmap from-partitions` writes a config from an ESP-IDF partition table: one flash column with
the bootloader, the partition table, every partition and the free space between them.

```bash
memmap from-partitions partitions.csv -o layout.yaml --sdkconfig sdkconfig
```

![The flash of an ESP32 HomeKit project, drawn by from-partitions](../tests/golden/partitions/esp32_homekit_custom_type.svg)

- [What it reads](#what-it-reads)
- [Where the settings come from](#where-the-settings-come-from)
- [What is drawn](#what-is-drawn)
- [Checks](#checks)
- [Limits](#limits)
- [Testing](#testing)

## What it reads

The table is either of the files ESP-IDF works from, told apart by their content:

| File | Where | Offsets |
| --- | --- | --- |
| Partition table CSV | the project, named by `CONFIG_PARTITION_TABLE_FILENAME` | worked out by memmap, as the build does |
| `partition-table.bin` | `build/partition_table/` | as the build wrote them |

A CSV has the columns `Name, Type, SubType, Offset, Size, Flags`, and memmap reads it the way
ESP-IDF's `gen_esp32part.py` does:

- Lines starting with `#` and blank lines are skipped. A header line without `#` is an error,
as it is for the build.
- Numbers are Python integers, so `0x9000`, `36864`, `0o110000` and `0x9_000` all work, with an
optional `K` or `M` suffix in either case.
- Types are `app`, `data`, `bootloader`, `partition_table` or a number. Subtypes are ESP-IDF's
names, such as `ota_0`, `nvs` or `spiffs`, or a number. A subtype name that a component
registers is kept as written.
- A blank offset follows the partition above it, aligned to 64K for an app and to 4K for
anything else. The first one starts after the partition table. Partitions with offsets must
be listed in address order, as the build places them in file order.
- A negative size gives the offset the partition ends at: `-0x400000` runs to the end of 4 MB.
- `bootloader` and `partition_table` entries with the `primary` subtype sit at the bootloader
and partition table offsets, whatever their offset and size columns say.
- Flags are `encrypted` and `readonly`, separated by `:`.
- A byte order mark selects UTF-8, UTF-16 or UTF-32.

A binary table is a list of 32-byte entries with an MD5 entry and `0xFF` padding. memmap checks
the MD5 and reads the entries as they are.

## Where the settings come from

The table gives the partitions, but not how big the flash is, where the partition table sits or
where the ROM loads the bootloader from. memmap takes each from the options, else from the
`sdkconfig` given with `--sdkconfig`, else from a default:

| Setting | Option | sdkconfig | Default |
| --- | --- | --- | --- |
| Flash size | `--flash-size 4M` | `CONFIG_ESPTOOLPY_FLASHSIZE` | none: memmap stops and says what to give |
| Partition table offset | `--table-offset 0x8000` | `CONFIG_PARTITION_TABLE_OFFSET` | 0x8000 |
| Bootloader offset | `--bootloader-offset 0x1000` | `CONFIG_BOOTLOADER_OFFSET_IN_FLASH`, else the chip in `CONFIG_IDF_TARGET` | not known |
| Recovery bootloader | | `CONFIG_BOOTLOADER_RECOVERY_OFFSET` | none |
| Secure boot V1 | | `CONFIG_SECURE_BOOT_V1_ENABLED` | off |

The bootloader starts at 0x1000 on the ESP32 and ESP32-S2, at 0x2000 on the ESP32-C5, ESP32-H4,
ESP32-P4 and ESP32-S31, and at 0 on the other chips. A binary table with `primary` bootloader
or partition table entries gives those offsets itself; an option or sdkconfig that says
otherwise belongs to another build, and memmap stops. The header comments of the config say
which source gave each setting.

Projects without an sdkconfig, such as Arduino-ESP32 or PlatformIO ones, give the flash size
and, for a drawing that places the bootloader, `--bootloader-offset`.

When the flash size is missing and an `sdkconfig` lies next to the table, the hint names it.

## What is drawn

One region, `flash`, from 0 to the flash size. Offsets are positions in the flash chip, not the
addresses the CPU reads the flash through. Its blocks, in address order:

- `Reserved` below the bootloader, on a chip whose bootloader does not start at 0.
- `Bootloader`, from the bootloader offset up to the partition table. Without a known offset it
starts at 0, with a note saying it starts at 0x0, 0x1000 or 0x2000 by chip.
- `Partition table`, one 4K sector.
- Each partition, named as in the table, with a note giving its type, subtype and flags, such as
`app, ota_0` or `data, nvs_keys, encrypted`. A subtype left blank in a CSV is not shown, as
its value depends on the ESP-IDF version.
- `Free` for the space between them, with its size.

A table that has its own `primary` bootloader or partition table entry is drawn with that entry
in place of the generated block. The region has 4K `sectors`, so `check` reports a partition
edge inside an erase sector as W100; the tick marks are turned off in `style`. The column title
gives how much of the flash the bootloader, the table and the partitions take.

## Checks

memmap stops, writing nothing, when the table cannot be drawn:

- the CSV cannot be read: an unknown type or flag, a blank type or size, a bad number;
- a partition starts before the end of the partition table, or overlaps one listed before it;
- a partition runs past the end of the flash, or has size 0;
- the partition table offset is 0, is not a multiple of 0x1000, or runs past the end of the
flash, or the bootloader is not below it;
- a binary table's own bootloader or partition table entry disagrees with the options or the
sdkconfig.

The message names the CSV line, and the hint says what to change. With `--json` it is E090.

A CSV that can be drawn but breaks a rule of ESP-IDF's own table check gets W113, which
`--strict` turns into a failure:

- an app starts off a 64K boundary, or anything else off a 4K one;
- an app's size is not a multiple of 4K, or of 64K with secure boot V1;
- a read-write NVS partition is smaller than 12K;
- `ota` or `coredump` data is marked `readonly`;
- there is more than one `otadata` or TEE `otadata` partition, or it is not 8K;
- two partitions share a name;
- a subtype name is not one ESP-IDF defines. The build accepts it only when a component
registers it, so check the spelling.

Some of these rules are newer than others: a table that ESP-IDF 5.2 built can get W113 from a
rule added later. A binary table is not checked, as the build that wrote it checked it already.

## Limits

- **Environment variables.** The build expands `$NAME` and `${NAME}` in a CSV from the
environment. memmap stops at any `$` followed by a name in a partition line, and asks for
the value, so that the drawing does not depend on the shell it ran in.
- **Blank subtypes.** ESP-IDF 5 reads a blank subtype as `0x06`, undefined. Earlier versions
read it as 0, which is also `otadata`'s subtype. memmap follows ESP-IDF 5.
- **Flash size `detect`.** Older sdkconfigs can leave the size to be detected when flashing.
Give `--flash-size` then.
- **Other formats.** Only ESP-IDF tables are read. Arduino-ESP32, PlatformIO and MicroPython use
the same CSV format, so their tables work too.
- **Table length.** ESP-IDF's binary table holds at most 95 entries. memmap does not check this.

## Testing

`tests/test_from_partitions.py` runs on real tables, kept in `tests/fixtures/esp_partitions/`
with their sources and SHA-256 in `index.json`:

- **Builds.** Five ESP-IDF projects that committed their partition table, their `sdkconfig` and
the `partition-table.bin` their build wrote. Between them: a partition table moved to 0x18000,
a custom partition type with a blank subtype, blank offsets, rules that newer ESP-IDF added,
a 1K partition, and bootloaders at 0x0, 0x1000 and 0x2000. The `.bin` is the ground truth:
memmap's placement of the CSV must match it entry for entry. The sdkconfig keeps only the
lines memmap reads.
- **Tables.** Two tables built into ESP-IDF, an ESP-IDF example with `bootloader` and
`partition_table` entries, and the default tables of Arduino-ESP32 and MicroPython.

Each one is imported to a golden config in `tests/golden/partitions/`, and two builds are also
drawn to a golden SVG, as the drawing code is the same for every table. Each fixture shows
something the others do not; keep it that way. To add a table, add it to `refresh.py` in the
fixtures folder, then:

```bash
uv run python tests/fixtures/esp_partitions/refresh.py --download
```

```bash
MEMMAP_UPDATE_GOLDEN=1 uv run pytest tests/test_from_partitions.py
```

Before the fixtures were chosen, the same comparison ran on 121 ESP-IDF builds found on GitHub.
memmap placed 117 exactly as their `partition-table.bin`. Two differed only in a blank subtype,
which their older ESP-IDF read as 0. The `.bin` of the other two came from a different table
than the one committed, a build left stale.
Loading
Loading