Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/workflows/default.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ jobs:
matrix:
python-version: ["3.10", "3.13"]
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v6
- uses: actions/checkout@v7
- uses: astral-sh/setup-uv@v10.3.3
with:
python-version: ${{ matrix.python-version }}
- run: uv sync --frozen
Expand All @@ -32,8 +32,8 @@ jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v6
- uses: actions/checkout@v7
- uses: astral-sh/setup-uv@v10.3.3
with:
python-version: "3.13"
- run: uv build
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ jobs:
env:
TAG: ${{ github.event.release.tag_name }}
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v6
- uses: actions/checkout@v7
- uses: astral-sh/setup-uv@v10.3.3
with:
python-version: "3.13"

Expand Down
31 changes: 26 additions & 5 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,20 @@ Run commands as `memmap ...` when it is installed, or `uv run memmap ...` inside
4. `memmap describe layout.yaml` -> confirm ranges and labels match intent.
5. `memmap render layout.yaml -o layout.svg --json --strict`

## Typical task: map a linked firmware image
1. Read `ORIGIN` and `LENGTH` of each region in the linker script's `MEMORY` block, or in the
`Memory Configuration` table of a GNU ld `.map` file.
2. `memmap from-elf firmware.elf -o layout.yaml --region flash=0x08000000:64K
--region ram=0x20000000:20K --json` -> writes the config, then checks it like `check`.
Fix W110 (a section outside every region) by correcting or adding a `--region`.
3. Continue from step 4 above. Rename or merge blocks in `layout.yaml` as needed. `--force`
overwrites an existing output.

In a region of more than 8 blocks, small sections fold into groups such as `small_sections`,
which keep the sections as `children` and each get a detail column. Zooms go one level deep, so
when W110 puts the regions in zooms of their own, their groups get a note instead.
`--min-size 0` turns folding off.

## Rules
- Never guess keys; run `memmap schema` when unsure.
- Ranges are inclusive: end = start + size - 1.
Expand All @@ -39,14 +53,16 @@ Run commands as `memmap ...` when it is installed, or `uv run memmap ...` inside
- One-off change: memmap render layout.yaml --set 'view.columns[0].width=150'
- Label one edge: view.overrides: [{ region: ahb1, top_label: end }]
- Block height: view.overrides: [{ region: boot, height: 48 }]
- Multi-line note: note: "line one\nline two" (long notes also wrap)

Quote `--set` values that contain `[` or `{`, since shells expand them.

## JSON output
With `--json`, `check` and `render` always print one object on stdout, for validation errors
and for usage and IO failures alike: `{"ok": ..., "diagnostics": [...]}`. `render` adds `output`
and `format` on success. `describe --json` prints the same object on failure. Only a mistake in
the command line itself, such as an unknown flag, is reported as plain text on stderr.
With `--json`, `check`, `render` and `from-elf` always print one object on stdout, for
validation errors and for usage and IO failures alike: `{"ok": ..., "diagnostics": [...]}`.
`render` adds `output` and `format` on success, `from-elf` adds `output` once it has written the
config. `describe --json` prints the same object on failure. Only a mistake in the command line
itself, such as an unknown flag, is reported as plain text on stderr.

## Checking the result
`memmap describe layout.yaml --json` returns:
Expand Down Expand Up @@ -87,11 +103,13 @@ Every diagnostic has a stable code, a YAML path, a line and usually a `hint`. In
| E025 | Column expands into itself |
| W100 | Region boundary not on a flash sector boundary |
| W101 | Gap filled as reserved (only with `--verbose`) |
| W102 | Label text forced a block over `max_height` |
| W102 | Label or note text forced a block over `max_height` |
| W103 | Region start not aligned to its size (needs `memory.check_alignment: true`) |
| W104 | Sector list does not add up to the region size |
| W105 | Block name was split inside a word to fit the column |
| W106 | Text overlaps other text, or is wider than its block |
| W110 | `from-elf`: section outside every `--region`, placed at the top level |
| W111 | `from-elf`: section overlaps another one, left out |
| E090 | Usage or IO failure, such as a missing file or bad `--set` (`--json` only) |
| E091 | Render or internal failure, such as missing cairo (`--json` only) |

Expand All @@ -102,5 +120,8 @@ Exit codes: `0` ok, `1` validation errors, `2` usage or IO error, `3` render err
3.10 and the latest supported version.
- After an intended drawing change, review the SVG diff, then run
`MEMMAP_UPDATE_GOLDEN=1 uv run pytest tests/test_golden.py`.
- After an intended change to `from-elf` or the drawing, also review and regenerate the
real-firmware goldens: `MEMMAP_UPDATE_GOLDEN=1 uv run pytest tests/test_elf_snapshots.py`.
docs/elf-testing.md describes them and how to add an image.
- A new diagnostic code or recipe goes in this file too. `memmap guide` ships it to users, and
`tests/test_cli.py` covers the recipes.
40 changes: 40 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,9 +66,38 @@ uv run memmap render layout.yaml -o layout.svg
| `memmap render CONFIG [-o OUT] [--format svg\|png\|pdf] [--json] [--strict] [--set key=value]` | Validate, then draw; `--strict` fails on warnings |
| `memmap check CONFIG [--json] [--strict]` | Validate only; `--strict` fails on warnings |
| `memmap describe CONFIG [--json]` | Resolved regions, blocks, labels and option origins |
| `memmap from-elf ELF [-o OUT] [--region id=start:size[:name]] [--min-size N] [--force] [--json] [--strict]` | Write a config from a linked ELF's sections, then check it |
| `memmap schema` | JSON Schema of the config |
| `memmap guide` | Print the agent guide, for agents using an installed copy |

### From an ELF file

`from-elf` writes a config from the allocated sections of a linked firmware image: `.text`,
`.data`, `.bss` and so on, each at its real address and size. The ELF does not record how big
flash and RAM are, so give them with `--region`, copying `ORIGIN` and `LENGTH` from the
linker script's `MEMORY` block, or from the `Memory Configuration` table of a GNU ld map file:

```bash
uv run memmap from-elf build/firmware.elf -o layout.yaml --region flash=0x08000000:64K --region ram=0x20000000:20K
```

You get one column per region, titled with how much of it is used:

- Initialised data is drawn twice: in RAM where it runs, and in flash where startup code copies
it from. Each loadable segment is one flash block: `.data (load)`, with a note such as
`copied to RAM at 0x2000 0000`, or `RAM init data` when it holds several sections, as Zephyr's
do (`code and data` when it includes code).
- Gaps of at least `--min-size` (default 64 bytes) become Free blocks with their size, rounded
down and marked `≈` when not exact.
- In a region of more than 8 blocks, runs of smaller sections fold into one block that holds
them as children, and each such group gets a detail column beside it. Zooms go one level
deep: when sections outside every `--region` put the regions in a zoom of their own, their
groups list the section names in a note instead.
- Without `--region`, sections are drawn on the whole address space. Object files converted
from a `.hex` or `.bin` image, which keep their real addresses, are read too.

The output is an ordinary config: edit it, then `render` it.

A minimal config:

```yaml
Expand Down Expand Up @@ -148,6 +177,17 @@ regenerate them:
MEMMAP_UPDATE_GOLDEN=1 uv run pytest tests/test_golden.py
```

`from-elf` is also tested on eight real firmware images, kept as small header snapshots with
golden configs and drawings in `tests/golden/elf/`. After an intended change to the importer or
the drawing, review and regenerate those too:

```bash
MEMMAP_UPDATE_GOLDEN=1 uv run pytest tests/test_elf_snapshots.py
```

[docs/elf-testing.md](docs/elf-testing.md) explains the approach, lists the images and shows
their maps.

Text is measured with a built-in Helvetica/Arial width table, so wrapping is slightly approximate
for other fonts.

Expand Down
121 changes: 121 additions & 0 deletions docs/elf-testing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# Testing the ELF import

`memmap from-elf` makes judgement calls: which sections to draw, which to fold, where free space
goes, what to call the flash copy of `.data`. Those are easy to get wrong on layouts nobody
thought of, so besides hand-made cases the tests run real firmware through the importer.

## Test layers

| Layer | Where | Covers |
| --- | --- | --- |
| Synthetic ELFs | `tests/test_from_elf.py`, built by `tests/elf_builder.py` | One rule per test: folding, free gaps, load images, several memory banks, ids, 64-bit, big-endian, converted images, bad files, the CLI |
| Real builds | `tests/fixtures/firmware*.elf`, from `tests/fixtures/elf_src/` | The reader on files binutils wrote: one firmware as 32- and 64-bit, little- and big-endian. `build.sh` rebuilds them with `arm-none-eabi-gcc` and GNU `objcopy` |
| Real firmware | `tests/test_elf_snapshots.py`, `tests/fixtures/elf_snapshots/` | Eight images from different toolchains, OSes and architectures, against golden configs and drawings |

## Snapshots instead of ELF files

The real images total 6.5 MB, mostly debug info memmap never reads, and they are other projects'
builds. So the repo keeps a snapshot of each instead: the section headers, the loadable segments,
the `--region` specs to import with, and the source URL and SHA-256. That is 3 to 6 KB of JSON,
with no code or data from the firmware.

The test rebuilds each snapshot into a minimal ELF, imports it, and checks that the import reports
nothing, that the config matches `tests/golden/elf/<name>.yaml` and passes `check --strict`, and
that its drawing matches `tests/golden/elf/<name>.svg`. The configs from snapshots were identical
to those from the real files when the snapshots were made.

Snapshots cannot catch quirks in how a toolchain writes the file itself, as the rebuilt files are
clean. The real builds and the synthetic tests cover the reader.

## The images

All come from [Renode](https://github.com/renode/renode), whose board scripts
(`scripts/single-node/*.resc`) load prebuilt firmware from `dl.antmicro.com`. Locally they live
in `build/elf/`, which git ignores.

| Snapshot | Firmware | Regions |
| --- | --- | --- |
| `nrf52840_zephyr_shell` | Zephyr shell, nRF52840 (Cortex-M4) | flash 1M at 0, RAM 256K at 0x20000000 |
| `r52_zephyr_philosophers` | Zephyr philosophers, Cortex-R52 | DRAM 2G at 0 |
| `fe310_zephyr_shell` | Zephyr shell, SiFive FE310 (RISC-V), runs from SPI flash | SPI flash 12M at 0x20400000, DTIM 16K at 0x80000000 |
| `stm32f412_tock_kernel` | Tock (Rust) kernel, STM32F412 | flash 1M at 0x08000000, SRAM 256K at 0x20000000 |
| `msp430f2619_hello` | Hello world, MSP430F2619 (16-bit) | flash 0x1DF00 at 0x2100, RAM 4K at 0x1100 |
| `stm32f746_dartino_lines` | Dartino (Dart VM) demo, STM32F746 | flash 1M at 0x08000000, SRAM 320K at 0x20000000 |
| `stm32f4discovery` | STM32F4 Discovery demo, STM32F407 | flash 1M at 0x08000000, SRAM 128K at 0x20000000 |
| `apollo4_hello_uart` | AmbiqSuite hello world, Ambiq Apollo4 (Cortex-M4F) | MRAM 1952K at 0x18000, TCM 384K at 0x10000000, shared SRAM 1M at 0x10060000 |

The region sizes are the chips' real flash and RAM, from datasheets and Renode's platform files
(`platforms/cpus/*.repl`), since the ELF does not record them. Where the firmware starts above
the bottom of its memory, the region starts there too: the FE310 shell runs from the user part of
SPI flash, and the Apollo4 application starts 96 KB into its 2 MB MRAM.

## Refreshing and adding images

`tests/fixtures/elf_snapshots/refresh.py` lists the images with their URL, description and
`--region` specs, and rewrites every snapshot whose ELF is in `build/elf/`. `--download` fetches
the missing ones first:

```bash
uv run python tests/fixtures/elf_snapshots/refresh.py --download
```

To add an image, add it to `MANIFEST` in that script and run it. Then, as after any intended
change to the importer or the drawing, regenerate the goldens and review their diff:

```bash
MEMMAP_UPDATE_GOLDEN=1 uv run pytest tests/test_elf_snapshots.py
```

## The maps

These are the goldens, so they always show what the current code draws.

### nRF52840, Zephyr shell

Two folded groups, each zoomed into, and one "RAM init data" block for the seven RAM sections
that startup code fills from flash.

![nRF52840 Zephyr shell](../tests/golden/elf/nrf52840_zephyr_shell.svg)

### Cortex-R52, Zephyr philosophers

Three folded groups with a detail column each, in one 2 GB memory.

![Cortex-R52 Zephyr philosophers](../tests/golden/elf/r52_zephyr_philosophers.svg)

### FE310, Zephyr shell

RISC-V running from SPI flash, with data in the 16 KB DTIM.

![FE310 Zephyr shell](../tests/golden/elf/fe310_zephyr_shell.svg)

### STM32F412, Tock kernel

A 1-byte `.apps` section far up flash, with free space on both sides.

![STM32F412 Tock kernel](../tests/golden/elf/stm32f412_tock_kernel.svg)

### Apollo4, AmbiqSuite hello world

Three banks: code in MRAM, data in TCM and in shared SRAM. Each bank's initial data is copied
from MRAM in a segment of its own, and each copy's note names the bank it goes to.

![Apollo4 hello world](../tests/golden/elf/apollo4_hello_uart.svg)

### MSP430F2619, hello world

Small sections drawn in place, and the reset vector at 0xFFFE splitting flash in two.

![MSP430F2619 hello world](../tests/golden/elf/msp430f2619_hello.svg)

### STM32F746, Dartino lines

A large `.data` copied from flash, and the stack and heap reserved as a section.

![STM32F746 Dartino lines](../tests/golden/elf/stm32f746_dartino_lines.svg)

### STM32F4 Discovery demo

A conventional STM32 layout.

![STM32F4 Discovery demo](../tests/golden/elf/stm32f4discovery.svg)
79 changes: 79 additions & 0 deletions memmap/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,13 @@
from .config import ConfigError
from .describe import describe_data, describe_text
from .diagnostics import Diagnostic
from .elf import ElfError, read_elf
from .from_elf import ImportProblem, import_elf, parse_region_spec
from .pipeline import Result, run
from .render.raster import RasterUnavailable, convert
from .render.svg import render_svg
from .schema import json_schema
from .units import parse_size

# Process exit codes, documented in AGENTS.md. Success is 0.
EXIT_VALIDATION = 1
Expand Down Expand Up @@ -240,6 +243,82 @@ def describe(config: Path, as_json: bool, sets: tuple[str, ...]) -> None:
click.echo(describe_text(result))


@app.command(
name="from-elf",
epilog=(
"\b\nExample:\n memmap from-elf firmware.elf -o layout.yaml "
"--region flash=0x08000000:64K --region ram=0x20000000:20K"
),
)
@click.argument("elf", type=click.Path(path_type=Path))
@click.option(
"-o", "--output", type=click.Path(path_type=Path), help="Config to write. Default: ELF.yaml."
)
@click.option(
"--region",
"regions",
multiple=True,
metavar="ID=START:SIZE[:NAME]",
help="A memory region, such as flash=0x08000000:64K:Flash (repeatable).",
)
@click.option(
"--min-size",
default="64",
show_default=True,
help="Fold runs of smaller sections into one block; smaller gaps get no Free block.",
)
@click.option("--force", is_flag=True, help="Overwrite the output if it exists.")
@json_option
@strict_option
def from_elf(
elf: Path,
output: Path | None,
regions: tuple[str, ...],
min_size: str,
force: bool,
as_json: bool,
strict: bool,
) -> None:
"""Write a config from the allocated sections of an ELF file.

ELF is a linked executable. Give each memory region with --region, as the ELF does not say
how big flash and RAM are; without any, sections are drawn on the whole address space. The
written config is then checked like `memmap check`.
"""
target = output or elf.with_suffix(".yaml")
if target.resolve() == elf.resolve():
raise _fail(f"output {target} would overwrite the ELF", EXIT_USAGE, as_json)
if target.exists() and not force:
raise _fail(f"{target} exists, use --force to overwrite it", EXIT_USAGE, as_json)
try:
threshold = 0 if min_size.strip() == "0" else parse_size(min_size)
except ValueError as exc:
raise _fail(f"bad --min-size: {exc}", EXIT_USAGE, as_json)
try:
memories = [parse_region_spec(spec) for spec in regions]
except ImportProblem as exc:
raise _fail(str(exc), EXIT_USAGE, as_json)
try:
data = elf.read_bytes()
except OSError as exc:
raise _fail(f"cannot read {elf}: {exc}", EXIT_USAGE, as_json)
try:
imported = import_elf(read_elf(data), elf.name, memories, threshold)
except (ElfError, ImportProblem) as exc:
raise _fail(f"{elf}: {exc}", EXIT_USAGE, as_json)
_write(target, imported.text.encode("utf-8"), as_json)
result = _load(target, (), as_json=as_json)
result.diagnostics[:0] = imported.diagnostics
ok = _passes(result, strict)
if as_json:
_print_json(ok, result, output=str(target))
else:
_print_diagnostics(result, target)
click.echo(str(target))
if not ok:
raise Exit(EXIT_VALIDATION)


@app.command(epilog="\b\nExample:\n memmap schema > memmap.schema.json")
def schema() -> None:
"""Print the JSON Schema of the config."""
Expand Down
Loading
Loading