Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,11 +94,21 @@ itself, such as an unknown flag, is reported as plain text on stderr.
- `labels`: every string drawn, with `role` (address, name, size, note, band, title,
column_title, footer, legend), `column` and an approximate pixel `box`.
- `columns`: placement, plus every option with its value and its `origin`.
- `connectors`: one per zoom, with its `region`, `from` and `to` columns and the points of its
two fan `lines`.

Output is deterministic: the same input gives a byte-identical SVG, so diffing is reliable.

W105 and W106 catch text that does not fit, using a built-in width table, so they are close but
not exact. Connector lines are not checked against text.
not exact. Connector lines are not checked against text, but they are kept out of columns:
a zoom whose detail column is two or more lanes to the right of its source moves that column up
or down so its fan lines bend around the columns between, and W107 reports when it cannot. One
to the left is not moved, and gets W107 when its lines cross a column. Detail columns stacked in
one lane go in the order of the blocks they zoom into, highest address on top. A column that
zooms into columns that stack up taller than it is gets stretched to their height, up to 5
times its own, so its fan lines do not start crammed together. A stack of zooms that would run
past the bottom of the drawing moves up, unless a column in it has an `offset`, and columns
stacked in one lane keep room between them for both end addresses.

## Diagnostics
Every diagnostic has a stable code, a YAML path, a line and usually a `hint`. In JSON it also has
Expand Down Expand Up @@ -131,6 +141,7 @@ Every diagnostic has a stable code, a YAML path, a line and usually a `hint`. In
| W104 | Sector list does not add up to the region size |
| W105 | Block name was split inside a word to fit the column |
| W106 | Text overlaps other text, or is wider than its block |
| W107 | Fan lines run through a column between a zoom's source and its detail column |
| W110 | `from-elf`: section outside every memory region, placed at the top level |
| W111 | `from-elf`: section overlaps another one, left out |
| W112 | `from-elf`: map region left out (no size, past the address space, overlap, too small) |
Expand Down
40 changes: 34 additions & 6 deletions docs/from-elf.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,11 +154,21 @@ Other things a map does not tell memmap:
## From sections to blocks

- **Which sections.** Every allocated section with a size. The thread-local `.tbss` is skipped,
as it only holds a template.
as it only holds a template. A section the linker script reserves without allocating it, such
as the RAM vector table, heap and stacks the Pico SDK declares as `(COPY)`, is drawn too when
it lies inside a memory region and overlaps no allocated section, with the note `not
allocated in the ELF, as a (COPY) section`. One at address 0 is left out, as it looks like a
debug section there. Names lose the blanks IAR leaves around them.
- **Load images.** Initialised data is drawn twice: in RAM where it runs, and in flash where
startup code copies it from. Each loadable segment stored away from where it runs is one flash
block: `.data (load)` with a note such as `copied to RAM at 0x2000 0000`, or `RAM init data`
when it holds several sections, as Zephyr's do, and `code and data` when it includes code.
- **armlink (Keil).** An execution region's RW and ZI data are two sections with one name, drawn
as `RW_IRAM1 (RW)` and `RW_IRAM1 (ZI)`. armlink stores the RW data inside the segment of the
code before it, so its flash copy, `RW_IRAM1 (load)`, is found from the segment rather than
from a segment of its own. When armlink compresses RW data, as it does by default, the section
header gives the compressed size and the ZI data starts further on than alignment explains:
the RW data is then drawn up to the ZI data, and its flash copy is noted as compressed.
- **Free space.** Gaps of at least `--min-size` (default 64 bytes) become Free blocks with their
size, rounded down and marked `≈` when not exact. Smaller gaps are drawn as a split edge.
- **Folding.** In a region of more than 8 blocks, runs of sections smaller than `--min-size`
Expand Down Expand Up @@ -208,7 +218,7 @@ maps through the importer.
| --- | --- | --- |
| Synthetic ELFs | `tests/test_from_elf.py`, built by `tests/elf_builder.py` | One rule per test: folding, free gaps, load images, several memory banks, ids, 64-bit, big-endian, converted images, bad files, the CLI |
| Real builds | `tests/fixtures/firmware*.elf`, from `tests/fixtures/elf_src/` | The reader on files binutils wrote: one firmware as 32- and 64-bit, little- and big-endian, and its GNU ld map. `build.sh` rebuilds them with `arm-none-eabi-gcc` and GNU `objcopy` |
| Real firmware | `tests/test_elf_snapshots.py`, `tests/fixtures/elf_snapshots/` | Eight images from different toolchains, OSes and architectures, against golden configs and drawings |
| Real firmware | `tests/test_elf_snapshots.py`, `tests/fixtures/elf_snapshots/` | Ten images from different toolchains, OSes and architectures, against golden configs and drawings |
| Linker maps | `tests/test_linker.py`, `tests/fixtures/linker_snapshots/` | The map readers on 13 real maps, against `tests/golden/linker/regions.json`, and every case of [the limits](#limits-when-the-map-is-not-enough): lld maps, maps without `MEMORY` or sizes, overlapping regions, sections across a region's edge, files that are not maps |

### Snapshots instead of ELF files
Expand All @@ -228,9 +238,10 @@ clean. The real builds and the synthetic tests cover the reader.

### The images

All come from [Renode](https://github.com/renode/renode), whose board scripts
(`scripts/single-node/*.resc`) load prebuilt firmware from `dl.antmicro.com`. Locally they live
in `build/elf/`, which git ignores.
Eight come from [Renode](https://github.com/renode/renode), whose board scripts
(`scripts/single-node/*.resc`) load prebuilt firmware from `dl.antmicro.com`. The other two are
builds committed to public repositories, pinned to a commit: a Keil one and a Pico SDK one,
whose layouts GNU ld builds do not show. Locally they live in `build/elf/`, which git ignores.

| Snapshot | Firmware | Regions |
| --- | --- | --- |
Expand All @@ -242,9 +253,12 @@ in `build/elf/`, which git ignores.
| `stm32f746_dartino_lines` | Dartino (Dart VM) demo, STM32F746 | flash 1M at 0x08000000, SRAM 320K at 0x20000000 |
| `stm32f4discovery` | STM32F4 Discovery demo, STM32F407 | flash 1M at 0x08000000, SRAM 128K at 0x20000000 |
| `apollo4_hello_uart` | AmbiqSuite hello world, Ambiq Apollo4 (Cortex-M4F) | MRAM 1952K at 0x18000, TCM 384K at 0x10000000, shared SRAM 1M at 0x10060000 |
| `stm32l4_keil_rfe_master` | [iCub](https://github.com/robotology/icub-firmware) RFE board, STM32L4, Keil armlink | flash 512K at 0x08000000, SRAM 160K at 0x20000000 |
| `rp2040_pico_rom` | [picoROM](https://github.com/nickbild/picoROM), RP2040, Pico SDK | flash 2M at 0x10000000, RAM 256K at 0x20000000, SCRATCH_X and SCRATCH_Y 4K each |

Only the ELF files are published, so the regions are `--region` specs: the chips' real
flash and RAM, from datasheets and Renode's platform files (`platforms/cpus/*.repl`). Where the
flash and RAM, from datasheets and Renode's platform files (`platforms/cpus/*.repl`), and for
the two repository builds the regions their committed linker maps give. Where the
firmware starts above the bottom of its memory, the region starts there too: the FE310 shell
runs from the user part of SPI flash, and the Apollo4 application starts 96 KB into its 2 MB
MRAM. The Tock kernel is linked by lld, so its regions would have to be given this way anyway.
Expand Down Expand Up @@ -343,3 +357,17 @@ A large `.data` copied from flash, and the stack and heap reserved as a section.
A conventional STM32 layout.

![STM32F4 Discovery demo](../tests/golden/elf/stm32f4discovery.svg)

### STM32L4, Keil armlink

RW and ZI data under one name, drawn as `(RW)` and `(ZI)`, and the RW data stored compressed
after the code: 212 bytes in flash that unpack to 460 in SRAM.

![STM32L4 Keil RFE](../tests/golden/elf/stm32l4_keil_rfe_master.svg)

### RP2040, Pico SDK

The RAM vector table, heap and core 0 stack are (COPY) sections, drawn with a note as they are
not allocated in the ELF.

![RP2040 picoROM](../tests/golden/elf/rp2040_pico_rom.svg)
6 changes: 5 additions & 1 deletion memmap/from_elf/elf.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,11 @@
from dataclasses import dataclass

# Section header types and flags used by the importer.
SHT_PROGBITS = 1
SHT_NOBITS = 8
SHF_WRITE = 0x1
SHF_ALLOC = 0x2
SHF_EXECINSTR = 0x4
SHF_TLS = 0x400
PT_LOAD = 1

Expand Down Expand Up @@ -42,6 +45,7 @@ class Section:
address: int
offset: int
size: int
align: int = 1

@property
def allocated(self) -> bool:
Expand Down Expand Up @@ -138,7 +142,7 @@ def read_elf(data: bytes) -> Elf:
names = data[table[4] : table[4] + table[5]]

sections = [
Section(_name(names, raw[0]), raw[1], raw[2], raw[3], raw[4], raw[5])
Section(_name(names, raw[0]), raw[1], raw[2], raw[3], raw[4], raw[5], max(raw[8], 1))
for raw in raw_sections[1:] # section 0 is always empty
]
# An unlinked object file has every section at address 0. One with real addresses, as
Expand Down
137 changes: 124 additions & 13 deletions memmap/from_elf/importer.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,24 @@
from ..diagnostics import Diagnostic
from ..units import diagnostic_address, format_address
from .blocks import Bucket, Item
from .elf import PT_LOAD, SHF_TLS, Elf, Section
from .elf import (
PT_LOAD,
SHF_EXECINSTR,
SHF_TLS,
SHF_WRITE,
SHT_NOBITS,
SHT_PROGBITS,
Elf,
Section,
Segment,
)
from .regions import ImportProblem, LeftOut, MemorySpec, check_memories, unique_id
from .sizes import address_literal, floor_size
from .writer import write_yaml

_NOTE_NAMES = 6 # section names a note lists before "+N more"
FOLD_OVER = 8 # a region with more blocks than this folds its runs of small sections
_SHF_EXECINSTR = 0x4
UNALLOCATED_NOTE = "not allocated in the ELF, as a (COPY) section"


@dataclass
Expand Down Expand Up @@ -66,6 +76,52 @@ def _name_list(names: list[str]) -> str:
return f"{shown}, +{extra} more" if extra > 0 else shown


def _align_up(value: int, align: int) -> int:
"""`value` rounded up to a multiple of `align`."""
return -(-value // align) * align


def _rw_zi_pairs(sections: list[Section]) -> list[tuple[Section, Section]]:
"""Pairs of a writable PROGBITS and a NOBITS section with the same name, the RW and ZI
parts of one armlink execution region, such as two sections called RW_IRAM1."""
by_name: dict[str, list[Section]] = {}
for section in sections:
by_name.setdefault(section.name.strip(), []).append(section)
pairs = []
for group in by_name.values():
data = [s for s in group if not s.nobits]
zero = [s for s in group if s.nobits]
if len(data) == 1 and len(zero) == 1 and data[0].flags & SHF_WRITE:
pairs.append((data[0], zero[0]))
return pairs


def _display_names(sections: list[Section]) -> dict[Section, str]:
"""The name each section is drawn with: its own without surrounding blanks, which IAR
leaves in, and with (RW) or (ZI) when an armlink region has both under one name."""
names = {section: section.name.strip() for section in sections}
for data, zero in _rw_zi_pairs(sections):
names[data] += " (RW)"
names[zero] += " (ZI)"
return names


def _unpacked_sizes(sections: list[Section]) -> dict[Section, int]:
"""RAM sizes of RW data that armlink stored compressed. Its section header then gives the
compressed size, and the ZI data of the same region starts further on than alignment
explains: the RW data runs up to it."""
sizes = {}
for data, zero in _rw_zi_pairs(sections):
packed_end = data.address + data.size
if zero.address <= _align_up(packed_end, zero.align):
continue
others = [other for other in sections if other is not data]
if any(packed_end <= other.address < zero.address for other in others):
continue
sizes[data] = zero.address - data.address
return sizes


class _Importer:
"""Places the sections of one ELF into buckets, collecting diagnostics as it goes."""

Expand Down Expand Up @@ -113,21 +169,51 @@ def place(self) -> None:
"""Put every wanted section in a bucket where it runs, and each load image where it is
stored."""
wanted = [section for section in self.elf.sections if _wanted(section)]
names = _display_names(wanted)
unpacked = _unpacked_sizes(wanted)
for section in wanted:
item = Item(self.ids.make(section.name), section.name, section.address, section.size)
name = names[section]
size = unpacked.get(section, section.size)
item = Item(self.ids.make(name), name, section.address, size)
self.bucket_for(item).items.append(item)
for item in self.unallocated(wanted):
self.bucket_for(item).items.append(item)
for image in self.load_images(wanted):
for image in self.load_images(wanted, set(unpacked)):
self.bucket_for(image).items.append(image)

def load_images(self, sections: list[Section]) -> list[Item]:
def unallocated(self, wanted: list[Section]) -> list[Item]:
"""Sections the linker script reserved space for without allocating them, such as the
heap and stacks the Pico SDK declares as (COPY): each with an address and a size that
lie inside one memory region and overlap no allocated section. Debug and other
non-memory sections sit at address 0, so a (COPY) section there is not told apart from
them and is left out."""
items = []
for section in self.elf.sections:
if section.allocated or not section.size or not section.address:
continue
if section.kind not in (SHT_PROGBITS, SHT_NOBITS):
continue
end = section.address + section.size - 1
memory = self.memory_at(section.address)
if memory is None or end > memory.end:
continue
if any(w.address <= end and section.address < w.address + w.size for w in wanted):
continue
name = section.name.strip()
items.append(
Item(self.ids.make(name), name, section.address, section.size, UNALLOCATED_NOTE)
)
return items

def load_images(self, sections: list[Section], compressed: set[Section]) -> list[Item]:
"""One block per loadable segment that is stored somewhere other than where it runs,
such as the initial values of .data. Startup code copies each in one piece."""
such as the initial values of .data. Startup code copies each in one piece. armlink
instead puts RW data in the segment of the code before it, so a segment loaded where
it runs can still hold sections stored apart, which get a block each."""
images = []
for segment in self.elf.segments:
if segment.kind != PT_LOAD or segment.file_size == 0:
continue
if segment.physical == segment.virtual:
continue
members = [
section
for section in sections
Expand All @@ -136,12 +222,13 @@ def load_images(self, sections: list[Section]) -> list[Item]:
]
if not members:
continue
if segment.physical == segment.virtual:
images += self.stored_apart(segment, members, compressed)
continue
target = format_address(segment.virtual, self.elf.bits // 4)
names = [section.name for section in members]
names = [section.name.strip() for section in members]
if len(members) == 1: # with several banks, a bare address leaves the reader to look
bank = self.memory_at(segment.virtual)
where = f"{bank.name} at {target}" if bank else target
name, note = f"{names[0]} (load)", f"copied to {where}"
name, note = f"{names[0]} (load)", f"copied to {self.bank_at(segment.virtual)}"
else:
name, note = self.image_name(segment.virtual, members), f"copied to {target}: "
note += _name_list(names)
Expand All @@ -150,14 +237,38 @@ def load_images(self, sections: list[Section]) -> list[Item]:
)
return images

def stored_apart(
self, segment: Segment, members: list[Section], compressed: set[Section]
) -> list[Item]:
"""The sections of a segment loaded where it runs whose bytes it loads somewhere other
than their own address, as armlink does with RW data. `compressed` sections are stored
packed, so their copy is smaller than the RAM they unpack to."""
images = []
for section in members:
stored = segment.physical + section.offset - segment.offset
if stored == section.address:
continue
size = min(section.size, segment.offset + segment.file_size - section.offset)
name = f"{section.name.strip()} (load)"
where = self.bank_at(section.address)
verb = "compressed, unpacked to" if section in compressed else "copied to"
images.append(Item(self.ids.make(name), name, stored, size, note=f"{verb} {where}"))
return images

def bank_at(self, address: int) -> str:
"""Where a load image is copied to: the memory region and address, or the address."""
target = format_address(address, self.elf.bits // 4)
bank = self.memory_at(address)
return f"{bank.name} at {target}" if bank else target

def memory_at(self, address: int) -> MemorySpec | None:
"""The memory region that holds `address`, if any."""
return next((m for m in self.memories if m.start <= address <= m.end), None)

def image_name(self, address: int, members: list[Section]) -> str:
"""A name for a load image of several sections, after where it is copied to and what
it holds: RAM init data, or DTIM code and data when it includes code."""
what = "code and data" if any(s.flags & _SHF_EXECINSTR for s in members) else "init data"
what = "code and data" if any(s.flags & SHF_EXECINSTR for s in members) else "init data"
target = self.memory_at(address)
return f"{target.name} {what}" if target else what.capitalize()

Expand Down
Loading
Loading