Skip to content

hi3516cv6xx: room to run and to update on 64 MB -- no vendor libraries, 128 KiB blocks, sysupgrade into the media zone - #2734

Merged
widgetii merged 4 commits into
masterfrom
hi3516cv6xx-memory
Oct 31, 2026
Merged

widgetii merged 4 commits into
masterfrom
hi3516cv6xx-memory

Conversation

@widgetii

Copy link
Copy Markdown
Member

Problem

On a 64 MB hi3516cv6xx (hi3516cv608 / cv610) the lite image leaves Linux too little memory to run, let alone update itself (OpenIPC/majestic#355).

  • No room to run. On the lite image the streamer ran with 3.3-4.4 MB available, and the kernel OOM-killed it in ordinary use.
  • Where it went. Two causes: 4.2 MB of the kernel's squashfs caches, held for good because lite was built with 1 MiB blocks, and 44 vendor MPP libraries (3.7 MB) the streamer named whether it used them or not.
  • sysupgrade cannot fit. It unpacks the image into /tmp, which is RAM, and cannot fit it:
    • on the 10-27 lite image it refused in every state measured;
    • where it did pass that check, the rootfs peek's mount of the new image (4 MB more at 1 MiB blocks) then thrashed until power was cut;
    • from the stock lite image it was OOM-killed in its own preliminaries.

What

Three commits:

  1. No vendor MPP libraries. majestic master now links the gen 5 SDK from its static archives and carries only what it reaches, so the 44 shared objects no longer ship. Nothing else in the image linked any of them: the sensor drivers declare only libc and take their callbacks from majestic. protobuf-c goes too; both variants carried it only because the old majestic named it.
  2. lite back to the 128 KiB default block. The 1 MiB block (hi3516cv6xx: add the lite variant, and drop the digital TV stack to f… #2447) bought ~430 KiB when the rootfs had to fit a fixed 5120 KiB slot. The split image (hi3516cv6xx, hi3519dv500: the split NOR image, as on the gk7205v200 family #2638) has none.
  3. sysupgrade unpacks into the idle media zone where hisi_mem carves it out (ot). phram maps the zone's free part, and jffs2 on it is a filesystem that costs the OS nothing. Every later step reads files from it as from /tmp, and it is the first choice where available. CONFIG_MTD_PHRAM=m on hi3516cv6xx. sysupgrade 1.0.81.

Hardware tested on

hi3516cv608 + SP2308, 64 MB DRAM, 16 MB NOR, with the lite image from this branch. It ran on the default layout: 32 MB for the OS, U-Boot env untouched.

Not booted: ultimate. It shares commit 1 (the libraries and protobuf-c), but this board ran only the lite image.

Evidence

Image size (make BOARD=hi3516cv6xx_lite):

before (master):  firmware.bin: [7360KB/7616KB]   squashfs 1 MiB blocks
after:            firmware.bin: [6592KB/7616KB]   squashfs 128 KiB blocks

Memory with the streamer running, same board, same 32 MB layout:

                      master lite        this branch
MemAvailable          3.3-4.4 MB         8.4 MB
kmalloc-4k objects    1048 (4.2 MB)      184
SUnreclaim            9.4 MB             5.7 MB
rootfs                5392 KB            4514 KB

On master lite the streamer was OOM-killed in normal running:

Out of memory: Killed process 1447 (majestic) total-vm:12896kB, anon-rss:2204kB

On this branch the streamer starts with no vendor library on disk or mapped. The sensor comes up through the callbacks it exports, and a snapshot returns 696855 bytes:

== sp2308 24Mclk 1080P30fps(MIPI) 2lane 10bit linear init success! ==

Before, the same upgrade from a stock lite image:

Stop services, sync files, free up memory
Killed

After, the image upgrading itself (sysupgrade --url=<same image> -k -r, serial console):

Short of memory by 6523 KB with majestic running: stopping it, which gives back what it holds in RAM.
Unpacking into the idle media zone (28 MB at 0x42400000) instead of RAM.
Received and unpacked
RAM left to flash from: 6748 KB
Handing PID 1 over to the RAM root.
Writing /tmp/stage/firmware.bin.hi3516cv6xx at 0x50000
Settings staged at 0x6c0000 for the first boot
Firmware updated
...
Found RootFS at 0x00250000 (4514k)
Settings restored.

After the reboot the board was still claimed, SSH key login worked, and sysupgrade --history recorded the upgrade.

Checks run:

  • bash .github/scripts/test_sysupgrade.sh: all checks pass.
  • STRICT=1 test_shell_parse.sh: clean.
  • STRICT=1 test_strip_shell_comments.sh: clean.
  • ci-matrix.py --self-test: ok.

Scope

  • No kernel patches under general/package/all-patches/linux/ (those go to OpenIPC/linux)
  • No files specific to a single retail camera model (those go to OpenIPC/builder)
  • No probing or bring-up tooling (that goes to OpenIPC/ipctool)
  • Nothing under general/overlay/ or in a shared load_<vendor> script hardcodes a value specific to my board
  • Package sources come from an OpenIPC repository, and any version bump keeps at least the specificity of the pin it replaces (a new package should pin a full 40-character SHA)
  • No LD_PRELOAD, and no binaries that cannot be rebuilt from source
  • New code is selected by a defconfig, so CI actually builds it

…tically

majestic for the gen 5 SDK now links the vendor libraries from their
static archives and keeps only the code it reaches; nothing else in the
image links any of them (the sensor drivers declare only libc and take
the ISP/AE/AWB callbacks and securec from majestic). The 44 shared
objects, 3.7 MB before compression, no longer ship, and protobuf-c, which
both variants carried only because the old majestic named it, goes too.
1 MiB blocks (#2447) saved ~430 KiB when the rootfs had to fit a fixed
5120 KiB slot; the split image (#2638) has none. The kernel holds its
squashfs caches in whole blocks for as long as root is mounted: 4.2 MiB
of unreclaimable slab at 1 MiB on a 64 MiB hi3516cv608 (1040 kmalloc-4k
pages from squashfs_cache_init), ~0.6 MiB at 128 KiB.
On a HiSilicon part whose media zone hisi_mem carves out of the OS's
sight ("ot"), the zone sits idle once majestic stops and is far larger
than an image, while the OS has 26-30 MB on a 64 MB board. The image is
now unpacked there: phram maps the zone's free part as an MTD, jffs2
makes it a filesystem, and every step after the unpack -- checksums, the
rootfs peek, the write -- reads files from it as it would from /tmp.

It is the first choice where possible, not a fallback: passing the RAM
check is not the end of what a run needs. The rootfs peek mounts the new
image, and its squashfs caches come out of the same RAM -- on a
hi3516cv608 with 36 MB for the OS, a run to a 1 MiB-block image passed
the check and then thrashed at that mount until power was cut.

- The start of the zone, where the kernel modules keep a few hundred KB,
  is left alone, and nothing is mapped unless every block they hold ends
  below the mapping: the zone's allocator does not know about it. A
  SIGKILLed majestic leaves its buffers there, and the run then declines
  the zone and unpacks to /tmp as before.
- The stage is exported across the PID 1 hand-off and read only by it.
- RAM-backed MTDs are left out of the flash geometry.
- Both refusal paths unmap it before majestic is restarted.
- CONFIG_MTD_PHRAM=m on hi3516cv6xx.

Measured on a hi3516cv608 + SP2308: from the image that thrashed, the
same upgrade staged 24 MB of the zone, flashed and rebooted with the
settings kept. On a lite image at 32 MB for the OS with 1 MiB blocks the
run was OOM-killed in its preliminaries before it reached the stage, so
it needs the RAM the rest of this change frees.
@qodo-free-for-open-source-projects

Copy link
Copy Markdown

PR Summary by Qodo

Make hi3516cv6xx lite runnable and upgradable on 64 MB boards

🐞 Bug fix ✨ Enhancement ⚙️ Configuration changes 🕐 40+ Minutes

Grey Divider

AI Description

• Remove unused vendor libraries and protobuf-c from hi3516cv6xx images to reduce memory and image
 size.
• Restore 128 KiB squashfs blocks on lite to reduce unreclaimable kernel cache.
• Stage upgrades in idle media memory when available, preserving the existing RAM fallback.
Diagram

graph TD
  Stop["Stop majestic"] --> Choice{"Media zone usable?"} --> Phram["phram mapping"] --> Stage["JFFS2 stage"] --> Verify["Verify images"] --> Pivot["RAM root handoff"] --> Flash["Flash firmware"]
  Choice --> Tmp["RAM stage"] --> Verify
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Keep upgrades in /tmp and reduce memory use further
  • ➕ Avoids mapping media memory and changing upgrade staging.
  • ➖ Still competes with the streamer, upgrade tools, and rootfs verification for scarce OS RAM.
2. Stream the download directly to flash
  • ➕ Avoids storing a complete unpacked image.
  • ➖ Risks writing firmware before checksum, compatibility, and rootfs checks finish; complicates recovery from interrupted downloads.

Recommendation: Prefer the proposed media-zone stage with its guarded /tmp fallback: it retains pre-flash validation while moving the image out of scarce OS RAM. Review the allocation-boundary checks and stage cleanup particularly carefully because phram operates outside the media allocator's awareness.

Files changed (5) +125 / -65

Enhancement (1) +8 / -37
hisilicon-osdrv-hi3516cv6xx.mkStop installing vendor MPP shared libraries +8/-37

Stop installing vendor MPP shared libraries

• Removes the 44-library install list and installation step because the published gen 5 majestic build links the required SDK code statically. Retains installation of load scripts, allocator setup, and sensor assets.

general/package/hisilicon-osdrv-hi3516cv6xx/hisilicon-osdrv-hi3516cv6xx.mk

Bug fix (1) +110 / -14
sysupgradeStage upgrades in idle HiSilicon media memory +110/-14

Stage upgrades in idle HiSilicon media memory

• Bumps sysupgrade to 1.0.81 and adds guarded phram/JFFS2 staging with a /tmp fallback. Routes unpacking, checksums, image inspection, and flashing through the selected stage, carries its path across the RAM-root handoff, excludes the RAM-backed MTD from flash geometry, and releases the stage before restarting majestic on non-reboot paths.

general/overlay/usr/sbin/sysupgrade

Other (3) +7 / -14
hi3516cv6xx.generic.configBuild phram as a module for hi3516cv6xx +1/-1

Build phram as a module for hi3516cv6xx

• Enables the kernel phram module so sysupgrade can expose free carved-out media memory as an MTD device.

br-ext-chip-hisilicon/board/hi3516cv6xx/hi3516cv6xx.generic.config

hi3516cv6xx_lite_defconfigRestore smaller squashfs blocks and remove protobuf-c +6/-8

Restore smaller squashfs blocks and remove protobuf-c

• Drops the 1 MiB squashfs block override, restoring the 128 KiB default to reduce unreclaimable cache. Removes protobuf-c, which the new majestic build no longer needs.

br-ext-chip-hisilicon/configs/hi3516cv6xx_lite_defconfig

hi3516cv6xx_ultimate_defconfigRemove unnecessary protobuf-c from ultimate +0/-5

Remove unnecessary protobuf-c from ultimate

• Stops selecting protobuf-c now that the hi3516cv6xx majestic build no longer requires it. This variant was not boot-tested in the reported hardware checks.

br-ext-chip-hisilicon/configs/hi3516cv6xx_ultimate_defconfig

@qodo-free-for-open-source-projects

qodo-free-for-open-source-projects Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Grey Divider


Action required

1. Live media buffers can be overwritten ✓ Resolved
Description
stage_in_mmz compares the highest existing media buffer's physical start with the staging
boundary, but never checks where that buffer ends. If a buffer starts below the 4 MiB skip and
extends across it, the check accepts it and phram maps the same physical memory for archive
extraction.
Code

general/overlay/usr/sbin/sysupgrade[R1573-1575]

+	top=$(awk -F'[(), ]+' '/MMB: phys/ { for (i = 1; i < NF; i++) if ($i == "phys") print $(i + 2) }' /proc/umap/media-mem 2>&3 |
+		while read -r e; do echo $((e)); done | sort -n | tail -1)
+	[ -n "$top" ] && [ "$top" -ge "$start" ] && return 1
Evidence
The code promises to leave every held block below the boundary, but extracts only phys and rejects
only starts at or above start; it then maps the entire region beginning at start.

general/overlay/usr/sbin/sysupgrade[1553-1555]
general/overlay/usr/sbin/sysupgrade[1568-1585]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The media-zone guard checks allocation start addresses rather than their ends, allowing staging to overlap a live buffer.
## Fix Focus Areas
- general/overlay/usr/sbin/sysupgrade[1573-1585]
## Recommended Fix
Parse and validate each allocation's end address, reject staging if any allocation overlaps the proposed mapping, and fail closed if occupancy cannot be read reliably.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. Some RAM pivots cannot find the image ✓ Resolved
Description
stage_dir is restored from handoff_stage_dir only when _handoff=1, although enter_ramfs also
re-executes the script without a handoff. When that non-handoff pivot follows media-zone staging,
the image remains under /tmp/stage while the flash phase resets its lookup path to /tmp.
Code

general/overlay/usr/sbin/sysupgrade[R113-114]

+stage_dir=/tmp
+[ "1" = "$_handoff" ] && [ -n "$handoff_stage_dir" ] && stage_dir=$handoff_stage_dir
Evidence
The stage is mounted under /tmp/stage and exported, and /tmp is moved into the RAM root; the
non-handoff path then re-executes without setting _handoff, so the new initialization loses the
stage path before image lookup.

general/overlay/usr/sbin/sysupgrade[113-114]
general/overlay/usr/sbin/sysupgrade[1587-1594]
general/overlay/usr/sbin/sysupgrade[2889-2891]
general/overlay/usr/sbin/sysupgrade[2920-2928]
general/overlay/usr/sbin/sysupgrade[4177-4189]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The non-handoff RAM pivot loses the media-zone staging path when it re-executes sysupgrade.
## Fix Focus Areas
- general/overlay/usr/sbin/sysupgrade[113-114]
- general/overlay/usr/sbin/sysupgrade[1592-1594]
- general/overlay/usr/sbin/sysupgrade[2920-2928]
## Recommended Fix
Restore the exported stage path in every RAM-root second phase, including the non-handoff re-exec, while retaining `/tmp` as the default when no media-zone stage was created.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

3. Local archives can fail a false RAM check ✓ Resolved
Description
check_unpack_room still compares the uncompressed archive size with free space on /tmp, even
when stage_in_mmz has selected a separate filesystem for extraction. On a local --archive
upgrade with less free /tmp space than the extracted image, it rejects the upgrade although only
the archive, not its extracted contents, needs to fit there.
Code

general/overlay/usr/sbin/sysupgrade[1956]

+		gzip -d "$archive" -c | tar xvf - -C "$stage_dir" $(tar_excludes) >"$unpacked_list" && echo_c 32 "Local archive unpacked" || die "Cannot extract $archive"
Evidence
The existing check calculates the unpacked size and tests df -k /tmp; the changed extraction
target is $stage_dir, which staging sets to a mounted JFFS2 filesystem at /tmp/stage.

general/overlay/usr/sbin/sysupgrade[1358-1378]
general/overlay/usr/sbin/sysupgrade[1585-1594]
general/overlay/usr/sbin/sysupgrade[1953-1956]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The local-archive capacity check assumes extraction consumes `/tmp` even when extraction goes to the media-zone filesystem.
## Fix Focus Areas
- general/overlay/usr/sbin/sysupgrade[1358-1378]
- general/overlay/usr/sbin/sysupgrade[1953-1956]
## Recommended Fix
Keep the existing `/tmp` double-copy check only when extracting to `/tmp`; when media-zone staging is active, check the actual destination and any RAM needed for unpacking separately.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


4. Failed upgrades can remove another mapping ✓ Resolved
Description
release_mmz_stage unloads phram whenever the module is loaded, without checking whether this
upgrade created its mapping. Because die calls that cleanup even before staging starts, a failed
upgrade can remove an unrelated existing phram device; the staging path also unloads a mapping
unless its MTD name is sysupgrade.
Code

general/overlay/usr/sbin/sysupgrade[R1603-1604]

+	rmdir /tmp/stage 2>&3
+	grep -q '^phram ' /proc/modules 2>&3 && rmmod phram 2>&3
Evidence
Cleanup is invoked on pre-flash errors regardless of staging state, while its rmmod check tests
module presence rather than ownership; staging itself attempts to remove an already-loaded module
with any other mapping name.

general/overlay/usr/sbin/sysupgrade[145-164]
general/overlay/usr/sbin/sysupgrade[1578-1585]
general/overlay/usr/sbin/sysupgrade[1598-1606]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Cleanup and staging can unload a phram mapping that sysupgrade did not create.
## Fix Focus Areas
- general/overlay/usr/sbin/sysupgrade[1578-1585]
- general/overlay/usr/sbin/sysupgrade[1598-1606]
## Recommended Fix
Track whether this invocation loaded phram and unload it only in that case. If an unrelated mapping already exists, leave it intact and decline media-zone staging rather than replacing it.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Tip of the day
💡 Did you know, you can keep summaries lean with Findings visible per group, which tucks the rest behind a View link

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread general/overlay/usr/sbin/sysupgrade Resolved
Comment thread general/overlay/usr/sbin/sysupgrade Resolved
Comment thread general/overlay/usr/sbin/sysupgrade Resolved
Comment thread general/overlay/usr/sbin/sysupgrade Resolved
- Stage in the media zone only when /proc/umap/media-mem reports its
  totals, so a format it cannot parse never maps over live buffers.
- Refuse to stage, or to unload, a phram that was loaded before us.
- Keep stage_dir across the hand-off only while its jffs2 is mounted.
- Check room for a local archive in the stage directory it unpacks into.
@widgetii
widgetii merged commit b34d2aa into master Oct 31, 2026
166 of 175 checks passed
@widgetii
widgetii deleted the hi3516cv6xx-memory branch October 31, 2026 07:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant