[davidschachterADFA]

Summary

Fix 5 of ADFA-6757, kept apart from #149 and #150 because it is the only one that changes native code and the prebuilt AAR.

The shipped libggml-cpu.so holds no dot-product (sdot) instructions. llama-impl/build.gradle.kts sets no ARM level, so the cross-compile falls back to basic ARMv8, and most phones sold in the last several years support dot-product. The model's integer maths for reading the prompt is where it matters.

• The CPU backend is now built as four libraries, libggml-cpu-android_armv8.0_1, _armv8.2_1 (dot-product), _armv8.2_2 (dot-product and fp16) and _armv8.6_1 (adds int8 matrix multiply), with -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON. The fork's existing Android variant list does the work. Checked with llvm-objdump: armv8.0_1 has 0 sdot, armv8.2_* have 418, armv8.6_1 also has 116 smmla.
• llama-android.cpp loads the best one at start-up (load_cpu_backend, called from backend_init). It tries the variants by name, best first; ggml refuses one that scores zero on the phone, and each needs everything the next one does, so the first to load is the best supported. Bare names let the linker find them in the IDE's extracted plugin directory and inside an APK alike. It runs once, logs which variant loaded, and logs dlerror's reason for any that did not.
• ARMv8.0 phones keep working. The ARMv8.0 build scores above zero on every arm64 phone. I did not hard-code armv8.2-a+dotprod, as the ticket says: a Cortex-A53 phone would crash on its first instruction.
• The download grows by 1.8 MB. The release .cgp goes from 15.66 MB to 17.47 MB (+11.6%); the AAR, where the libraries are compressed, from 2.0 MB to 2.9 MB. PrebuiltAarAbiTest now requires all four variants instead of libggml-cpu.so, so an AAR that dropped one fails a test. It still checks the source hash and 16 KB alignment, and both pass.

Measured on a Pixel 6 Pro (Android 17)

Same phone, same AI-Core (de06201, the compact tool set from #150), qwen2.5-coder-1.5b-instruct-q4_k_m, a fresh chat per prompt, one run each. Baseline is AI-Agent-Local built from main; the other is this branch. Time is from the log line announcing the prompt read to the first parsed output, so it includes writing the reply's first tokens.

Prompt	main	This branch
"Hello" (1,237-token prompt)	81.2 s	64.2 s (-21%)
"Run the unit tests" (1,240 tokens)	84.4 s	68.8 s (-18%)
"List the files in the project" (366 tokens read, 1,246 cached)	83.1 s	69.2 s (-17%)

• The phone's CPU reports asimddp and fphp. The log shows armv8.2_2 was the one picked, and System Info: ... DOTPROD = 1 (the main build shows none). These timings are from 332d7a0; after the review fix (88c9cac) one more "Hello" run took 60.1 s with the same variant.
• The plugin loader extracted all nine native libraries to the plugin's directory, so the variants are found.
• With the other three variants deleted from that directory, the ARMv8.0 variant loaded (System Info shows no DOTPROD) and answered "Hello" in 82.6 s, the same as main. Repeated on 88c9cac: each missing variant logs dlopen failed: library ... not found, then ARMv8.0 loads. It shows the baseline build works; it is a subset of the others, so this phone cannot show that a real ARMv8.0 CPU picks it.
• Reading only 4 tokens from cache on the first turn is fix 1 (ADFA-6757 | Keep the local prompt cache across chat titles; size generation threads to the fast cores #149), which these builds don't include.

Review

An independent review (given the diff and the artifacts, not my conclusions) found four issues; I confirmed each and fixed all four in 88c9cac:

• The loader failed when the libraries stay inside an APK (medium). The first version listed the directory holding libllama-android.so, and this plugin's own APK sets extractNativeLibs="false", so NativeContextFallbackTest (connectedDebugAndroidTest) found no backend: on the Pixel it failed with load_model() failed and backends: none loaded from …/base.apk!/lib/arm64-v8a. The installed plugin was not affected, because CoGo extracts the libraries. Variants are now loaded by name, and the test passes on the Pixel.
• The size figure quoted the AAR, not the download (low). Corrected above and in NATIVE-BUILD.md.
• A variant that failed to load gave no reason (low): release ggml swallows dlopen errors. The loader now logs dlerror.
• The test required only two of the four variants (low). It now requires all four.

It also confirmed: no variant can execute an instruction the CPU lacks before its score is checked (the scoring code is plain ARMv8.0); nothing assumes a linked-in CPU backend; all nine libraries are 16 KB aligned; and a rebuild from the submodule and committed flags gives the same four variants.

Not verified

• No Pixel 7, no ARMv8.0 phone. The ticket wants a before-and-after llama-bench on the Pixel 7 and a run on an ARMv8.0 phone. I had neither, so the first is replaced by the in-app Pixel 6 Pro numbers above, and the second by the single-variant test.
• The armv8.6_1 (int8 matrix multiply) variant never ran. The Pixel 6 Pro lacks that feature; its library loaded and scored without crashing, but no phone here exercised it.
• One run per row. The gains are in the same range as the ticket's llama-bench estimate (1.2 to 1.5x on reading), smaller as the prompt grows.
• Tested only on a debug build on the phone. The release .cgp builds clean (revision=88c9cacbfeb7, revision_source=git, nine arm64 libraries and nothing else), but I didn't install it.
• The ARMv8.6 variant carries a few ARMv8.3/8.4 load-acquire instructions its score does not check. No phone CPU is known to have int8 matrix multiply without them, so this is theoretical.
• The AAR was built on a laptop with NDK r27 (27.0.12077973), as NATIVE-BUILD.md already says is not ideal.

Things a reviewer should know

• The binary AAR is in the diff, so review the .cpp and build.gradle.kts changes and the AAR contents (unzip -l, then llvm-objdump -d on each variant, as NATIVE-BUILD.md now describes).
• Every variant is opened once at start-up so it can score itself. SELinux logs avc: granted { execute } audit lines for them; they are not denials.
• This branch is based on main and does not touch ADFA-6757 | Keep the local prompt cache across chat titles; size generation threads to the fast cores #149 or ADFA-6757 | Offer a small on-device model a compact tool set #150's files.

Test plan

• AI-Agent-Local testDebugUnitTest: 243 tests, 0 failures
• assemblePlugin (release, clean tree, revision_source=git); uv run --directory tools/addons addons --root "$PWD" check exits 0
• connectedDebugAndroidTest (NativeContextFallbackTest) on the Pixel 6 Pro: passes (it failed before 88c9cac)
• Same-phone A/B above; the ARMv8.0 variant alone
• A Pixel 7 llama-bench, an ARMv8.0 phone, and a phone with int8 matrix multiply

🤖 Generated with Claude Code

https://claude.ai/code/session_01Vh2DAaJxvRXKRYkgiDqwGQ