[Feature] - Prepare MiniExcel V2 for a Rust native backend with multi-platform NuGet packages

[shps951023]

Feature Summary

Prepare MiniExcel V2 to use the MiniExcel Rust engine through an in-process C ABI, with platform-specific native packages distributed using a model similar to npm packages that publish one JavaScript facade plus optional packages for each operating system and architecture.

The immediate goal is to turn the existing Windows x64 proof of concept into a production-ready, opt-in backend. Replacing the managed OpenXML implementation by default should happen only after compatibility, deployment, and performance gates pass.

Document Formats

• XLSX
• CSV
• XLSM
• Other

Motivation

The existing MiniExcel.Rust NuGet proof of concept already validates the core architecture: managed C# calls a Rust cdylib through a versioned C ABI, reads rows in bounded batches, and preserves deterministic disposal.

On the 100,000-row benchmark, the Rust-backed path showed:

• 3.49x faster cold query execution.
• 79.1% lower cold first-row latency.
• 78.7% lower managed allocation.
• 32.8% lower cold peak working set.
• 2.80x faster warm sustained query execution.

However, the current package contains only runtimes/win-x64/native/miniexcel_ffi.dll. MiniExcel V2 needs a repeatable multi-platform build, packaging, restore, and test strategy before the Rust backend can become a supported implementation.

Proposed Solution

1. Define the package architecture

Keep a stable managed facade and distribute native binaries per .NET Runtime Identifier (RID), analogous to npm's main package plus platform-specific optional packages.

Target matrix:

• win-x64
• win-arm64
• linux-x64
• linux-arm64
• linux-musl-x64
• linux-musl-arm64
• osx-x64
• osx-arm64

Evaluate and document the NuGet-compatible design before implementation:

• Preferred: MiniExcel.Rust as the managed facade plus RID-specific packages such as MiniExcel.Rust.Native.win-x64, with MSBuild/NuGet logic selecting the correct native asset.
• Fallback: one MiniExcel.Rust package containing all assets under runtimes/{rid}/native/ if reliable RID-specific dependency selection cannot be achieved without fragile custom restore behavior.

The selected design must avoid loading an asset for the wrong architecture and should avoid downloading every native binary where NuGet permits it.

2. Build the native libraries in MiniExcel-Rust

Add reproducible release builds for the Rust FFI library and produce the correct native filename per platform:

• Windows: miniexcel_ffi.dll
• Linux GNU/musl: libminiexcel_ffi.so
• macOS: libminiexcel_ffi.dylib

Pin the Rust MSRV and lockfile, strip release binaries where appropriate, generate SHA-256 checksums, and retain build provenance for every artifact.

3. Stabilize the C ABI and managed interop layer

• Keep the ABI version explicit and reject incompatible native libraries with an actionable error.
• Define ownership for handles, buffers, strings, errors, and cancellation.
• Ensure all native resources are released on completion, early enumeration disposal, exceptions, and cancellation.
• Keep data transfer batched to minimize P/Invoke overhead and managed allocation.
• Add clear diagnostics for unsupported RIDs, missing native assets, load failures, and ABI mismatches.

4. Integrate with the MiniExcel V2 API

• Preserve the public MiniExcel V2 importer API instead of exposing Rust implementation details to callers.
• Introduce the Rust implementation as an explicit opt-in backend first.
• Define behavior for environments where a native binary cannot be loaded: either an explicit managed fallback or a documented fail-fast policy.
• Consider making Rust the default XLSX query backend only after all acceptance gates pass and a migration/rollback plan exists.

5. Establish correctness parity

Run the shared XLSX parity contract against managed V2 and Rust for:

• Header and headerless dynamic rows.
• Typed mapping and value conversion.
• Sheet selection, start cells, empty rows, and sparse worksheets.
• Strings, numbers, booleans, blanks, errors, dates, times, datetimes, and durations.
• Corrupt/truncated input and exception mapping.
• Early disposal and cancellation.

Any intentional difference must be documented before enabling the Rust backend by default.

6. Add CI and package-consumer tests

For every supported RID:

• Build the Rust FFI artifact.
• Pack the NuGet package from clean artifacts.
• Install it into a separate test application from a temporary local feed.
• Run query parity and native load smoke tests on the matching runner/container.
• Verify package contents and reject missing, duplicate, or incorrectly named native assets.

Also validate framework-dependent, self-contained, single-file, trimmed, and NativeAOT publish scenarios where supported.

7. Define performance and memory gates

Measure cold start, first-row latency, throughput, managed allocation, peak working set, and repeated-query retention on representative small, medium, and large workbooks.

Initial gates:

• Throughput must be at least as fast as managed V2 on every supported platform.
• First-row latency must not regress by more than 10%.
• Peak working set must remain bounded as workbook size grows.
• Repeated queries and early disposal must not cause process memory growth.
• P/Invoke batching must not materially regress small-file performance.

Publish benchmark environments and raw results so regressions can be reproduced.

8. Roll out in stages

1. Preview packages and explicit opt-in API.
2. Cross-platform compatibility and deployment testing.
3. Release-candidate period with telemetry from benchmarks and user reports.
4. Decision on making Rust the default V2 XLSX query backend.
5. Keep a documented rollback path to the managed backend for at least the first stable release.

Acceptance Criteria

• The package architecture and RID resolution behavior are documented.
• Native packages are produced for all eight target RIDs.
• A clean consumer project restores and loads only the correct native library.
• Managed/Rust XLSX parity tests pass on Windows, Linux GNU, Linux musl, and macOS for x64 and arm64.
• Handles and buffers are released correctly on success, error, cancellation, and early disposal.
• CI builds, packs, inspects, installs, and smoke-tests every package.
• Single-file, trimming, self-contained, and NativeAOT compatibility results are documented.
• Cross-platform performance and memory gates pass.
• The opt-in, fallback, migration, and rollback behavior is documented.
• Default-backend switching is handled as a separate decision after the preview gates pass.

Alternatives Considered

• Put every native binary in one NuGet package. This is simpler and follows standard runtimes/{rid}/native conventions, but increases package download size.
• Keep the managed V2 implementation only. This avoids native deployment complexity but gives up the measured throughput and memory improvements.
• Call the existing NativeAOT CLI as a child process. This is language-neutral, but process startup and JSON serialization make it unsuitable as the primary in-process C# backend.

Additional Context

• Rust project: https://github.com/mini-software/MiniExcel-Rust
• Previous Rust tracking issue: Rust Version #995
• Rust-backed NuGet proof of concept and benchmark: https://github.com/mini-software/MiniExcel/blob/master/benchmarks/results/rust-nuget-poc.md

This issue is a packaging and integration plan. CSV, templates, workbook editing, and unrelated V2 API changes are out of scope for the first native-backend milestone.

[shps951023]

Good day Michele @michelebastione , team, could we implement test version on V2 and set up beta or preview tag?

And I created a demo repository an NuGet we can test: https://github.com/mini-software/MiniExcelRustNuGetTest

[michelebastione]

@shps951023 Here are my thoughts.

I believe these changes are being implemented in a little too rushed of a manner. Swapping the backend engine of the library should be, in my opinion, an endeavour to be pursued only after careful discussions around the pros and cons and detailed planning. Despite the performance advantages I am against it for the following reasons:

• It would take a lot of effort to implement, test and maintain every single native runtime; many important ones haven't even been listed (android, ios, maccatalyst, wasm, linux-riscv, win-x86) and to take care of all of them would require so much time and expertise.
• The complexity stemming from these changes would make developing bugfixes and new features significantly more difficult and time consuming, especially considering many devs from the broader .NET community who could potentially be interested in the project wouldn't be able to contribute to it if it becomes mainly rust based; incidentally this would make it very hard for me to keep contributing as well.
• The end users would become unable to directly inspect the code and step through it with the debugger, significantly hindering their experience when using the library in an interactive manner or when trying to troubleshoot issues; they would also find themselves downloading lots of native binaries on their machines, when up to this point MiniExcel has been a clutter-free piece of software.
• Last but not least, even though not very likely, it would become easier for bad actors to contribute malicious code to the project, as the vetting process would be a lot more challenging for manitainers unfamiliar with rust (like myself); moreover once it's shipped to the user it would effectively become a black box for them.

Given these points I suggest you reconsider the current course of action, but if you still want to continue on this path what I strongly recommend is creating a new assembly to be distributed as a separate package, something like MiniExcel.NativeOpenXml that users can knowingly choose to import instead of the idiomatic MiniExcel.OpenXml.

All that said, I hope you don't take this criticism personally; these are just the opinions I have as a maintainer and as a user, but as always the final say lays with you.

[shps951023]

@michelebastione No problem, and I agree that compatibility and maintainability need to come first.

Because you're leading our MiniExcel project, your opinion on this decision is most important.

Based on feedback, I see three options:

1. Create a completely separate NuGet package, as you suggested, such as MiniExcel.NativeOpenXml or MiniExcel.Rust.
2. Include both implementations in the same MiniExcel package, but expose the Rust backend through a clearly separate API, for example MiniExcelRust.Query(...).
3. Start with a separate preview package, then consider integrating it into the main package later through a separate API such as MiniExcelRust.Query(...), but only after it becomes mature and passes the necessary compatibility, deployment, and maintenance checks.

Which would you prefer? 🙌🙌🙌

[michelebastione]

Thank you for having my opinion in such high regard. What I would plan for the time being is option 1: create a separate package users can decide whether to use or not by explicitly importing it. This does not preclude the other two options if we change our mind in the future, while the opposite would not be possible.

[shps951023]

@michelebastione Sounds good! Let's proceed with Option 1 ❤