Fixes relative to RPATH handling

[dnicolodi]

Builds on top of #783 and replaces #724

Fixes #725
Fixes #813

[dnicolodi]

This should fix RPATH handling for good.

The only thing not included is automatic translation of the $ORIGIN anchor in install_rpath arguments to @loader_path on macOS. It would not be hard to implement, however, it would be a deviation from what Meson implements. I am not sure it is a good idea to implement it.

There is one case in which this may break projects that work now: when libraries or modules require setting an RPATH to dynamically link to a library installed in the Python install path (with something like install_dir: py.get_install_dir() / 'package') and that work now for how because meson-python does not remove build RPATH entries added by Meson and the source layout matches the install layout. This was the case for example for SciPy 1.15, see #724 (comment) and previous discussion. The comments there indicated that, for what SciPy is concerned, it should be fine to break this now. Note that SciPy 1.15 used install_rpath arguments that are implemented here, thus it would be fine, but it uses the $ORIGIN anchor on macOS too, and that does not work. See above. The behavior changes only with Meson 1.9.0 or later (older Meson versions did not export the required metadata), thus projects that pin the Meson version are fine.

@rgommers I think I added test cases for all scenarios we discussed. It would be great if you could test with packages that may be affected and that do not pin the meson-python version to any released version.

[dnicolodi]

I think it is about time that we straighten out our RPATH story, thus tentatively marking this for the next release. However, it still needs a decision regarding the handling of the anchors in install_rpath, see comments above. I'll work on testing downstream projects.

[dnicolodi]

@rgommers I want to add extended tests in CI that test building SciPy (and maybe some other projects later). I had a quick look at how to do it and I didn't find what the preferred way of building SciPy is these days. Should I use pixi, spin, or something else? Or is downloading the source distribution and installing with pip or uv the way to go? The latter seems the easiest but pip does not have a way to install the build dependencies for a package and disabling build isolation is by far the simplest to test a local version of meson-python.

[rgommers]

Hmm, are you sure - that's a really heavy build (10-15 min), and it's nontrivial to set up because of the BLAS dependency. Or are you thinking as a one-off or some sort of extended test suite?

We do test meson-python's main branch in SciPy CI, but only on Linux I believe: https://github.com/scipy/scipy/blob/c18ebd1c4871ff1a04c397247815d859eec54de5/.github/workflows/linux.yml#L183

And that doesn't test all RPATH rewriting issues. I think what we probably need for more coverage is a macOS SciPy job against main?

Re building:

• If we'd want it in CI here, then I'd say that a regular Ubuntu job where the system dependencies are installed with apt is probably best. And then a regular pip install scipy --no-binary scipy. There no job that can be copied 1:1, but something like this job
• pixi is by far the easiest for local development I think, if you clone the scipy repo and replace the meson-python dependency in pixi.toml with a path dependency to the local clone of meson-python.

---

I did plan on getting back to this PR after the release. It'd be great to get this merged soon.

[dnicolodi]

I was thinking to add an extended test suite gated on all the existing CI jobs to complete successfully. However, I didn't expect SciPy to take 15 min to compile, it is a long time since I had to do it. Do you think it is worth having something like this in our CI?

[eli-schwartz]

Does ccache not help here? I know that scipy takes a while because it has 1200 compile edges, with quite a bit of C++. On my laptop (znver5, -j16) it takes

$ time FEATURES=ccache ebuild scipy-1.17.1.ebuild clean install
real	4m50.092s
user	37m28.643s
sys		2m10.715s

But on a second run, it completes significantly faster:

real	0m37.002s
user	3m35.543s
sys		0m23.999s

[eli-schwartz]

(Needing to compile all of highs despite a system package existing for it, surely is not helping compile times. Compiling that package from source with emerge -1 =sci-mathematics/highs-1.14.0-r3 takes 48 seconds all on its own.)

[rgommers]

I was thinking to add an extended test suite gated on all the existing CI jobs to complete successfully. However, I didn't expect SciPy to take 15 min to compile, it is a long time since I had to do it. Do you think it is worth having something like this in our CI?

I'm inclined to say no here. I think there's a lot of value in our CI being simple and very fast. We'd make the completion time way longer, and the setup quite a bit more maintenance-intensive. I'd much rather add more CI jobs in SciPy's CI that use meson-python's main branch on different platforms.

I do like the idea of better integration testing though. I think we can (a) push more of our large users to test our main branch, and (b) have the kind of CI machinery you have in mind here in a separate repo, so it's easier to test at scale. I think https://github.com/rgommers/external-deps-build/ does a lot of what you have in mind here. We can clone that, tweak it a bit so it can be used to test PRs/branches, and populate it with a lot of packages that use meson-python. WDYT?

Does ccache not help here?

Yes it does, and we use it in SciPy - but that comes with its own extra work; two job steps per build, cache management, having to run CI jobs on main to populate the cache for PRs.

[rgommers]

The only thing not included is automatic translation of the $ORIGIN anchor in install_rpath arguments to @loader_path on macOS. It would not be hard to implement, however, it would be a deviation from what Meson implements. I am not sure it is a good idea to implement it.

I did a bit of searching, and it's recommended in our docs, and SciPy isn't the only project that used it.

• SciPy: code in scipy 1.15.x (meson-python<0.20.0, so won't break), and a blog post talking about that feature
• VapourSynth, current master branch (meson-python unpinned): https://github.com/vapoursynth/vapoursynth/blob/5b2d5562726a91d9a75441cc4728a90e6c9f4f27/meson.build#L308
• dwave-optimization, current main branch (meson-python~=0.20.0): https://github.com/dwavesystems/dwave-optimization/blob/0962fc5e3ac4adfec1d2b592ef7a0b3c016f39cd/meson.build#L162

If I can find three significant projects using the feature quickly, and we've recommended it in our docs for a while, there will be more users. We should add a backwards compat shim. That doesn't cost much; the Meson behavior of retaining a literal $ORIGIN isn't all that useful after all. We can document it as legacy, and in a few releases start warning about it so package authors can update to the new recommendation.

[rgommers]

My review plan here is more a test plan - since I can't just read through the code and reason about changed test packages for completeness, I'll plan to add some new ones with more layouts, and run that in CI on my fork on Linux and macOS. And then add NumPy (same as SciPy, just builds faster), VapourSynth and dwave-optimization builds, and check that they all work without patching and don't contain duplicate RPATHs.

[rgommers]

This PR should also update the docs on shared libraries, so we recommend the new preferred form. Which I think is @loader_path on macOS, and $ORIGIN elsewhere.

[dnicolodi]

Translating $ORIGIN into @loader_path never worked. install_rpath never worked, actually. Thus, all packages that ever set an RPATH worked by accident because of other bugs that left RPATH entries in place when they should have been removed.

This PR implements what IMO is correct handling of RPATH entries, including removal of build RPATH entries. This will break packages that have been relying (without realizing it, I assume) on build RPATH entries. The anchor translation will help the project that have (no-op till now) install_rpath directives with the $ORIGIN anchor. I suspect there may be others out there that will simply break.

[rgommers]

Translating $ORIGIN into @loader_path never worked. install_rpath never worked, actually. Thus, all packages that ever set an RPATH worked by accident because of other bugs that left RPATH entries in place when they should have been removed.

Sure, I understand that. However, it had the effect of working in practice - one bug cancelling out another made that so. Let's make it easy: I'll see if I can find a simple-enough way of adding in backwards compat handling here. If I don't succeed, there's nothing to discuss. If it does work, it's probably easy to decide to add it in, so we can hard affected people rather than hard-break them in the next release.

[dnicolodi]

I'll see if I can find a simple-enough way of adding in backwards compat handling here.

If you mean adding the RPATH anchor translation, that should be easy. Knowing the code, I can do it quickly, I think.

[rgommers]

Yes, that's what I meant. I was actually planning to leave it alone right now, focus on testing, see something fail (the two projects in the list above), and then get the respective CI jobs green again with a bit of backwards compat handling.