Skip to content

Run MANA on Open MPI - #538

Closed
pera-noaa wants to merge 3 commits into
mpickpt:mainfrom
pera-noaa:open-mpi
Closed

pera-noaa wants to merge 3 commits into
mpickpt:mainfrom
pera-noaa:open-mpi

Conversation

@pera-noaa

Copy link
Copy Markdown
Contributor

Under Open MPI 4.1.4, programs did not launch, and after a checkpoint they hung.

  1. Open MPI's predefined handles are addresses, often with bit 31 set, so the sign-extended lookup missed MPI_COMM_WORLD. Try the exact value first (no change for MPICH).
  2. Open MPI's MPI_REQUEST_NULL differs between the halves. A send the drain completed reached the lower half's MPI_Test as the upper half's null, and never completed. Pass the lower half's null.
  3. libmpistub.so lacks Open MPI's ompi_mpi_* objects, so libmana.so failed to load. Keep the real library under Open MPI. @xuyao00, if you'd rather the stub define them, I'll drop this commit.

Hercules, 2 nodes (with #527): 18/23 in both modes. The rest are the Fortran tests (the wrappers assume MPICH's handles), attributes (#531), and two tests not built for Open MPI. Intel MPI: unchanged, 22/23 (#524).

🤖 Generated with Claude Code

get_real_id_slow() looked up the sign-extended 32-bit value whenever bit 31 was set, a workaround for MPICH's negative datatype handles. Open MPI's predefined handles are addresses in libmpi.so, and above 2 GiB bit 31 is set (MPI_COMM_WORLD at 0x837c4bc0 on Hercules). The lookup then failed and the rank stopped in get_virt_id_entry() at the first MPI_Comm_rank. Try the exact value first, then the sign-extended one.
…eted

The drain completes pending sends and non-blocking collectives by mapping their virtual request to MPI_REQUEST_NULL, with no status. MPI_Test_internal(), behind every Wait and Test wrapper, then passed that upper-half MPI_REQUEST_NULL to the lower half's MPI_Test. In MPICH it is one integer in both halves. In Open MPI it is &ompi_request_null, a different address in each half, so the lower half took it for a live request that never completes: after a checkpoint, the application spun forever in MPI_Wait (p2p_nonblocking, p2p_large, nonblocking_collectives). Pass the lower half's MPI_REQUEST_NULL instead; MPI_Request_get_status had the same problem.
Since 9b1c14c the upper half loads libmpistub.so in place of every library named libmpi, Open MPI's libmpi.so.40 included. The stub has MPICH's ABI and defines none of Open MPI's ompi_mpi_* objects, so libmana.so, built with Open MPI's mpi.h, fails to load: "undefined symbol: ompi_mpi_real". Under Open MPI, skip the swap, as before 9b1c14c.
@coderabbitai

coderabbitai Bot commented Oct 20, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: c0066739-8f4d-4b65-9bf4-855332645434
📥 Commits

Reviewing files that changed from the base of the PR and between d602397 and 58eeda4.

📒 Files selected for processing: 3
  • mpi-proxy-split/lower-half/lower-half.cpp
  • mpi-proxy-split/mpi-wrappers/mpi_request_wrappers.cpp
  • mpi-proxy-split/virtual_id.cpp
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@xuyao00

xuyao00 commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator

Can you check if PR#529 solves you problem? I think it's covered in PR#529.

@pera-noaa

Copy link
Copy Markdown
Contributor Author

Sorry, I hadn't seen #529 when I opened this. Yes, #529 covers all three: the lookup key by handle size, the lower half's MPI_REQUEST_NULL for requests MANA completed, and the stub's Open MPI objects. On Hercules, with #529 merged onto main (2 nodes, 3 checkpoints per test):

MPI polling blocking
Open MPI 4.1.4 (ob1, TCP) 41 of 41 41 of 41
Intel MPI 2021.7.1 40 of 41 41 of 41

This PR had 18 of 23 with Open MPI 4.1.4. The Intel failure is p2p_probe ("told to stop at iteration N"), which fails on main too (5 of 10 runs, against 4 of 10 with #529).

The merge has two conflicts with #522, both small. In p2p_drain_send_recv.cpp, commName() now names MPI_COMM_WORLD 0 and MPI_COMM_SELF 1, and commByName() maps them back. In test/Makefile, fortran_handles is added to the new lists.

#529 also fixes #531: on Open MPI, the first user keyval is now 7. #532 is still open with #529: MPI_UNIVERSE_SIZE, MPI_APPNUM and MPI_LASTUSEDCODE return MPI_ERR_KEYVAL under both MPIs. I'll do it once #529 is in.

I'm closing this in favor of #529.

@pera-noaa pera-noaa closed this Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants