Repository navigation
Use DMTCP's main branch - #544
Merged
Merged
Conversation
|
No actionable comments were generated in the recent review.
 |
| Layer / File(s) | Summary |
|---|---|
C++ standard detection m4/ax_cxx_compile_stdcxx.m4, configure |
Compiler checks now cover C++17 and C++20 in configure; the Autoconf macro adds C++20 and C++23 modes and MSVC-aware version checks. |
C++20 requirement and build wiring configure.ac, configure, mpi-proxy-split/Makefile_config.in |
Configure now requires C++20, updates its feature macro, and removes the timing and logging options. Both MPI compiler configurations select C++20. |
MPI checkpoint and shared-memory changes
| Layer / File(s) | Summary |
|---|---|
DMTCP interfaces and lower-half restore dmtcp, mpi-proxy-split/lower-half/lower-half.cpp, mpi-proxy-split/lower_half_ckpt.cpp |
The DMTCP submodule reference changes. Lower-half restore code adopts DMTCP assertion and logging macros, and its post-restart callback now takes one integer. |
Saved memory-range selection mpi-proxy-split/mpi_plugin.cpp |
The plugin removes its cached mmap list and selects, clips, or skips memory areas using saved mmap ranges and the upper-half stack. |
MPI wrapper assertions and warnings mpi-proxy-split/mpi-wrappers/* |
MPI wrapper sources and the wrapper generator replace legacy assertion and warning macros with DMTCP macros. |
MPI message replay and draining checks mpi-proxy-split/p2p_drain_send_recv.cpp, mpi-proxy-split/p2p_log_replay.cpp, mpi-proxy-split/seq_num.cpp |
These code paths retain their described checks and control flow while adopting DMTCP assertion and logging macros and adding diagnostic values. |
Shared-memory end-to-end test mpi-proxy-split/test/Makefile, mpi-proxy-split/test/autotest.py, mpi-proxy-split/test/shared_memory.c |
The test targets add a two-rank test that validates sampled shared-memory values and an MPI-reduced checksum. |
Priority:
Estimated code review effort: 4 (Complex) | ~45 minutes
Change: Other
Possibly related issues
- Issue 485: The PR advances the DMTCP submodule and updates MANA for the DMTCP interface described by the issue.
Suggested reviewers: gc00, pera-noaa
Merge Risk: 
c31d6
-
No confirmed issue blocks merging. The checkpoint memory-skip behavior remains unverified against the pinned DMTCP version.
Pre-merge checks: 
-
Check name Status Explanation Resolution Docstring Coverage Docstring coverage is 41.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 60 functions across 21 files. Write docstrings for the functions missing them to satisfy the coverage threshold. -
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled. Title check The title "Use DMTCP's main branch" matches the primary change. The PR moves MANA from DMTCP's 4.2-dev branch to main, and the .gitmodules change and dmtcp submodule commit bump implement that move. T… Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request. Full details: Title check
-
Explanation
The title "Use DMTCP's main branch" matches the primary change. The PR moves MANA from DMTCP's 4.2-dev branch to main, and the .gitmodules change and dmtcp submodule commit bump implement that move. The title is short and specific. It does not mention the C++20 requirement, the memory-skip hook change, or the shared-memory test, but the title does not need to cover every detail.
-
-
Finishing Touches
-
🧪 Generate unit tests (beta)
-
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
DMTCP's main branch requires C++20. Its logging header, which MANA's plugin and lower half include, uses std::source_location and concepts. Therefore, configure.ac now requires C++20, and MANA compiles with -std=c++20. MANA's copy of the AX_CXX_COMPILE_STDCXX macro knew only C++11 to C++17. This commit takes the newer macro from DMTCP. As before, autoconf 2.71 regenerates configure. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit moves the dmtcp submodule from 97ab11e4 on 4.2-dev to
7c683bef on main. Every fix on 4.2-dev is also on main. In addition,
.gitmodules now tracks main, instead of master.
DMTCP's main branch replaced jalib's jassert.h with dmtcp_assert.h.
Therefore, MANA's JASSERT(cond)(x).Text("msg") becomes
ASSERT(cond, "msg: x={}", x). Likewise, JWARNING, JTRACE and JNOTE
become WARN, TRACE and NOTE. TRACE is now turned on at run time, with
DMTCP_LOG_LEVEL=trace. As a result, DMTCP's configure no longer
accepts --enable-logging, nor --enable-timing. MANA's configure
passes its options on to DMTCP's configure. Therefore, MANA's
configure drops both options too.
ThreadList::postRestart() now takes only the restart pause. However,
the lower half called it with the old (double, int) signature. This
signature is still in dmtcp.h, as PostRestartFnPtr_t. Therefore, the
lower half now declares its own function type, with the new signature.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
MANA used to skip every area named /dev/shm, /dev/kgni, /dev/xpmem, [heap] or /anon_hugepage. It did so before it looked at its list of the upper half's mappings, which the lower half's mmap wrapper keeps. However, the lower half's areas with these names are never in this list. The list records the upper half's mmap calls only, and the heap grows by brk instead. Therefore, the names changed the decision only for upper-half memory, which they dropped from the image. For example, they dropped an application's POSIX shared memory and named semaphores in /dev/shm, and its MAP_HUGETLB memory. The name "/dev/zero" once dropped CUDA's staging buffers in the same way. MANA now decides by the list alone. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An application's POSIX shared memory is a file in /dev/shm. MPI libraries map files in /dev/shm, too, but in the lower half. The new test shared_memory checks that MANA saves the application's object. Each rank maps an object of its own, and fills it once. In every iteration, the rank then checks a few words of the object. Before the previous commit, MANA did not save this object, and the restarted ranks crashed with SIGSEGV. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
xuyao00
force-pushed
the
pr/dmtcp-main
branch
from
October 26, 2026 15:40
6455f66 to
54bd7f7
Compare
xuyao00
marked this pull request as ready for review
October 26, 2026 15:46
Move the dmtcp submodule to a39bd30f, on DMTCP's main branch. There, dmtcp_skip_memory_region_ckpting() decides about the start of an area only. It returns 1 to skip [area->addr, *uncheckedAddr), or 0 to save it, and it sets *uncheckedAddr to the first address that it has not decided about. DMTCP then asks again about the unchecked rest. MANA's definition now decides with one binary search of the lower half's list of the upper half's mappings. It reads the live list. In contrast, the copy that the hook made at its first call misses memory that DMTCP maps while it writes the image, such as its allocator's chunks. This matters now, because DMTCP also asks the hook while it records guard pages, before it writes the image. The upper-half stack, which the lower half maps before its mmap wrapper exists, is one more range of the upper half's memory. Therefore, the range of an area inside the stack is saved, whereas such an area was skipped before. Finally, a range that MANA saves goes on to the next definition of the hook. As a result, another plugin can still skip some of it.
xuyao00
force-pushed
the
pr/dmtcp-main
branch
from
October 26, 2026 20:52
54bd7f7 to
c31d674
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR moves MANA from DMTCP's 4.2-dev branch to its main branch, which is stable and will become DMTCP 5.0. It needs dmtcp/dmtcp#1311, which is now merged. The dmtcp submodule points at a39bd30f, the merged #1311 on DMTCP's main branch.
Before the move, I compared the two branches commit by commit. Every fix on 4.2-dev is also on main. Only three changes of 4.2-dev are missing, and MANA needs none of them: the 4.2.0 version, the event
DMTCP_EVENT_RUNNING_AFTER(main hasDMTCP_EVENT_RUNNING_AFTER_LAUNCH,_RESUMEand_RESTARTinstead), and a new default for the program name prefix, which 4.2-dev changed in the generatedconfigureonly.5 commits:
std::source_locationand concepts.configure.acnow requires C++20, with theAX_CXX_COMPILE_STDCXXmacro from DMTCP, and MANA compiles with-std=c++20. As before, autoconf 2.71 regeneratesconfigure..gitmodulestracks main instead of master. About 90 calls ofJASSERT,JWARNING,JTRACEandJNOTEbecomeASSERT,WARN,TRACEandNOTEofdmtcp_assert.h. MANA'sconfiguredrops--enable-loggingand--enable-timing, because DMTCP'sconfigurenow rejects them. The lower half callsThreadList::postRestart()with its new signature,(int restartPause)./dev/shm,/dev/kgni,/dev/xpmem,[heap]or/anon_hugepage. However, the lower half's areas with these names are never in MANA's list of the upper half's mappings. Therefore, the names dropped only upper-half memory, such as an application's POSIX shared memory.shared_memorymaps an object fromshm_open()in each rank, and checks it in every iteration. Before the previous commit, the restarted ranks crashed with SIGSEGV.dmtcp_skip_memory_region_ckpting()a new interface. MANA's definition now decides with one binary search of the live list of the upper half's mappings. In addition, it saves the range of an area inside the upper-half stack, and it passes a range that it saves on to the next plugin's definition.Why dmtcp/dmtcp#1311
DMTCP's main branch reinstalls the guard pages of a process after restart. However, it also recorded the guard pages of the memory that MANA skips, which is the lower half. glibc 2.42 and later put a guard page at the low end of each thread stack, including the stacks of UCX's threads in the lower half. After restart, DMTCP put these guard pages into the new lower half's memory, and UCX crashed with SIGSEGV. In the CI's container (Ubuntu 26.04, MPICH 4.3 on UCX), 4 tests crashed this way in 4 runs of the blocking mode. With #1311, no test crashed in 3 runs. DMTCP's 4.2-dev branch records no guard pages.
Notes for review
psshows a rank asDMTCP:lower-hal. DMTCP's main branch adds theDMTCP:prefix to the program name by default.dmtcp.hstill declaresPostRestartFnPtr_twith the old(double, int)signature. Therefore, the lower half declares its own function type.mpi_fortran_wrappers.txt,p2p_drain_send_recv.cppandtest/Makefile; Implement MPI_Attr_get, MPI_Attr_put and MPI_Attr_delete in the upper half #535 inmpi_comm_wrappers.cpp; Serialize the mmap wrapper and block signals in it (#542) #543 intest/Makefile. Never replace existing mappings when placing one in the arena #537 merges cleanly. MANA with Open MPI and C++20 is not tested, because Support Open MPI 4.1 and 5.0 #529 is not merged.Testing
autotest.py, 41 tests with the newshared_memory, on the last commit with the submodule at the head of #1311 (21400c16):On Fedora 44,
p2p_ssend_collectivetimed out once at a checkpoint in blocking mode, as it sometimes does on main. Its checkpoint hung in the drain, before any memory was written. The test then passed 10 of 10. In addition,test-mana-restart.pypassed, and the first commit passed 4 tests with DMTCP's 4.2-dev branch. The third commit passed 7 tests, includingshared_memory, in both modes.After #1311 was merged, this PR was rebased onto main, and the submodule moved to a39bd30f. On Fedora 44 (MPICH 5.0.1), the CI's steps then passed:
test-mana-restart.py10 of 10,test-loader-debug.py15 of 15, andautotest.py41 of 41 in polling mode and 41 of 41 in blocking mode, with no retries.🤖 Generated with Claude Code
Summary by CodeRabbit
--enable-timingand--enable-loggingconfiguration options are no longer available.