Release the GIL for the duration of LambdaLever::setLambda. - #481
Merged
Conversation
lohedges
had a problem deploying
to
sire-build
September 8, 2026 14:49 — with
GitHub Actions
Failure
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Profiling a replica exchange run with
py-spy record --gilshowed thatsetLambdaaccounted for 82% of all GIL-held time, serialising the worker threads against one another during the lambda switches that follow each dynamics block. The obvious fix, addingbp::release_gil_policy()to the wrapper, does nothing, becauserelease_gil_policyis compiled out codebase-wide bySIRE_DISABLE_GIL_POLICY, and it cannot simply be re-enabled:boost/python/detail/caller.hppruns the result converter between precall and postcall, so with the policy live every wrapped function that returns a converted value builds a Python object with the GIL released (enabling it segfaults duringimport sire). This addsHelpers/scoped_gil_release.hpp, an RAII guard that releases the GIL inside the wrapped call so that argument and result conversion still hold it, and uses it forsetLambdavia a thin hand-written wrapper.setLambdatouches no Python: itsOpenMM::Contextargument is converted before the release, and the only QM interaction is a plain setter. A comment onSIRE_DISABLE_GIL_POLICYnow records why it stays.Measured on a 12-window, 4-worker repex run: GIL-held time fell from 28.6% to 6.4% of wall, for +6.4% throughput (mean cycle (This was a small system, so the gain is likely smaller on a full protein-ligand box, where MD dominates a larger share of the cycle.)
The same approach can be applied to other functions as profiling identifies them. The recipe is to run
py-spy record --gilon a representative threaded workload, look for functions holding the GIL for a large share of the time, and wrap those withSireHelpers::ScopedGILRelease. Each candidate needs checking by hand first: the function must not re-enter Python, either directly or via a callback, and any from-Python argument conversion must complete before the GIL is released (boost constructs the arg_from_python converters ahead of the call, so this holds for ordinary arguments). This is deliberately case-by-case rather than blanket, i.e. the codebase-wide route is closed for the reasons above, and a wrapper is only worth adding where profiling shows it pays.Debugged and implemented with help of Claude Code.
develinto this branch before issuing this pull request (e.g. by runninggit pull origin devel): [y]