Fix/libpython loading

README.md

@@ -17,6 +17,26 @@ This project is a fork of the original PythonFMU repository available at https:/
 Please take a look at the examples to see the supported features.
 We currently support both the Model exchange and Cosimulation Interfaces.
 
+See the [changelog](docs/changelog.md) for what changed in each release.
+
+### Where can I run the FMU?
+
+In any FMI 3.0 importer: Python-based tools such as FMPy, and native tools such as
+OMSimulator, `fmusim` or Simulink.
+
+The machine running the simulation needs Python 3 with a **shared** Python library, plus
+the packages your model imports (see `pythonfmu3 deploy` below). Inside a Python-based
+importer the FMU uses that interpreter. Otherwise, on Linux and macOS the FMU finds and
+loads `libpython` itself; if it picks the wrong one, or none, point it at the right library:
+
+```bash
+export PYTHONFMU_PYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.12.so.1.0
+```
+
+On Windows, `python3.dll` must be on the `PATH`. See
+[Running FMUs outside Python](docs/install.md#running-fmus-outside-python-linux-and-macos)
+for the full lookup order.
+
 ### How do I build an FMU from python code?
 
 1. Install `pythonfmu3` package:

docs/changelog.md

@@ -0,0 +1,32 @@
+# Changelog
+
+## Unreleased
+
+### Added
+- FMUs can be loaded by native (non-Python) FMI importers on Linux and macOS, such as
+  OMSimulator, `fmusim` and Simulink. Previously they only worked inside a Python-based
+  importer such as FMPy. See
+  [Running FMUs outside Python](install.md#running-fmus-outside-python-linux-and-macos).
+- `PYTHONFMU_PYTHON_LIBRARY` environment variable to choose the Python library the FMU loads.
+- Python errors raised by a model are reported with their full traceback in the FMU log.
+
+### Changed
+- On Linux and macOS, `binaries/<platform>/` now holds two libraries: `<modelIdentifier>`
+  (the FMI entry point) and `libpythonfmu-backend`. Keep them together.
+- `resources/slavemodule.txt` also records the model class name, and
+  `resources/pythonfmu_libpython.txt` records the building interpreter's Python library.
+- The builder finds the model class by importing the script, so class definitions whose
+  base classes span several lines work. A script must define a single FMU class.
+
+### Fixed
+- Crash (use-after-free) when freeing an instance after a Python exception.
+- Possible deadlock when instantiation failed without a Python exception.
+- Out-of-bounds reads when an importer requested more values than the model returned.
+- `fmi3DoStep` now always sets `eventHandlingNeeded`, `terminateSimulation` and
+  `earlyReturn`, and honours the matching `Fmi3StepResult` fields.
+- Memory leaks in logging, FMU-state handling and failed instantiation.
+- Crash when the importer passes no log callback (allowed by FMI 3.0).
+- Default CoSimulation/ModelExchange capability flags in the model description when
+  building from Python (e.g. `canHandleVariableCommunicationStepSize` came out `false`).
+- `--terminals` packed the wrong file as `terminalsAndIcons.xml`.
+- Array variable setters on NumPy 2.5 or later.

docs/dependencies.md

@@ -28,4 +28,16 @@ optional arguments:
   -f FMU, --file FMU    Path to the Python FMU.
   -e ENVIRONMENT, --env ENVIRONMENT
                         Requirements or environment file.
-```
+```
+
+### Which Python environment does the FMU use?
+
+- **Python-based importer** (e.g. FMPy): the interpreter running the importer, with its
+  packages. Run `pythonfmu3 deploy` in that same environment.
+- **Native importer on Linux/macOS** (OMSimulator, `fmusim`, Simulink, ...): the Python
+  whose `libpython` the FMU loads (see
+  [Running FMUs outside Python](install.md#running-fmus-outside-python-linux-and-macos)).
+  That is a base installation, not a virtual environment: packages installed only in a
+  venv are not found unless you add its `site-packages` folder to `PYTHONPATH`, e.g.
+  `export PYTHONPATH=/path/to/venv/lib/python3.12/site-packages`.
+- **Windows**: the Python providing the `python3.dll` found on the `PATH`.

docs/index.md

@@ -85,12 +85,19 @@ import fmpy
 result = fmpy.simulate_fmu('MyModel.fmu', stop_time=10.0)
 ```
 
+The FMU also runs in native FMI 3.0 importers such as OMSimulator, `fmusim` or Simulink.
+The simulating machine needs Python 3 with a shared Python library and your model's
+packages; see [Running FMUs outside Python](install.md#running-fmus-outside-python-linux-and-macos).
+
 ## Architecture
 
 PythonFMU3 uses a hybrid architecture combining the performance of C++ with the flexibility of Python:
 
 - **C++ Runtime**: High-performance FMI interface implementation
 - **Embedded Python**: Isolated Python interpreter for model execution
+- **Python loader (Linux/macOS)**: the FMU's library has no link-time dependency on
+  Python. It uses the host's interpreter when there is one, and otherwise loads
+  `libpython` on first instantiation, so native (non-Python) importers can load the FMU
 - **Smart Bridging**: Efficient data exchange between C++ and Python
 - **Memory Management**: Automatic resource cleanup and garbage collection
 

docs/install.md

@@ -28,3 +28,26 @@ or,
 ```bash
 pip install .
 ```
+
+## Running FMUs outside Python (Linux and macOS)
+
+An FMU built with PythonFMU3 runs Python code, so the machine that simulates it needs a
+Python 3 installation with a **shared** Python library (`libpython3.x.so` on Linux,
+`libpython3.x.dylib` or a `Python.framework` on macOS), plus any packages the model imports.
+
+When the importer is itself a Python program (e.g. FMPy), the FMU uses that interpreter.
+Otherwise (OMSimulator, `fmusim`, Simulink, ...) the FMU loads `libpython` itself, looking in
+this order:
+
+1. the `PYTHONFMU_PYTHON_LIBRARY` environment variable, e.g.
+   `export PYTHONFMU_PYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.12.so.1.0`;
+2. the library of the Python that built the FMU (recorded in
+   `resources/pythonfmu_libpython.txt`);
+3. standard library names for Python 3.8 to 3.14 (and the usual Homebrew and
+   python.org framework locations on macOS).
+
+If none can be loaded, instantiation fails with a message listing the paths tried.
+Interpreters built without a shared library (some `pyenv` builds, for example) cannot be
+used; with `pyenv`, reinstall with `PYTHON_CONFIGURE_OPTS="--enable-shared"`.
+
+On Windows the FMU links `python3.dll`, which must be on the `PATH`.

mkdocs.yml

@@ -5,4 +5,5 @@ nav:
     - Co-simulation: usage.md
     - Model Exchange: usageMX.md
     - Dependencies: dependencies.md
+    - Changelog: changelog.md
 theme: readthedocs

pythonfmu3/builder.py

@@ -6,10 +6,11 @@
 import re
 import shutil
 import sys
+import sysconfig
 import tempfile
 import zipfile
 from pathlib import Path
-from typing import Iterable, Optional, Tuple, Union
+from typing import Iterable, List, Optional, Tuple, Union
 from xml.dom.minidom import parseString
 from xml.etree.ElementTree import Element, SubElement, tostring
 from .osutil import get_lib_extension, get_platform
@@ -95,6 +96,29 @@ def get_model_description(filepath: Path, module_name: str) -> Tuple[str, Elemen
     return instance.modelName, instance.to_xml(), class_name
 
 
+def _is_fmu_entry_library(path: Path) -> bool:
+    """True for the library exporting the FMI functions (renamed to the model identifier)."""
+    return path.stem in ("pythonfmu-export", "libpythonfmu-export")
+
+
+def _libpython_candidates() -> List[str]:
+    """Paths of the shared libpython of the running interpreter, if it has one."""
+    if sys.platform == "win32":
+        return []  # the Windows library links python3.dll directly
+    libdir = sysconfig.get_config_var("LIBDIR") or ""
+    framework_prefix = sysconfig.get_config_var("PYTHONFRAMEWORKPREFIX") or ""
+    candidates = []
+    for name in (sysconfig.get_config_var("INSTSONAME"), sysconfig.get_config_var("LDLIBRARY")):
+        if not name or name.endswith(".a"):
+            continue  # statically linked interpreter: nothing to point at
+        for base in (libdir, framework_prefix):
+            if base:
+                candidate = Path(base) / name
+                if candidate.is_file() and str(candidate) not in candidates:
+                    candidates.append(str(candidate))
+    return candidates
+
+
 class FmuBuilder:
 
     @staticmethod
@@ -200,6 +224,15 @@ def build_FMU(
                         relative_f = f.relative_to(temp_dir)
                         zip_fmu.write(f, arcname=(resource / relative_f))
 
+                # Hint for the Linux/macOS shim: where the Python that built this FMU
+                # keeps its shared library (used when the importer is not Python itself).
+                libpython_hints = _libpython_candidates()
+                if libpython_hints:
+                    zip_fmu.writestr(
+                        str(resource.joinpath("pythonfmu_libpython.txt")),
+                        "\n".join(libpython_hints) + "\n",
+                    )
+
                 # Add information for the Python loader: line 1 is the module, line 2 the class.
                 # Older wrappers only read the first line, so this stays backward compatible.
                 zip_fmu.writestr(
@@ -226,12 +259,13 @@ def build_FMU(
                     src_binaries.rglob("*.dylib"),
                 ):
                     relative_f = f.relative_to(src_binaries)
-                    arcname = (
-                        binaries
-                        / relative_f.parent
-                        / f"{model_identifier}{relative_f.suffix}"
-                    )
-                    zip_fmu.write(f, arcname=arcname)
+                    # Only the FMI entry-point library takes the model identifier's name;
+                    # helper libraries (the Linux/macOS backend) keep their own name.
+                    if _is_fmu_entry_library(f):
+                        name = f"{model_identifier}{relative_f.suffix}"
+                    else:
+                        name = relative_f.name
+                    zip_fmu.write(f, arcname=(binaries / relative_f.parent / name))
 
                 # Add the documentation folder
                 if documentation_folder is not None:

pythonfmu3/pythonfmu-export/generate_shim.py

@@ -0,0 +1,88 @@
+"""Generate src/shim/fmi3_forwarders.cpp from src/fmi/fmi3FunctionTypes.h.
+
+The shim is the library the importer loads on Linux/macOS. It has no link-time
+dependency on libpython: on instantiation it loads libpython (unless the host process
+already provides it, e.g. when the importer is itself written in Python) and then the
+real implementation (the "backend" library), and forwards every FMI call to it.
+
+Re-run this script after updating the FMI headers:
+    python generate_shim.py
+"""
+import re
+from pathlib import Path
+
+HERE = Path(__file__).parent
+TYPES_HEADER = HERE / "src" / "fmi" / "fmi3FunctionTypes.h"
+OUTPUT = HERE / "src" / "shim" / "fmi3_forwarders.cpp"
+
+TYPEDEF_RE = re.compile(r"typedef\s+([\w\s\*]+?)\s*\b(fmi3\w+)TYPE\s*\((.*?)\)\s*;", re.S)
+INSTANTIATE = {
+    "fmi3InstantiateModelExchange",
+    "fmi3InstantiateCoSimulation",
+    "fmi3InstantiateScheduledExecution",
+}
+
+
+def param_names(params: str):
+    params = " ".join(params.split())
+    if params in ("", "void"):
+        return []
+    return [re.findall(r"\w+", p.replace("[]", ""))[-1] for p in params.split(",")]
+
+
+def main():
+    functions = [
+        (" ".join(ret.split()), name, " ".join(params.split()))
+        for ret, name, params in TYPEDEF_RE.findall(TYPES_HEADER.read_text())
+    ]
+    forwarded = [f for f in functions if f[1] != "fmi3GetVersion"]
+
+    out = [
+        "// GENERATED by generate_shim.py from fmi/fmi3FunctionTypes.h - do not edit by hand.",
+        '#include "shim/shim_loader.hpp"',
+        "",
+        "namespace pythonfmu_shim",
+        "{",
+        "const char* const kFunctionNames[] = {",
+        *[f'    "{name}",' for _, name, _ in forwarded],
+        "};",
+        f"const std::size_t kFunctionCount = {len(forwarded)};",
+        "} // namespace pythonfmu_shim",
+        "",
+        'extern "C" {',
+        "",
+        "FMI3_Export const char* fmi3GetVersion(void)",
+        "{",
+        "    return fmi3Version;",
+        "}",
+        "",
+    ]
+    for index, (ret, name, params) in enumerate(forwarded):
+        names = param_names(params)
+        args = ", ".join(names)
+        out.append(f"FMI3_Export {ret} {name}({params})")
+        out.append("{")
+        if name in INSTANTIATE:
+            out.append(
+                "    if (!pythonfmu_shim::ensureBackendLoaded(resourcePath, instanceEnvironment, logMessage)) return nullptr;"
+            )
+        out.append(f"    const auto fn = reinterpret_cast<{name}TYPE*>(pythonfmu_shim::function({index}));")
+        if ret == "void":
+            out.append(f"    if (fn) fn({args});")
+        elif ret == "fmi3Status":
+            out.append("    if (!fn) return fmi3Error;")
+            out.append(f"    return fn({args});")
+        else:
+            out.append("    if (!fn) return nullptr;")
+            out.append(f"    return fn({args});")
+        out.append("}")
+        out.append("")
+    out.append('} // extern "C"')
+    out.append("")
+    OUTPUT.parent.mkdir(exist_ok=True)
+    OUTPUT.write_text("\n".join(out))
+    print(f"wrote {OUTPUT.relative_to(HERE)} ({len(functions)} functions)")
+
+
+if __name__ == "__main__":
+    main()

pythonfmu3/pythonfmu-export/src/CMakeLists.txt

@@ -1,4 +1,3 @@
-
 set(headers
         cppfmu/cppfmu_cs.hpp
         cppfmu/cppfmu_common.hpp
@@ -17,29 +16,58 @@ set(sources
         pythonfmu/PySlaveInstance.cpp
         )
 
-add_library(pythonfmu-export ${sources} ${headers})
-target_compile_features(pythonfmu-export PUBLIC "cxx_std_17")
+set(BINARY_OUTPUT_DIR "${CMAKE_SOURCE_DIR}/../resources/binaries/${TARGET_PLATFORM}")
 
-target_include_directories(pythonfmu-export
+if (WIN32)
+  # Windows: a single library, linked against the version-independent python3.dll.
+  set(PYTHONFMU_IMPL_TARGET pythonfmu-export)
+else ()
+  # Linux/macOS: the Python-dependent implementation lives in a "backend" library
+  # (FMI functions renamed with a prefix). The FMU library itself (pythonfmu-export,
+  # renamed to <modelIdentifier> by the builder) is a small shim with no link-time
+  # dependency on libpython, so non-Python importers can load it. See shim/shim_loader.cpp.
+  set(PYTHONFMU_IMPL_TARGET pythonfmu-backend)
+endif ()
+
+add_library(${PYTHONFMU_IMPL_TARGET} ${sources} ${headers})
+target_compile_features(${PYTHONFMU_IMPL_TARGET} PUBLIC "cxx_std_17")
+target_include_directories(${PYTHONFMU_IMPL_TARGET}
         PRIVATE
         "${Python3_INCLUDE_DIRS}"
         "${CMAKE_CURRENT_SOURCE_DIR}"
         )
 
 if (WIN32)
-  target_link_libraries(pythonfmu-export PRIVATE ${Python3_LIBRARIES})
-elseif (APPLE)
-  set_target_properties(pythonfmu-export PROPERTIES LINK_FLAGS "-undefined dynamic_lookup")
-endif ()
-
-if (WIN32)
-  set_target_properties(pythonfmu-export
+  target_link_libraries(${PYTHONFMU_IMPL_TARGET} PRIVATE ${Python3_LIBRARIES})
+  set_target_properties(${PYTHONFMU_IMPL_TARGET}
           PROPERTIES
-          RUNTIME_OUTPUT_DIRECTORY_RELEASE "${CMAKE_SOURCE_DIR}/../resources/binaries/${TARGET_PLATFORM}"
+          RUNTIME_OUTPUT_DIRECTORY_RELEASE "${BINARY_OUTPUT_DIR}"
           )
 else ()
+  # Python symbols are resolved at load time against the process (the host Python, or
+  # the libpython the shim opened with RTLD_GLOBAL).
+  target_compile_definitions(${PYTHONFMU_IMPL_TARGET} PRIVATE FMI3_FUNCTION_PREFIX=pythonfmu_backend_)
+  if (APPLE)
+    set_target_properties(${PYTHONFMU_IMPL_TARGET} PROPERTIES LINK_FLAGS "-undefined dynamic_lookup")
+  endif ()
+  set_target_properties(${PYTHONFMU_IMPL_TARGET}
+          PROPERTIES
+          LIBRARY_OUTPUT_DIRECTORY "${BINARY_OUTPUT_DIR}"
+          )
+
+  add_library(pythonfmu-export
+          shim/shim_loader.cpp
+          shim/shim_loader.hpp
+          shim/fmi3_forwarders.cpp
+          )
+  target_compile_features(pythonfmu-export PUBLIC "cxx_std_17")
+  target_include_directories(pythonfmu-export PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}")
+  target_link_libraries(pythonfmu-export PRIVATE ${CMAKE_DL_LIBS})
   set_target_properties(pythonfmu-export
           PROPERTIES
-          LIBRARY_OUTPUT_DIRECTORY "${CMAKE_SOURCE_DIR}/../resources/binaries/${TARGET_PLATFORM}"
+          LIBRARY_OUTPUT_DIRECTORY "${BINARY_OUTPUT_DIR}"
+          # Only the FMI functions (marked FMI3_Export) are exported
+          CXX_VISIBILITY_PRESET hidden
+          VISIBILITY_INLINES_HIDDEN ON
           )
 endif ()