diff --git a/docs/design/SPECIFICATION_TECHNIQUE_FINALE_POPS_ARCHITECTURE.md b/docs/design/SPECIFICATION_TECHNIQUE_FINALE_POPS_ARCHITECTURE.md index af5e0e05d..797cf5ff2 100644 --- a/docs/design/SPECIFICATION_TECHNIQUE_FINALE_POPS_ARCHITECTURE.md +++ b/docs/design/SPECIFICATION_TECHNIQUE_FINALE_POPS_ARCHITECTURE.md @@ -1491,11 +1491,13 @@ scientifiques choisissent obligatoirement un `ParallelMode` typé : d'un unique writer rang 0, `COLLECTIVE` pour les hyperslabs HDF5 MPIO exacts, ou `PER_RANK` pour des artefacts locaux qualifiés par rang et un reçu agrégé. Le mode, le format, la sélection, la cible et l'identité de chaque pièce native (`global_box_index`, `owner_rank`, `replicated`) sont authentifiés -entre rangs avant toute écriture. La route `COLLECTIVE` appelle le backend C++ HDF5 parallèle avec -la lane MPI dupliquée possédée par la session observateur ; le writer ne redécouvre ni n'emprunte -`MPI_COMM_WORLD`. `h5py` reste uniquement un lecteur/écrivain série optionnel et n'est jamais un -transport MPI. Une dépendance HDF5 parallèle native absente, un mode incompatible ou un backend -Kokkos GPU/device handle non supporté est refusé avant le +entre rangs avant toute écriture. La capture native `ROOT` reçoit uniquement une lane consommateur +dupliquée pour le run et la libère collectivement à sa fermeture ; les façades +`System`/`AmrSystem` n'acceptent plus le singleton monde pour cette route. La route `COLLECTIVE` +appelle le backend C++ HDF5 parallèle avec la lane MPI dupliquée possédée par la session observateur ; +le writer ne redécouvre ni n'emprunte `MPI_COMM_WORLD`. `h5py` reste uniquement un +lecteur/écrivain série optionnel et n'est jamais un transport MPI. Une dépendance HDF5 parallèle +native absente, un mode incompatible ou un backend Kokkos GPU/device handle non supporté est refusé avant le constructeur de `System`/`AmrSystem`; aucune route série implicite ne remplace une demande MPI. Les maillages non structurés, mobiles/déformables ou changeant de topologie, de nouvelles familles de diff --git a/docs/design/exact-output-consumers.md b/docs/design/exact-output-consumers.md index e2ab048b5..beac3137a 100644 --- a/docs/design/exact-output-consumers.md +++ b/docs/design/exact-output-consumers.md @@ -82,8 +82,10 @@ count, target suffix, or writer availability: - `SERIAL` requires the proved serial `ExecutionContext` (rank 0, size 1) and one complete snapshot. - `ROOT` requires a distributed context. Every rank participates in the authenticated native - gather, but only rank 0 prepares, verifies and atomically publishes the single-file writer. - Preparation failures and the final receipt are broadcast to every participant. + gather over a run-scoped duplicated consumer lane, but only rank 0 prepares, verifies and + atomically publishes the single-file writer. The native `System`/`AmrSystem` output bridge + accepts only that owned lane, never the process-world singleton. Preparation failures and the + final receipt are broadcast to every participant. - `COLLECTIVE` requires a distributed context, an authenticated collective resource plan and the native C++ parallel-HDF5 provider. The observer runtime owns a duplicated MPI lane for the complete writer session; neither the Python writer nor the native HDF5 adapter borrows or rediscovers the @@ -193,10 +195,11 @@ therefore write NPZ, HDF5 or the complete VTU/PVTU/PVD/state ParaView bundle. `q retained detached snapshots; a full queue deliberately applies backpressure. The selected format owns the topology. `SERIAL` uses the sole rank. `ROOT` performs the complete -snapshot gather on the main execution path, then writes from the rank-zero worker without worker -MPI. `PER_RANK` and `COLLECTIVE` run one worker per rank over a run-scoped communicator duplicated -collectively before any worker starts. That private lane has a distinct MPI context from -`MPI_COMM_WORLD`, so numerical and output collective orderings cannot alias. PoPS requires +snapshot gather on the main execution path over one run-scoped duplicated consumer lane, then +writes from the rank-zero worker without MPI. `PER_RANK` and `COLLECTIVE` run one worker per rank +over a run-scoped communicator duplicated collectively before any worker starts. Those private +lanes have distinct MPI contexts from `MPI_COMM_WORLD`, so numerical and output collective +orderings cannot alias. PoPS requires `MPI_THREAD_MULTIPLE`, authenticates the lane on every worker call and fixes distributed `max_attempts` to one: retrying after entry into an MPI publication would not be safe. Supported mode combinations remain those of the format itself; in particular, ParaView has no `COLLECTIVE` mode and @@ -385,9 +388,10 @@ re-emission; a rank-local `KeyboardInterrupt`/`SystemExit` cannot split collecti - HDF5 uses native datasets and `read_hdf5()` verification. Serial/root fields must be complete. Collective mode requires the compiled C++ parallel-HDF5 route before preparation; every rank writes its declared non-overlapping hyperslabs through the exact authenticated communicator and - the manifest authenticates all pieces. A synchronous consumer uses the execution communicator; - an asynchronous consumer uses its private duplicated worker lane. Python never emulates this - mode with a gather-to-root writer: the compiled provider owns the MPIO dataset transfers. + the manifest authenticates all pieces. The HDF5 session uses its private duplicated observer lane; + neither synchronous nor asynchronous publication borrows the process world. Python never + emulates this mode with a gather-to-root writer: the compiled provider owns the MPIO dataset + transfers. Partition validation scales with piece count rather than global cell count, and shared geometry is written once by rank zero. Unlike the default relayed PVTU topology, the single collective HDF5 target is opened by every rank through parallel HDF5/MPI-IO and must therefore be genuinely diff --git a/include/pops/parallel/comm.hpp b/include/pops/parallel/comm.hpp index 02b6e8e06..66d009dbb 100644 --- a/include/pops/parallel/comm.hpp +++ b/include/pops/parallel/comm.hpp @@ -96,6 +96,17 @@ inline void require_mpi_success(int code, std::string_view operation) { throw_mpi_error(code, operation); } +inline int chunk_capacity(int ranks) { + const int divisor = std::max(1, ranks); + return std::max(1, std::numeric_limits::max() / divisor); +} + +inline const char* chunk_pointer(const std::string& payload, unsigned long long offset, int count) { + if (count == 0) + return nullptr; + return payload.data() + static_cast(offset); +} + inline bool comm_active_unlocked() noexcept { int initialized = 0; int finalized = 0; diff --git a/include/pops/parallel/execution_lane.hpp b/include/pops/parallel/execution_lane.hpp index b811d904e..eea796b0e 100644 --- a/include/pops/parallel/execution_lane.hpp +++ b/include/pops/parallel/execution_lane.hpp @@ -560,11 +560,20 @@ class ObserverMpiLane { throw std::out_of_range("observer collective root is outside the lane"); const int me = lane.rank(); - std::optional> result; + long length_overflow = 0; + if constexpr (sizeof(std::size_t) > sizeof(unsigned long long)) { + if (payload.size() > static_cast(std::numeric_limits::max())) + length_overflow = 1; + } + if (all_reduce_max(length_overflow, lane) != 0) + throw std::overflow_error("consumer gather payload exceeds the MPI length domain"); + const unsigned long long local_length = static_cast(payload.size()); + + std::vector lengths; long allocation_failed = 0; if (me == root) { try { - result.emplace(static_cast(ranks)); + lengths.resize(static_cast(ranks), 0ULL); } catch (const std::bad_alloc&) { allocation_failed = 1; } catch (const std::length_error&) { @@ -572,25 +581,94 @@ class ObserverMpiLane { } } if (all_reduce_max(allocation_failed, lane) != 0) - throw std::runtime_error("observer root could not allocate gathered results"); + throw std::runtime_error("consumer root could not allocate gathered lengths"); + detail::require_mpi_success( + MPI_Gather(&local_length, 1, MPI_UNSIGNED_LONG_LONG, me == root ? lengths.data() : nullptr, + 1, MPI_UNSIGNED_LONG_LONG, root, lane.native_handle()), + "MPI_Gather(consumer payload lengths)"); - for (int source = 0; source < ranks; ++source) { - std::string source_payload; - long copy_failed = 0; - if (me == source) { + unsigned long long maximum_length = local_length; + detail::require_mpi_success( + MPI_Allreduce(MPI_IN_PLACE, &maximum_length, 1, MPI_UNSIGNED_LONG_LONG, MPI_MAX, + lane.native_handle()), + "MPI_Allreduce(maximum consumer gather length)"); + + std::optional> result; + std::vector counts; + std::vector displacements; + allocation_failed = 0; + if (me == root) { + try { + result.emplace(static_cast(ranks)); + counts.resize(static_cast(ranks), 0); + displacements.resize(static_cast(ranks), 0); + for (int rank = 0; rank < ranks; ++rank) { + const unsigned long long length = lengths[static_cast(rank)]; + if (length > static_cast(std::numeric_limits::max())) { + allocation_failed = 1; + break; + } + (*result)[static_cast(rank)].resize(static_cast(length)); + } + } catch (const std::bad_alloc&) { + allocation_failed = 1; + } catch (const std::length_error&) { + allocation_failed = 1; + } + } + if (all_reduce_max(allocation_failed, lane) != 0) + throw std::runtime_error("consumer root could not allocate gathered payloads"); + + const int capacity = detail::chunk_capacity(ranks); + for (unsigned long long offset = 0; offset < maximum_length; + offset += static_cast(capacity)) { + int total = 0; + if (me == root) { + for (int rank = 0; rank < ranks; ++rank) { + const unsigned long long length = lengths[static_cast(rank)]; + const int count = offset < length + ? static_cast(std::min( + length - offset, static_cast(capacity))) + : 0; + counts[static_cast(rank)] = count; + displacements[static_cast(rank)] = total; + total += count; + } + } + std::vector round; + long round_allocation_failed = 0; + if (me == root) { try { - source_payload = payload; + round.resize(static_cast(total)); } catch (const std::bad_alloc&) { - copy_failed = 1; + round_allocation_failed = 1; } catch (const std::length_error&) { - copy_failed = 1; + round_allocation_failed = 1; } } - if (all_reduce_max(copy_failed, lane) != 0) - throw std::runtime_error("an observer rank could not stage its gather payload"); - std::string received = broadcast_bytes(std::move(source_payload), source); - if (me == root) - (*result)[static_cast(source)] = std::move(received); + if (all_reduce_max(round_allocation_failed, lane) != 0) + throw std::runtime_error("consumer root could not allocate a gathered chunk"); + const int send_count = + offset < local_length + ? static_cast(std::min( + local_length - offset, static_cast(capacity))) + : 0; + detail::require_mpi_success( + MPI_Gatherv(detail::chunk_pointer(payload, offset, send_count), send_count, MPI_BYTE, + me == root ? round.data() : nullptr, me == root ? counts.data() : nullptr, + me == root ? displacements.data() : nullptr, MPI_BYTE, root, + lane.native_handle()), + "MPI_Gatherv(consumer payload chunk)"); + if (me != root) + continue; + for (int rank = 0; rank < ranks; ++rank) { + const int count = counts[static_cast(rank)]; + if (count == 0) + continue; + std::copy_n( + round.data() + displacements[static_cast(rank)], count, + (*result)[static_cast(rank)].data() + static_cast(offset)); + } } return result; #else diff --git a/include/pops/parallel/world_communicator.hpp b/include/pops/parallel/world_communicator.hpp index 830936ff4..e66d2fb8d 100644 --- a/include/pops/parallel/world_communicator.hpp +++ b/include/pops/parallel/world_communicator.hpp @@ -60,17 +60,6 @@ inline int validated_collective_root(int root) { return root; } -inline int chunk_capacity(int ranks) { - const int divisor = std::max(1, ranks); - return std::max(1, std::numeric_limits::max() / divisor); -} - -inline const char* chunk_pointer(const std::string& payload, unsigned long long offset, int count) { - if (count == 0) - return nullptr; - return payload.data() + static_cast(offset); -} - #endif } // namespace detail diff --git a/include/pops/runtime/amr_system.hpp b/include/pops/runtime/amr_system.hpp index 7600c0c01..9d45216b1 100644 --- a/include/pops/runtime/amr_system.hpp +++ b/include/pops/runtime/amr_system.hpp @@ -58,7 +58,7 @@ namespace pops { -class WorldCommunicator; +class ObserverMpiLane; namespace runtime::program { class AmrProgramContext; } @@ -668,7 +668,7 @@ class AmrSystem { /// Exact rank-local valid-cell pieces for one qualified field provider. The returned metadata /// explicitly marks replicated level-zero ownership so output modes never infer it from box counts. std::vector output_field_local_pieces(const std::string& provider_slot, int level); - std::vector output_field_root_pieces(const WorldCommunicator& world, + std::vector output_field_root_pieces(const ObserverMpiLane& lane, const std::string& provider_slot, int level); /// Transaction bracket used by the accepted-state reader after complete payload preflight. Every /// hierarchy, @@ -1039,7 +1039,7 @@ class AmrSystem { /// without allocating a global level buffer. std::vector output_state_local_pieces(const std::string& name, int k); std::vector output_geometry_boxes(); - std::vector output_state_root_pieces(const WorldCommunicator& world, + std::vector output_state_root_pieces(const ObserverMpiLane& lane, const std::string& name, int k); /// Owner rank per box of level @p k (the shared layout's DistributionMapping), aligned with the /// level-@p k rows of patch_boxes(). The v3 checkpoint (ADC-542) serializes it so a restart diff --git a/include/pops/runtime/output_piece_collective.hpp b/include/pops/runtime/output_piece_collective.hpp index 258c52778..57b30bb4a 100644 --- a/include/pops/runtime/output_piece_collective.hpp +++ b/include/pops/runtime/output_piece_collective.hpp @@ -5,10 +5,10 @@ /// /// Local providers are evaluated on every rank under an all-rank error consensus. Metadata and /// IEEE-754 values are framed in a versioned, endian-stable native wire payload and transferred by -/// WorldCommunicator's chunked MPI_Gatherv transport. Only rank zero materializes the global piece -/// vector; Python never gathers NumPy arrays or executes an MPI collective. +/// an explicitly owned consumer lane. Only rank zero materializes the global piece vector; Python +/// never gathers NumPy arrays or executes an MPI collective. -#include +#include #include #include @@ -185,13 +185,21 @@ inline std::string current_exception_text() { /// Evaluate a local OutputPiece provider and gather its exact result onto MPI rank zero. template -std::vector output_pieces_to_root(const WorldCommunicator& world, +std::vector output_pieces_to_root(const ObserverMpiLane& lane, std::string operation_identity, Provider&& provider) { - world.require_active_mpi_world(); - const int rank = world.rank(); +#ifndef POPS_HAS_MPI + (void)lane; + (void)operation_identity; + (void)provider; + throw std::runtime_error("native output-piece ROOT gather requires an MPI-enabled build"); +#endif + if (!lane.active()) + throw std::runtime_error( + "native output-piece root gather requires an active consumer MPI lane"); + const int rank = lane.rank(); - const std::vector operations = world.allgather_bytes(operation_identity); + const std::vector operations = lane.allgather_bytes(operation_identity); if (!std::all_of(operations.begin(), operations.end(), [&](const std::string& value) { return value == operation_identity; })) throw std::invalid_argument("output-piece root gather arguments differ across MPI ranks"); @@ -215,19 +223,19 @@ std::vector output_pieces_to_root(const WorldCommunicator& world, local_error = detail::current_exception_text(); } - const std::vector errors = world.allgather_bytes(local_error); + const std::vector errors = lane.allgather_bytes(local_error); for (std::size_t source = 0; source < errors.size(); ++source) { if (!errors[source].empty()) throw std::runtime_error("native output-piece provider failed on rank " + std::to_string(source) + ": " + errors[source]); } - const std::optional> gathered = world.gather_bytes(packed, 0); + const std::optional> gathered = lane.gather_bytes(packed, 0); std::vector result; std::string root_error; if (rank == 0) { try { - if (!gathered || gathered->size() != static_cast(world.size())) + if (!gathered || gathered->size() != static_cast(lane.size())) throw std::runtime_error("native output-piece root gather has invalid rank cardinality"); for (std::size_t source = 0; source < gathered->size(); ++source) { std::vector decoded = @@ -248,7 +256,7 @@ std::vector output_pieces_to_root(const WorldCommunicator& world, root_error = detail::current_exception_text(); } } - root_error = world.broadcast_bytes(std::move(root_error), 0); + root_error = lane.broadcast_bytes(std::move(root_error), 0); if (!root_error.empty()) throw std::runtime_error("native output-piece reconstruction failed: " + root_error); return result; diff --git a/include/pops/runtime/system.hpp b/include/pops/runtime/system.hpp index 50bab6654..bf2b2288c 100644 --- a/include/pops/runtime/system.hpp +++ b/include/pops/runtime/system.hpp @@ -47,7 +47,7 @@ namespace pops { -class WorldCommunicator; +class ObserverMpiLane; class PreparedSystemLayoutTransfer; namespace component { @@ -1333,9 +1333,9 @@ class System { std::vector output_field_local_pieces(const std::string& provider_slot, int level); /// Collective ROOT views. Local provider errors are agreed before native MPI_Gatherv; only rank /// zero receives complete pieces and every non-root rank receives an empty vector. - std::vector output_state_root_pieces(const WorldCommunicator& world, + std::vector output_state_root_pieces(const ObserverMpiLane& lane, const std::string& name, int level) const; - std::vector output_field_root_pieces(const WorldCommunicator& world, + std::vector output_field_root_pieces(const ObserverMpiLane& lane, const std::string& provider_slot, int level); /// @} diff --git a/python/bindings/core/init/init_amr.cpp b/python/bindings/core/init/init_amr.cpp index c9a3b55be..dc38a4c83 100644 --- a/python/bindings/core/init/init_amr.cpp +++ b/python/bindings/core/init/init_amr.cpp @@ -1,5 +1,5 @@ #include "../bindings_detail.hpp" -#include +#include #include "boundary_component_install.hpp" #include "output_geometry_binding.hpp" @@ -956,16 +956,16 @@ void bind_amr_data(py::class_& cls) { "Exact compact valid-cell pieces of one qualified field owned by this rank.") .def( "output_field_root_pieces", - [](AmrSystem& s, const WorldCommunicator& world, const std::string& provider_slot, + [](AmrSystem& s, const ObserverMpiLane& lane, const std::string& provider_slot, int level) { std::vector pieces; { py::gil_scoped_release release; - pieces = s.output_field_root_pieces(world, provider_slot, level); + pieces = s.output_field_root_pieces(lane, provider_slot, level); } return output_pieces_to_python(pieces); }, - py::arg("world"), py::arg("provider_slot"), py::arg("level"), + py::arg("lane"), py::arg("provider_slot"), py::arg("level"), "Collectively gather compact field pieces in C++; complete only on MPI rank zero.") .def( "_output_geometry_snapshot", @@ -1008,15 +1008,15 @@ void bind_amr_data(py::class_& cls) { "Exact compact valid-cell pieces of one qualified state owned by this rank.") .def( "output_state_root_pieces", - [](AmrSystem& s, const WorldCommunicator& world, const std::string& name, int level) { + [](AmrSystem& s, const ObserverMpiLane& lane, const std::string& name, int level) { std::vector pieces; { py::gil_scoped_release release; - pieces = s.output_state_root_pieces(world, name, level); + pieces = s.output_state_root_pieces(lane, name, level); } return output_pieces_to_python(pieces); }, - py::arg("world"), py::arg("block"), py::arg("level"), + py::arg("lane"), py::arg("block"), py::arg("level"), "Collectively gather compact state pieces in C++; complete only on MPI rank zero.") .def( "set_block_level_state", diff --git a/python/bindings/core/init/init_system.cpp b/python/bindings/core/init/init_system.cpp index 40153d799..8e93c6ee9 100644 --- a/python/bindings/core/init/init_system.cpp +++ b/python/bindings/core/init/init_system.cpp @@ -1,5 +1,5 @@ #include "../bindings_detail.hpp" -#include +#include #include "boundary_component_install.hpp" #include "output_geometry_binding.hpp" @@ -921,28 +921,27 @@ void bind_system_data(py::class_& cls) { "Exact compact valid-cell field pieces owned by this rank.") .def( "output_state_root_pieces", - [](const System& s, const WorldCommunicator& world, const std::string& block, int level) { + [](const System& s, const ObserverMpiLane& lane, const std::string& block, int level) { std::vector pieces; { py::gil_scoped_release release; - pieces = s.output_state_root_pieces(world, block, level); + pieces = s.output_state_root_pieces(lane, block, level); } return output_pieces_to_python(pieces); }, - py::arg("world"), py::arg("block"), py::arg("level"), + py::arg("lane"), py::arg("block"), py::arg("level"), "Collectively gather compact state pieces in C++; complete only on MPI rank zero.") .def( "output_field_root_pieces", - [](System& s, const WorldCommunicator& world, const std::string& provider_slot, - int level) { + [](System& s, const ObserverMpiLane& lane, const std::string& provider_slot, int level) { std::vector pieces; { py::gil_scoped_release release; - pieces = s.output_field_root_pieces(world, provider_slot, level); + pieces = s.output_field_root_pieces(lane, provider_slot, level); } return output_pieces_to_python(pieces); }, - py::arg("world"), py::arg("provider_slot"), py::arg("level"), + py::arg("lane"), py::arg("provider_slot"), py::arg("level"), "Collectively gather compact field pieces in C++; complete only on MPI rank zero.") .def( "_output_geometry_snapshot", diff --git a/python/pops/_pops.pyi b/python/pops/_pops.pyi index 559803ef5..e9a3dfe32 100644 --- a/python/pops/_pops.pyi +++ b/python/pops/_pops.pyi @@ -283,10 +283,10 @@ class System: self, provider_slot: str, level: int ) -> tuple[dict[str, object], ...]: ... def output_state_root_pieces( - self, world: _NativeWorldCommunicator, block: str, level: int + self, lane: _NativeObserverMpiLane, block: str, level: int ) -> tuple[dict[str, object], ...]: ... def output_field_root_pieces( - self, world: _NativeWorldCommunicator, provider_slot: str, level: int + self, lane: _NativeObserverMpiLane, provider_slot: str, level: int ) -> tuple[dict[str, object], ...]: ... @@ -309,10 +309,10 @@ class AmrSystem: self, provider_slot: str, level: int ) -> tuple[dict[str, object], ...]: ... def output_state_root_pieces( - self, world: _NativeWorldCommunicator, block: str, level: int + self, lane: _NativeObserverMpiLane, block: str, level: int ) -> tuple[dict[str, object], ...]: ... def output_field_root_pieces( - self, world: _NativeWorldCommunicator, provider_slot: str, level: int + self, lane: _NativeObserverMpiLane, provider_slot: str, level: int ) -> tuple[dict[str, object], ...]: ... diff --git a/python/pops/runtime/_runtime_consumers.py b/python/pops/runtime/_runtime_consumers.py index 0d7dbddd1..70327ad31 100644 --- a/python/pops/runtime/_runtime_consumers.py +++ b/python/pops/runtime/_runtime_consumers.py @@ -1288,6 +1288,7 @@ def __init__(self, owner: Any) -> None: self._rank, self._size, self._communicator = rank, size, communicator self._observer_queues: dict[tuple[str, str], PostCommitObserverQueue] = {} self._observer_lanes: dict[tuple[str, str], Any] = {} + self._root_output_lanes: dict[str, Any] = {} self._observer_workers: dict[str, PostCommitObserverWorker] = {} self._observer_journals: dict[tuple[str, str], Any] = {} self._observer_preflight_sessions: dict[str, Any] = {} @@ -1321,6 +1322,14 @@ def __init__(self, owner: Any) -> None: ) self._builtin_catalyst_consumers = tuple(sorted(builtin_catalyst)) self._builtin_catalyst_run_started = False + self._root_output_consumers = tuple( + sorted( + candidate.qualified_id + for candidate in owner._consumer_graph.nodes + if candidate.kind is ConsumerKind.SCIENTIFIC_OUTPUT + and candidate.parallel_mode is ParallelMode.ROOT + ) + ) from pops import interfaces for manifest in owner._consumer_graph.nodes: @@ -1789,6 +1798,21 @@ def begin_post_commit_consumers(self, run_identity: Identity) -> None: """ self._observer_key("run-begin", run_identity) + if run_identity.token in self._closed_observer_runs: + raise RuntimeError("post-commit consumers cannot reopen an already closed run") + if self._root_output_consumers: + if run_identity.token in self._root_output_lanes: + raise RuntimeError( + "the ROOT scientific-output MPI lane is already active for this run" + ) + if self._communicator is None: + raise RuntimeError( + "ROOT scientific output lost its authenticated execution communicator" + ) + lane_identity = "scientific-output/root/%s" % run_identity.token + self._root_output_lanes[run_identity.token] = ( + self._communicator.duplicate_observer_lane(lane_identity) + ) if self._builtin_catalyst_consumers: if self._builtin_catalyst_run_started: raise RuntimeError( @@ -2215,6 +2239,23 @@ def flush_live_visualizations( self._observer_diagnostics.append(rendered) failures.append(rendered) if close: + root_lane = self._root_output_lanes.pop(run_identity.token, None) + if self._root_output_consumers and root_lane is None: + rendered = "ROOT scientific-output MPI lane disappeared before close" + if rendered not in self._observer_diagnostics: + self._observer_diagnostics.append(rendered) + failures.append(rendered) + elif root_lane is not None: + try: + root_lane.close_collectively() + except BaseException as error: + rendered = ( + "ROOT scientific-output MPI lane close failed: %s" + % _exception_text(error) + ) + if rendered not in self._observer_diagnostics: + self._observer_diagnostics.append(rendered) + failures.append(rendered) worker = self._observer_workers.pop(run_identity.token, None) if worker is not None: try: @@ -2255,6 +2296,20 @@ def close_live_visualizations( run_identity, close=True, raise_on_failure=raise_on_failure ) + def _root_output_communicator(self) -> Any: + """Return the one active duplicated lane used by native ROOT snapshot gathers.""" + + if not self._root_output_consumers: + raise RuntimeError("the ConsumerGraph declares no ROOT scientific output") + if len(self._root_output_lanes) != 1: + raise RuntimeError( + "ROOT scientific output requires exactly one active run-scoped MPI lane" + ) + lane = next(iter(self._root_output_lanes.values())) + if lane.active is not True or lane.closed is not False: + raise RuntimeError("ROOT scientific-output MPI lane is not active") + return lane + def diagnostic_restart_state(self) -> dict[str, Any]: """Return the complete last-accepted typed diagnostic registry.""" baselines = dict(self._baselines) @@ -3241,10 +3296,20 @@ def _distributed_pieces( else method_name ) try: + native_communicator = communicator + if mode is ParallelMode.ROOT: + lane_provider = getattr( + self._owner._publisher, "_root_output_communicator", None + ) + if not callable(lane_provider): + raise RuntimeError( + "ROOT scientific output has no run-scoped MPI lane provider" + ) + native_communicator = lane_provider() local = self._local_pieces( native_engine, selected_method, - (communicator, *args) if mode is ParallelMode.ROOT else args, + (native_communicator, *args) if mode is ParallelMode.ROOT else args, mode=mode, rank=rank, require_local_owner=mode is not ParallelMode.ROOT, diff --git a/src/runtime/amr/amr_system.cpp b/src/runtime/amr/amr_system.cpp index ae23d0cd3..6c28de408 100644 --- a/src/runtime/amr/amr_system.cpp +++ b/src/runtime/amr/amr_system.cpp @@ -1751,11 +1751,11 @@ std::vector AmrSystem::output_field_local_pieces(const std::string& return p_->runtime->output_field_local_pieces(provider_slot, level); } -std::vector AmrSystem::output_field_root_pieces(const WorldCommunicator& world, +std::vector AmrSystem::output_field_root_pieces(const ObserverMpiLane& lane, const std::string& provider_slot, int level) { return output_pieces_to_root( - world, detail::output_collective_identity("AmrSystem", "field", provider_slot, level), + lane, detail::output_collective_identity("AmrSystem", "field", provider_slot, level), [&] { return output_field_local_pieces(provider_slot, level); }); } @@ -4327,9 +4327,9 @@ std::vector AmrSystem::output_geometry_boxes() { return p_->runtime->output_geometry_boxes(); } -std::vector AmrSystem::output_state_root_pieces(const WorldCommunicator& world, +std::vector AmrSystem::output_state_root_pieces(const ObserverMpiLane& lane, const std::string& name, int k) { - return output_pieces_to_root(world, + return output_pieces_to_root(lane, detail::output_collective_identity("AmrSystem", "state", name, k), [&] { return output_state_local_pieces(name, k); }); } diff --git a/src/runtime/system/system_fields.cpp b/src/runtime/system/system_fields.cpp index 460c04b42..f6c9ff4ac 100644 --- a/src/runtime/system/system_fields.cpp +++ b/src/runtime/system/system_fields.cpp @@ -918,19 +918,19 @@ std::vector System::output_field_local_pieces(const std::string& pr return output_local_pieces(field, 0, false); } -std::vector System::output_state_root_pieces(const WorldCommunicator& world, +std::vector System::output_state_root_pieces(const ObserverMpiLane& lane, const std::string& name, int level) const { - return output_pieces_to_root(world, + return output_pieces_to_root(lane, detail::output_collective_identity("System", "state", name, level), [&] { return output_state_local_pieces(name, level); }); } -std::vector System::output_field_root_pieces(const WorldCommunicator& world, +std::vector System::output_field_root_pieces(const ObserverMpiLane& lane, const std::string& provider_slot, int level) { return output_pieces_to_root( - world, detail::output_collective_identity("System", "field", provider_slot, level), + lane, detail::output_collective_identity("System", "field", provider_slot, level), [&] { return output_field_local_pieces(provider_slot, level); }); } diff --git a/tests/cpp/integration/mpi/test_mpi_amr_distributed_coarse.cpp b/tests/cpp/integration/mpi/test_mpi_amr_distributed_coarse.cpp index 4af988742..978113157 100644 --- a/tests/cpp/integration/mpi/test_mpi_amr_distributed_coarse.cpp +++ b/tests/cpp/integration/mpi/test_mpi_amr_distributed_coarse.cpp @@ -40,6 +40,7 @@ #include "amr_tagging_test_authority.hpp" #include +#include #include #include @@ -302,7 +303,10 @@ static Result run(int n, int nsteps, double dt, bool distribute) { // contract of level_{state,potential}_global(0). R.state = sys.level_state_global(0); R.output_local_pieces = sys.output_state_local_pieces("gas", 0); - R.output_root_pieces = sys.output_state_root_pieces(WorldCommunicator::world(), "gas", 0); + auto output_lane = + ObserverMpiLane::duplicate_world_collectively("test/amr-distributed-coarse/root-output"); + R.output_root_pieces = sys.output_state_root_pieces(output_lane, "gas", 0); + output_lane.close_collectively(); R.phi = sys.potential(); R.phi_global = sys.level_potential_global(0); R.mass = sys.mass(); diff --git a/tests/cpp/integration/mpi/test_mpi_system_io_gather.cpp b/tests/cpp/integration/mpi/test_mpi_system_io_gather.cpp index 918ca7c7b..de7f4a1d8 100644 --- a/tests/cpp/integration/mpi/test_mpi_system_io_gather.cpp +++ b/tests/cpp/integration/mpi/test_mpi_system_io_gather.cpp @@ -53,6 +53,7 @@ #include #include +#include #include #include @@ -144,12 +145,12 @@ static int pops_run_test_mpi_system_io_gather(int argc, char** argv) { // === T1 : gather == reference connue (np-invariant), sur le champ fraichement pose =========== // Tous les rangs appellent les accesseurs collectifs ; le resultat egale BIT-A-BIT la reference. + auto output_lane = ObserverMpiLane::duplicate_world_collectively("test/system-io/root-output"); { const std::vector dG = sys.density_global("gas"); const std::vector sG = sys.state_global("gas"); const std::vector local = sys.output_state_local_pieces("gas", 0); - const std::vector root = - sys.output_state_root_pieces(WorldCommunicator::world(), "gas", 0); + const std::vector root = sys.output_state_root_pieces(output_lane, "gas", 0); chk(dG.size() == nn, "T1_density_global_size"); chk(sG.size() == 4 * nn, "T1_state_global_size"); chk(dG == rho_ref, "T1_density_global_eq_ref_no_double_count"); @@ -172,6 +173,7 @@ static int pops_run_test_mpi_system_io_gather(int argc, char** argv) { chk(piece.ncomp == 4 && piece.values == sG, "T1_output_state_root_values"); } } + output_lane.close_collectively(); // === T2 : apres des pas COLLECTIFS, gather == accesseur local sur le proprietaire ============ const double dt = 0.01; diff --git a/tests/cpp/unit/parallel/test_world_communicator.cpp b/tests/cpp/unit/parallel/test_world_communicator.cpp index 7c6028cc8..eca1caed8 100644 --- a/tests/cpp/unit/parallel/test_world_communicator.cpp +++ b/tests/cpp/unit/parallel/test_world_communicator.cpp @@ -111,12 +111,12 @@ TEST(WorldCommunicator, TransfersEmptyNullAndVariableSizedBytes) { } TEST(WorldCommunicator, GathersOutputPiecesOnlyOnRoot) { - pops::WorldCommunicator& world = pops::WorldCommunicator::world(); + auto lane = pops::ObserverMpiLane::duplicate_world_collectively("test/output-piece/gather"); #ifdef POPS_HAS_MPI - const int rank = world.rank(); - const int size = world.size(); + const int rank = lane.rank(); + const int size = lane.size(); std::vector result = pops::output_pieces_to_root( - world, pops::detail::output_collective_identity("test", "state", "tracer", 0), [rank] { + lane, pops::detail::output_collective_identity("test", "state", "tracer", 0), [rank] { pops::OutputPiece piece; piece.box = pops::PatchBox{0, rank, 0, rank, 0}; piece.global_box_index = rank; @@ -141,18 +141,19 @@ TEST(WorldCommunicator, GathersOutputPiecesOnlyOnRoot) { } #else EXPECT_THROW((void)pops::output_pieces_to_root( - world, pops::detail::output_collective_identity("test", "state", "tracer", 0), + lane, pops::detail::output_collective_identity("test", "state", "tracer", 0), [] { return std::vector{}; }), std::runtime_error); #endif + lane.close_collectively(); } TEST(WorldCommunicator, SelectsOneCanonicalReplicatedOutputContributor) { - pops::WorldCommunicator& world = pops::WorldCommunicator::world(); + auto lane = pops::ObserverMpiLane::duplicate_world_collectively("test/output-piece/replicated"); #ifdef POPS_HAS_MPI - const int rank = world.rank(); + const int rank = lane.rank(); std::vector result = pops::output_pieces_to_root( - world, pops::detail::output_collective_identity("test", "state", "replicated", 0), [rank] { + lane, pops::detail::output_collective_identity("test", "state", "replicated", 0), [rank] { pops::OutputPiece piece; piece.box = pops::PatchBox{0, 0, 0, 0, 0}; piece.global_box_index = 0; @@ -173,10 +174,10 @@ TEST(WorldCommunicator, SelectsOneCanonicalReplicatedOutputContributor) { EXPECT_TRUE(result.empty()); } #else - EXPECT_THROW( - (void)pops::output_pieces_to_root( - world, pops::detail::output_collective_identity("test", "state", "replicated", 0), - [] { return std::vector{}; }), - std::runtime_error); + EXPECT_THROW((void)pops::output_pieces_to_root( + lane, pops::detail::output_collective_identity("test", "state", "replicated", 0), + [] { return std::vector{}; }), + std::runtime_error); #endif + lane.close_collectively(); } diff --git a/tests/python/architecture/test_root_output_consumer_lane_fence.py b/tests/python/architecture/test_root_output_consumer_lane_fence.py new file mode 100644 index 000000000..e48dbe16f --- /dev/null +++ b/tests/python/architecture/test_root_output_consumer_lane_fence.py @@ -0,0 +1,46 @@ +"""ADC-683 fences for run-owned native ROOT scientific-output communication.""" + +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[3] +COLLECTIVE = ROOT / "include/pops/runtime/output_piece_collective.hpp" +SYSTEM = ROOT / "include/pops/runtime/system.hpp" +AMR = ROOT / "include/pops/runtime/amr_system.hpp" +SYSTEM_BINDING = ROOT / "python/bindings/core/init/init_system.cpp" +AMR_BINDING = ROOT / "python/bindings/core/init/init_amr.cpp" +RUNTIME = ROOT / "python/pops/runtime/_runtime_consumers.py" +STUB = ROOT / "python/pops/_pops.pyi" + + +def test_native_root_output_surface_requires_an_owned_consumer_lane(): + collective = COLLECTIVE.read_text(encoding="utf-8") + system = SYSTEM.read_text(encoding="utf-8") + amr = AMR.read_text(encoding="utf-8") + + assert "WorldCommunicator" not in collective + assert "MPI_COMM_WORLD" not in collective + assert "const ObserverMpiLane& lane" in collective + assert "const ObserverMpiLane& lane" in system + assert "const ObserverMpiLane& lane" in amr + + +def test_python_root_output_bridge_rejects_the_process_world_type(): + system = SYSTEM_BINDING.read_text(encoding="utf-8") + amr = AMR_BINDING.read_text(encoding="utf-8") + stub = STUB.read_text(encoding="utf-8") + + assert "WorldCommunicator" not in system + assert "WorldCommunicator" not in amr + assert "const ObserverMpiLane& lane" in system + assert "const ObserverMpiLane& lane" in amr + assert "lane: _NativeObserverMpiLane" in stub + + +def test_runtime_materializes_and_closes_one_root_output_lane_per_run(): + runtime = RUNTIME.read_text(encoding="utf-8") + + assert 'lane_identity = "scientific-output/root/%s" % run_identity.token' in runtime + assert "self._communicator.duplicate_observer_lane(lane_identity)" in runtime + assert "root_lane.close_collectively()" in runtime + assert "native_communicator = lane_provider()" in runtime diff --git a/tests/python/unit/runtime/test_runtime_instance_gate.py b/tests/python/unit/runtime/test_runtime_instance_gate.py index 52941529b..7b549d975 100644 --- a/tests/python/unit/runtime/test_runtime_instance_gate.py +++ b/tests/python/unit/runtime/test_runtime_instance_gate.py @@ -259,13 +259,13 @@ def output_state_local_pieces(self, block, level): ) def output_state_root_pieces(self, communicator, block, level): - """Expose the exact singleton-world gather required by ROOT publication tests.""" - from pops._native_collectives import require_world, size + """Expose the exact duplicated consumer lane required by ROOT publication tests.""" + from pops._native_collectives import require_communicator, size expected = self._plan.execution_context.communicator - if communicator is not expected.handle: - raise ValueError("ROOT gather did not receive the installed communicator handle") - native = require_world(communicator) + native = require_communicator(communicator, allow_world=False) + if expected.identity != "MPI_COMM_WORLD": + raise ValueError("ROOT gather requires an MPI execution context") if size(native) != 1: raise RuntimeError( "runtime-instance unit executor only implements a singleton ROOT gather" @@ -1731,6 +1731,83 @@ def test_checkpoint_diagnostic_baseline_schema_is_finite_and_canonical(): ) +def test_root_output_lane_requires_one_active_run_scoped_communicator(): + from pops.runtime._runtime_consumers import RuntimeConsumerPublisher + + publisher = object.__new__(RuntimeConsumerPublisher) + lane = SimpleNamespace(active=True, closed=False) + publisher._root_output_consumers = ("scientific_output/root",) + publisher._root_output_lanes = {"run": lane} + assert publisher._root_output_communicator() is lane + + publisher._root_output_lanes = {} + with pytest.raises(RuntimeError, match="exactly one active"): + publisher._root_output_communicator() + + publisher._root_output_lanes = {"run": SimpleNamespace(active=False, closed=False)} + with pytest.raises(RuntimeError, match="not active"): + publisher._root_output_communicator() + + publisher._root_output_consumers = () + with pytest.raises(RuntimeError, match="declares no ROOT"): + publisher._root_output_communicator() + + +def test_root_output_lane_is_materialized_and_closed_once_per_run(): + from pops.runtime._runtime_consumers import RuntimeConsumerPublisher + + class _Lane: + active = True + closed = False + + def __init__(self): + self.close_calls = 0 + + def close_collectively(self): + self.close_calls += 1 + self.active = False + self.closed = True + + class _World: + def __init__(self, lane): + self.lane = lane + self.identities = [] + + def duplicate_observer_lane(self, identity): + self.identities.append(identity) + return self.lane + + run_identity = make_identity("run", {"case": "root-output-lane"}) + lane = _Lane() + world = _World(lane) + publisher = object.__new__(RuntimeConsumerPublisher) + publisher._root_output_consumers = ("scientific_output/root",) + publisher._root_output_lanes = {} + publisher._communicator = world + publisher._closed_observer_runs = set() + publisher._builtin_catalyst_consumers = () + publisher._builtin_catalyst_run_started = False + publisher._owner = SimpleNamespace( + _consumer_graph=SimpleNamespace(nodes=()), + ) + publisher._observer_diagnostics = [] + publisher._observer_workers = {} + publisher._observer_reports = {} + publisher._observer_queues = {} + publisher._observer_pending_failures = {} + + publisher.begin_post_commit_consumers(run_identity) + assert world.identities == ["scientific-output/root/%s" % run_identity.token] + assert publisher._root_output_communicator() is lane + + assert publisher.close_live_visualizations(run_identity) == () + assert lane.close_calls == 1 + assert publisher.close_live_visualizations(run_identity) == () + assert lane.close_calls == 1 + with pytest.raises(RuntimeError, match="already closed"): + publisher.begin_post_commit_consumers(run_identity) + + def test_diagnostic_component_requires_one_explicit_role_for_multicomponent_state(): from pops.runtime._runtime_consumers import RuntimeConsumerPublisher