Load-balance, active-box windowing, block-structured AMR - #1628
Open
sbryngelson wants to merge 439 commits into
Open
Load-balance, active-box windowing, block-structured AMR#1628sbryngelson wants to merge 439 commits into
sbryngelson wants to merge 439 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces an opt-in (“default-off”) family of performance/diagnostic features (load-weight and SFC partition diagnostics, weighted init-time decomposition, rank timing), plus major simulation capabilities (active-box RHS windowing and block-structured AMR) and corresponding post-processing support and documentation/validation updates.
Changes:
- Adds new runtime parameters and toolchain metadata/validation hooks for the experimental performance/AMR feature family.
- Extends the simulation code with new modules for active-box restriction, load-weight diagnostics, SFC partition reporting, rank timing, and AMR integration points (including restart/output plumbing).
- Updates post_process to read/write AMR fine-block overlays and adds/updates golden metadata plus documentation/indexing.
Reviewed changes
Copilot reviewed 82 out of 94 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| toolchain/mfc/params/descriptions.py | Adds user-facing descriptions for new experimental/performance parameters. |
| toolchain/mfc/params/definitions.py | Registers new parameters (AMR, hybrid sensors, load-balance diagnostics) and target applicability. |
| toolchain/mfc/lint_docs.py | Treats new validator checks as non-physics doc checks. |
| tests/F980C769/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ECABA006/golden-metadata.txt | Adds golden metadata for active-box test coverage. |
| tests/DD4CD8F3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/CC4213FD/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BD21A5C0/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BCBA6E74/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ACE05393/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/987D9025/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/852CCB81/golden-metadata.txt | Adds golden metadata for AMR-related golden tests. |
| tests/65C375B4/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/4DADE04B/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/454C565F/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/3A474BEE/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/2FC423D3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/13945217/golden-metadata.txt | Adds golden metadata for moving-IB under AMR test coverage. |
| src/simulation/m_viscous.fpp | Clamps FD coefficient indexing to avoid ghost-region coefficient OOB in IB drag gradient evaluation. |
| src/simulation/m_time_steppers.fpp | Integrates active-box bounds into RK update loops and interleaves AMR fine-stage/subcycle operations. |
| src/simulation/m_start_up.fpp | Wires up new modules (rank timing, active-box, load-weight, SFC partition, AMR) into init/timestep/finalize and restart I/O. |
| src/simulation/m_sfc_partition.fpp | Adds analysis-only SFC tiling + weighted partition prediction and reporting. |
| src/simulation/m_rank_timing.fpp | Adds per-rank wall-time imbalance measurement helpers and reporting. |
| src/simulation/m_load_weight.fpp | Adds per-cell load-weight field construction and rank-level imbalance reporting. |
| src/simulation/m_hypoelastic.fpp | Refactors FD coefficient setup into a callable update routine (supporting AMR grid swaps). |
| src/simulation/m_global_parameters.fpp | Adds AMR working-state mirrors and slot selection helper plus defaults for new parameters. |
| src/simulation/m_data_output.fpp | Adds output/report hooks for load-weight, SFC partition, and rank-time diagnostics. |
| src/simulation/m_checker.fpp | Adds input validation/prohibits for active-box, hybrid sensors, load-balance, and AMR configurations. |
| src/simulation/m_active_box.fpp | Adds active-box initialization/growth and debug envelope checking. |
| src/simulation/m_acoustic_src.fpp | Adds AMR-aware handling of acoustic source support (bounding boxes and overlap abort). |
| src/post_process/m_start_up.fpp | Calls AMR fine-data reader and AMR overlay writer when amr is enabled. |
| src/post_process/m_global_parameters.fpp | Adds default-off amr flag for post_process overlay behavior. |
| src/post_process/m_data_output.fpp | Implements AMR fine-block overlay mesh/variables output (Silo/binary) and multimesh registration. |
| src/common/m_phase_change.fpp | Exposes per-cell Newton iteration count and threads it through relaxation to support load-weighting. |
| src/common/m_global_parameters_common.fpp | Adjusts start_idx lifecycle/allocation and makes load_weight_wrt visible to GPU macros. |
| src/common/m_derived_types.fpp | Introduces a simple t_box type used by new partitioning infrastructure. |
| src/common/m_box.fpp | Adds box/partition arithmetic helpers (equal/weighted splits, box-from-splits). |
| src/common/m_boundary_common.fpp | Skips BC buffer population during AMR fine advance to rely on coarse-driven ghost fill. |
| docs/module_categories.json | Registers new modules under documentation categories. |
| docs/documentation/readme.md | Adds AMR section link to the documentation index. |
| .typos.toml | Adds project-specific abbreviations to the spelling allowlist. |
| D = ((gs_min(lp) - 1.0_wp)*cvs(lp))/((gs_min(vp) - 1.0_wp)*cvs(vp)) | ||
|
|
||
| #ifdef MFC_SIMULATION | ||
| if (relax .and. load_weight_wrt) then |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #1628 +/- ##
==========================================
+ Coverage 61.04% 62.12% +1.07%
==========================================
Files 83 93 +10
Lines 20978 25578 +4600
Branches 3099 4189 +1090
==========================================
+ Hits 12807 15890 +3083
- Misses 6126 6972 +846
- Partials 2045 2716 +671 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Member
Author
|
Upstream latent gap found during the MHD+AMR investigation (independent of this PR): Also for the record: MHD+AMR was attempted and re-gated on measured evidence rather than assumption — the coarse/fine seam is a continuous O(1) div(B) source that cleaning spreads but cannot remove (details in the amr.md support matrix row and commit ac203b1). |
…_amr_restart (parallel_io) required the writing rank count; it now REPARTITIONS. Whole-block ownership makes each block's fine data one contiguous region-sized chunk at a computable offset - the only rank-count dependence in the file layout is the 3*np_old per-block extents record. So: read the writer's np_old from the header, use it for the file-layout strides (blk_base + data offset), re-assign block owners for THIS run via s_amr_assign_block_owners, and each new owner reads its whole blocks (the EXSCAN yields offset 0 for the block's sole current owner). The same-decomposition per-rank layout check is skipped when np_old /= num_procs (it cannot match by design; the end-of-read file-size check still fails closed on truncation/corruption). np_old == num_procs is byte-identical to the previous same-rank path. Serial (non-parallel_io) mode still needs the writing rank count (message now points to parallel_io for repartition). NO write-side change - existing checkpoints are already repartition-readable. VALIDATED CPU reldebug -b mpirun (1D AMR dynamic-regrid, save at step 3, restart 3->6): np=2->np=1 AND np=1->np=2 both print the repartition line, restore the fine level (2 blocks), and conserve (mass 8e-13 / energy 1e-12); 5EFB3277 (np=2 same-rank parallel restart) still passes.
…e_slot (no-op) - Foundation for lazy owned-only slot sizing (right-size a rank's fine memory to ~1/num_procs of the pool instead of all amr_max_blocks slots on every rank). Move the per-slot allocation (coords + the 6 device-resident field vectors + non-poly QBMM side-state) out of the init loop and the matching deallocation out of finalize into s_amr_alloc_slot(islot) / s_amr_free_slot(islot); promote the sizing locals (max_f1/2/3, mbuf1..3_lo/hi) to module scope so the helpers see them; add amr_slot_live(:) to track which slots are allocated. Init/finalize still call the helpers for EVERY slot, and the single QBMM RHS scratch (amr_rhs_pb_f/mv_f) is allocated once outside the per-slot helper - so this commit is bit-identical; the lazy reconcile at regrid/init/restart follows. VALIDATED CPU reldebug -b mpirun: 21C71558 + 6C20B752 (np=1 IGR) + 5EFB3277 (np=2) all pass.
…um_procs - Build on the alloc/free helpers (daa36af): a rank now allocates a slot's per-block field arrays (coords + the 6 device-resident field vectors + non-poly QBMM side-state, all sys_size x block) ONLY for blocks it owns, instead of all amr_max_blocks slots on every rank. s_amr_reconcile_slots allocates owned / frees the rest, called once ownership is known: init (after s_amr_assign_block_owners), restart (serial + parallel, before the owned read), and regrid. The regrid keeps the transient slots live across the critical section - old-owned (for the P2P migration SEND) + newly-owned (for the build) + received-old (alloc'd before the q_cons_stor unpack) - then reconciles to new-owned-only at the end, freeing the rest. Safe because every heavy-array access is ownership-guarded (fill/advance early-return on .not. amr_rank_owns_block), so an unowned/freed slot is never touched; np=1 owns all blocks so it is bit-identical (only the inactive pool slots beyond amr_num_blocks go unallocated). Per-rank fine memory now scales ~1/num_procs instead of num_procs x the data. (IB marker fields are allocated en masse by m_ibm and are per-cell integers, not sys_size x block - left global.) VALIDATED: CPU reldebug np=1 bit-identical (21C71558/6C20B752 IGR), np=2 (79B334C7 stretched regrid+migration, 5EFB3277, BD21A5C0 chem), repartition restart 2->1 conserves (mass 8e-13); GPU mp (Phoenix V100, clean build) 5/5 (21C71558/6C20B752/79B334C7/5EFB3277/660FFBFE) - device present-table consistent through the lazy alloc/free at regrid.
…pped) - The SFC map distributes WHOLE blocks, so with fewer blocks than ranks some ranks own no fine work - the fine-level load balance is capped at min(num_blocks, amr_max_blocks)/num_procs. Surfaced by the np>=4 load-balance demonstration: a concentrated region balanced PERFECTLY (1.00x imbalance, exact conservation) at np=4 (4 blocks) and np=8 (8 blocks with amr_max_blocks=16), but the default amr_max_blocks=4 capped np=8 at 4 blocks -> 2.0x. Add a rank-0 init warning when amr_max_blocks < num_procs pointing the user to raise it. np=1/small runs unaffected (fires only when num_procs > amr_max_blocks).
…-compiler flips The 4 hybrid-sensor golden tests at hybrid_weno_eps=0.5 (BA4340EA, 60739A3E and the AMR 053C5DDA, DDC4BA8A) failed on aggressive-optimizer lanes (intel no-debug, NVHPC 24.1/24.3, Frontier GPU) with percent-level LOCAL diffs. Root cause: the Jameson sensor's per-cell phi = |c+ - 2c0 + c-|/(|c+|+2|c0|+|c-|) is catastrophic cancellation at a shock, so a compiler's FP contraction/reordering jitters phi; where phi sits NEAR eps a cell flips between central and WENO -> a large local change. eps=0.5 sat at the edge of a plateau, ~0.001 from a shock cell at phi~=0.499. Fix is not a tolerance bump but a well-placed threshold. An eps-sweep of each exact case (run at many eps, diff outputs) shows the output is bit-identical across eps in [0.20, 0.40] -> no cell's phi lands in that band on ANY timestep = a clean phi GAP. eps=0.30 is centred there with a +/-0.10 margin: the full cross-compiler SOLUTION drift (~5e-4) moves phi only ~1e-3, ~100x inside the margin, so no compiler can flip a cell. The sensor still fires consequentially (answer moves ~4e-4 vs a dead all-WENO sensor, 7-10x the 5e-5 tol). Same gap holds for weno and riemann modes and on the AMR fine level (all measured). VALIDATED (Phoenix GNU CPU): 4/4 pass vs regenerated goldens under the normal build AND --fastmath (aggressive FP reordering). Robustness is by construction (measured phi gap); the intel/NVHPC-24.x lanes are the final confirmation in CI.
CCE OpenMP-offload never initializes the descriptor of the bare module-scope scalar_field allocatable amr_cg, so allocate(amr_cg) aborts with lib-4425 at startup - crashing every AMR case on gpu-mp/gpu-omp/AMD-flang. Allocate a local (which gets a valid descriptor) and move_alloc it into amr_cg, then map. OpenACC unaffected.
s_find_ghost_points assigns ghost-point slots via GPU atomic-capture (thread-completion order), which is nondeterministic; the order-sensitive surface-force reduction and the discrete image-point stencil then make moving AMR-IB diverge across backends. Sort the list on-device into lexicographic (i,j,k) order - no host round-trip, since a GPU_UPDATE here fails Cray's present-table lookup and this routine runs mid-swap in the AMR fine path. Also drops the routine's dummy argument to use the module-global ghost_points directly. Verified: tests/13945217 fails at 1e-10 without the sort, passes with it.
# Conflicts: # toolchain/mfc/test/cases.py
The AMR fine/coarse ghost-point swap copied the declare-target ghost_points through the host (GPU_UPDATE host -> host park -> GPU_UPDATE device) only because the parking storage ib_fine%gps was a plain host array. An audit confirmed the swap park is the ONLY host access of ghost_points: every consumer (correction kernel, s_find_ghost_points, levelset, image points, interpolation coeffs, moving recompute) runs on-device, the collision-forces ghost_points arg is dead, and there is no cross-rank exchange. Replace ib_fine%gps with a device-resident gp_park(:,:) and swap via on-device kernels, eliminating the host round-trip. gp_park is mapped dynamically with move_alloc + GPU_ENTER_DATA (the amr_cg idiom) so the bare derived-type allocatable gets a valid descriptor and device mapping without the CCE OpenMP-offload lib-4425 uninitialized-descriptor abort; a direct GPU_DECLARE + @:ALLOCATE would corrupt a sibling descriptor. ghost_point is trivially copyable, so the device struct assignment is well-defined. This supersedes the ghost_points(1:num_gps) slice band-aid: removing the whole-array ghost_points GPU_UPDATE also fixes the amdflang IBM+AMR offload crash (per-element custom mapper -> ROCm HSA OUT_OF_RESOURCES) at its source, without keeping the host round-trip. Verified 4/4 IBM-AMR cases (static + moving) on Frontier gpu-mp (CCE OpenMP-offload), gpu-acc (OpenACC), and CPU.
…FlowCode#1636) The AMR moving-IBM-circle test (13945217) ran a wall at vel=0.2 for 10 steps, long/fast enough to develop the spurious-pressure / CFL blow-up inherent to MFC's moving-IB method (no geometric-conservation-law treatment). The marginal instability NaNs on OpenMP-offload (CCE gpu-mp and amdflang) while passing on OpenACC and CPU. The real fix (a GCL/cut-cell treatment) is tracked in MFlowCode#1636 and deferred. Soften the case so it stays sub-marginal on all backends while still exercising the per-substage fine-IB recompute: slow the wall to vel=0.02 and stop at 4 steps (matching the two-body case's short-run pattern). Regenerated goldens. Verified 13945217 passes on gpu-mp (CCE OpenMP-offload), gpu-acc (OpenACC), and CPU.
…sent-table leak) s_amr_free_slot freed each slot's fields with @:DEALLOCATE only, but @:ACC_SETUP_SFs maps two things per scalar field on Cray - the descriptor ([arg]) and the field data ([arg%sf]) via GPU_ENTER_DATA copyin. Since Cray 'exit data delete' decrements the reference counter, the lone @:DEALLOCATE(arg%sf) undoes only the @:ALLOCATE create ref, leaving the descriptor and the ACC_SETUP %sf copyin ref dangling in the present table. Latent at program exit, but s_amr_free_slot frees slots MID-RUN (regrid / restart / reconcile), so the leaked host addresses get reused by the next allocation. On the AMR restart path the reused address is Gs_rs (Riemann init runs after s_read_amr_restart frees the init slot), tripping 'CRAY_ACC_ERROR - Error placing gs_rs: another instance already present' and killing the run (test 5EFB3277, AMR 1D dynamic regrid, 2 MPI ranks, gpu-acc). Add @:ACC_TEARDOWN_SFs (exact inverse of @:ACC_SETUP_SFs) and call it before each field's @:DEALLOCATE in s_amr_free_slot so every ACC_SETUP enter is matched by a delete. Cray-only (#ifdef _CRAYFTN), so non-Cray is unchanged. Verified 5EFB3277 passes on gpu-acc (2 ranks + restart roundtrip) and CPU.
The single-body moving-IBM case (13945217) was slowed to vel=0.02 in 4c877aa to dodge the un-GCL'd blow-up (MFlowCode#1636). The two-body case is pushed on top of it in the CaseGeneratorStack WITHOUT a pop, so it inherited that change - silently dropping body 1's wall speed 0.2 -> 0.02 and invalidating 43AF9F25's golden (which was generated at 0.2). That regressed 43AF9F25 across all backends (the softened setup diverges more on CPU than on GPU, hence the confusing CPU 91% vs gpu-acc 1.4x-tol spread). Restore body 1's vel=0.2 in the two-body push so the softening stays scoped to 13945217. 43AF9F25 is unchanged from its known-good config; no golden regen. Verified 43AF9F25 passes against its existing golden on CPU, gpu-mp (CCE OpenMP-offload), and gpu-acc (OpenACC).
…igns (amdflang shape crash) On the non-case-optimized amdflang path (#:if not MFC_CASE_OPTIMIZATION and USING_AMD), weno reconstruction scratch arrays are sized to the max 0:4 since weno_num_stencils is not a compile constant. The six central-weight assignments 'omega(:) = d_cbL/R(:, sv)' then assign the 0:weno_num_stencils-shaped d_cb (3 elements for WENO5) into the whole 0:4 omega (5), which amdflang's runtime array-shape check aborts on: 'Assign: mismatching element counts, to 5 from 3' (SIGABRT, exit 134). CCE/nvfortran use the exact-sized #:else path so never hit it. Only 0:weno_num_stencils of omega is meaningful, so slice the target. Surfaced by the hybrid_weno sensor's use_central branch: fixes the amdflang gpu-omp crashes on 71E57E55, 053C5DDA, BA4340EA (verified pass on amdflang; CCE/nvfortran/CPU unaffected - the sliced target is identical on the exact path).
master MFlowCode#1572 decomposed the HLLC hot path into shared m_riemann_state device helpers. Reconcile up/mega's hybrid-smooth-flux feature onto that refactor and, per request, extend it from hllc-only to all Riemann solvers. Factor the central (hybrid_smooth_flux=1) / Rusanov (=2) smooth-flux block into one shared device subroutine s_compute_hybrid_smooth_flux in m_riemann_state (no Fypp macro), called by hll/hllc/lf at each WENO-smooth face using the states the solver already computed. hlld reuses its own F_L/F_R/U_L/U_R for the MHD Rusanov override. Relax the hllc-only and MHD hybrid_riemann prohibitions in m_checker. Add hll/lf/hlld hybrid_riemann golden tests.
chi_vw/k_mw/rho_mw were module-scope GPU_DECLARE'd scalars used as per-cell scratch inside the s_compute_bubble_EE_source parallel loop (s_bwproperty writes them; s_vflux/f_bpres_dot read them via host association) but were not private, so GPU threads clobbered each other's values between write and read - a scheduling-dependent error (amdflang 3.6e-7, CCE-OMP 1.6e-9) on the nonpolytropic partial pressures. Thread them through arguments instead (fchi_vw/frho_mw to s_vflux, fk_mw to f_bpres_dot) with per-thread locals, and delete the module scalars. Validated: 454C565F now passes at its 5e-9 tol on amdflang.
The consequential hybrid_weno_eps=0.3 is tuned to HLLC's Jameson-phi gap; reusing it for other solvers puts a cell at the sensor threshold that flips smooth/discontinuous under FP reordering (~2% CPU-vs-GPU). Use the liveness eps (1e-2) for the hll and hlld hybrid golden tests (goldens regenerated on CPU, verified on amdflang). LF stays enabled for hybrid_riemann and shares s_compute_hybrid_smooth_flux (covered by the HLLC consequential + HLL cases), but its heavy base dissipation minus the reduced-dissipation smooth flux amplifies FP noise from any gradient, so no non-trivial LF case yields a backend-stable golden - dropped.
master MFlowCode#1572 replaced HLL's G > 1000 elastic-energy stability floor with G > verysmall. For a cont_damage interface, a heavily-damaged state drives G -> 0 while the reconstructed stress does not relax with it, so tau^2/(4G) blows up - finite (and negligible) on CCE/Linux-gfortran but NaN under macOS gfortran's libm, crashing AMR -> 1D -> hypoelastic static block -> cont_damage (D731AB7A). Gate the elastic-energy term with merge(1.e3, verysmall, cont_damage): keeps master's verysmall ruling for undamaged states (G >> floor) and restores the stability floor only under damage. All 6 cont_damage tests pass on CCE.
Extending hybrid_riemann to LF via the shared s_compute_hybrid_smooth_flux helper diverged ~2% across compilers (CPU/CCE-GPU/amdflang): LF's flux carries a low-Mach pcorr term and a distinct advection form the generic Rusanov helper lacks, so overwriting only WENO-smooth faces injected a sensor-flip discontinuity. Instead, under hybrid_riemann LF simply switches its dissipation to the local (normal-velocity) wave speed and does NOT call the helper - LF becomes local Lax-Friedrichs everywhere, a single consistent scheme (and a genuine accuracy upgrade over global-LF). Re-added the LF hybrid golden; passes on CPU and CCE-GPU (was ~2% off before).
Under --case-optimization, CCE emits a standalone OMP device-routine clone of s_compute_hybrid_smooth_flux and rejects it (ftn-7066: 'Global in accelerator routine') because the clone cannot resolve hybrid_smooth_flux - a declare-target scalar from the shared common module. This failed the Case-Opt gpu-omp CI build (the Pre-Build SLURM step) for every benchmark case. Pass hybrid_smooth_flux as a scalar argument instead of reading the module global, mirroring the existing Re_size_loc pattern for common-module globals that misbehave in device-routine clones. Pure refactor (identical result); the flux-array writes were never the problem. Builds under case-opt CCE-OMP; all 10 hybrid regression tests pass on CPU.
… not work Traced the hot path: exactly one s_amr_swap_to_fine / s_amr_restore_coarse pair per block per RK stage, wrapping s_compute_rhs in s_amr_fine_stage_rhs. The RK pass and the fill phase do not swap. So a timestep costs 3*nblocks swap round-trips, each carrying s_amr_sync_grid_state_to_device plus s_amr_recompute_weno_coefs on nonuniform grids. Also records the obvious optimization that is NOT available: hoisting the restore so consecutive blocks swap fine->fine fails because m_time_steppers calls s_amr_p2p_reflux_faces and s_amr_apply_reflux between blocks inside the advance loop, both in the coarse frame. Any batching design has to move the reflux out of the per-block loop or make it frame-independent first. Docs only. precheck clean.
…erequisite) The fine stage advanced and refluxed each block in one loop: s_amr_fine_stage_advance runs with the block's grid globals swapped in, then s_amr_p2p_reflux_faces / s_amr_apply_reflux write the coarse rhs_vf at coarse indices. Interleaving the two frames forces a swap/restore round trip per block per RK stage, and it is what prevents consecutive blocks from swapping fine->fine. Phase 3 now advances every block; phase 4 refluxes the level-1 blocks afterwards in the coarse frame. Behavior-preserving: no block's advance reads rhs_vf, the regrid merge invariant keeps blocks >= buff_size apart so their c/f corrections are disjoint, and every rank still visits the same slots in the same order, preserving the collective ordering of s_amr_p2p_reflux_faces. level>=2 blocks still skip L0 reflux (they apply a state reflux into their parent after the stage loop). This does not by itself reduce the measured per-block cost - it removes the structural obstacle to doing so, which docs/documentation/amr_block_batching.md names as the prerequisite for batching the advances. Validated on hpcfund (MI250X, amdflang AFAR 23.2.0, OpenMP offload): the full 56-case AMR golden set passes, including the multi-block, multi-level, np=2 and reflux-exactness cases, plus the 6 L0/coexist goldens. precheck clean. Cray not exercised locally.
…p sites The note claimed the reflux frame was the prerequisite for hoisting the restore. That was incomplete. Traced s_amr_fine_stage_rk: it reads no grid globals itself (slot arrays plus dt and feature flags), but it calls s_amr_pressure_relax_fine, s_amr_ib_correct_fine and s_amr_update_mib_fine, and each opens its OWN s_amr_swap_to_fine / s_amr_restore_coarse pair assuming the coarse frame on entry. With the restore deferred they would swap while already swapped and trip the nested-swap assert. So there are two blockers. The reflux frame is resolved (advance and reflux loops are now split). The nested swap sites are open and are the larger of the two: three of the five swap/restore sites in m_amr.fpp nest inside the RK pass, which makes hoisting the restore a swap-CONTRACT change (make the swap re-entrant, or make those three routines no-op when already on their own slot) rather than a call-site change. Also records that any swap-contract change needs a Cray GPU run before it can be trusted, per the CCE-acc-only stale-device class in .claude/rules/common-pitfalls.md. Docs only. precheck clean.
…e' curve) Measured on hpcfund (MI250X, amdflang, OpenMP offload), 2D 256^2, static single-level AMR, 20 steps: np compact block (32x32) wide block (128x32) 1 0.1009 s (1.00x) 0.1009 s (1.00x) 2 0.0980 s (1.03x) 0.1190 s (0.85x) 4 0.0983 s (1.03x) 0.1267 s (0.80x) AMR does not strong-scale, and wide blocks anti-scale. Both cases hold exactly one fine block (a static block cannot exceed amr_maxc, so tiling never triggers), and single-owner distribution puts all its work on one rank whatever np is. The compact case is flat because extra ranks only split the coarse grid, which is not the bottleneck. The wide case is worse than flat: the block spans more ranks' coarse subdomains, so each added rank buys more coarse<->fine P2P gather/scatter with no fine parallelism to offset it. Caveat recorded in the note: this is the single-block regime and does not test multi-block distribution, which needs dynamic regrid to reach several blocks. That is the curve batching should move; these numbers are the floor it has to beat. Docs only. precheck clean.
s_amr_swap_to_fine asserted .not. amr_swapped, forbidding nesting outright. That assert is what makes hoisting the restore across blocks a contract change rather than a driver change: s_amr_fine_stage_rk calls s_amr_pressure_relax_fine, s_amr_ib_correct_fine and s_amr_update_mib_fine, and each opens its own swap/restore pair, so a deferred restore would trip it three times per block. Replaces the paired-swap logical with a depth counter. Only the outermost swap saves the coarse state into the sw_* bounce buffers (every save is now depth-guarded; the installs are not, since re-installing the same slot is idempotent), and only the matching outermost restore puts it back - an inner restore returns early, leaving the enclosing frame's slot installed. Sound because every nested swap site swaps to the SAME slot, amr_cur. Behavior-preserving by construction: no nesting occurs today (the RK pass's helpers run after the RHS pass has already restored), so the depth never exceeds 1 and every path is exactly as before. The driver is untouched. What this buys is that hoisting the restore is now a change to m_time_steppers alone. Validated on hpcfund (MI250X, amdflang AFAR 23.2.0, OpenMP offload): 56 AMR goldens and 5 L0/coexist goldens pass, precheck clean. Cray not exercised locally - per .claude/rules/common-pitfalls.md the stale-device class around this contract is CCE-acc-specific, so the Frontier lanes are the real gate.
…-and-seed in one routine Correcting an overclaim: with the swap now re-entrant I said hoisting the restore was a driver-level edit. It is not. s_amr_swap_to_fine ends with 'if (igr) call s_amr_igr_swap_sigma()', which both SAVES the coarse jac/jac_old into sw_jac/sw_jac_old and SEEDS the block's sigma from the parent, reading sw_idwbuff for the coarse extent. Once nesting is actually used an inner swap re-runs it and overwrites sw_jac with fine state, so it must be split into a depth-guarded save and a per-block seed first. Marks prerequisite 2 (re-entrant swap) resolved and records this as prerequisite 3. Also records that this routine already caused an OpenACC-only crash - its own comment notes the sw_* host-only state makes OpenACC's present lookup fail while OpenMP's implicit map(to) tolerates it - as a worked example of why an OpenMP-offload pass does not validate this area. Docs only. precheck clean.
…te 3 of 3) s_amr_igr_swap_sigma both saved the coarse jac/jac_old into sw_jac/sw_jac_old and seeded the block's sigma from the parent. Under nesting an inner swap would re-run the save and put FINE state into sw_jac, which both the seed and s_amr_igr_restore_sigma then work from. The save loop is now depth-guarded; the seed loop is not. The seed reads sw_jac, which still holds the coarse state, so every nested block seeds from the correct parent - guarding only the save is both necessary and sufficient. Behavior-preserving today: no nesting occurs, so the depth is always 1 and the path is exactly as before. This was the last of the three obstacles to hoisting the restore across blocks. With the reflux split (advance and reflux loops separated) and the re-entrant depth-counted swap already in, hoisting is now a change to m_time_steppers alone: open a swap frame across the phase-3 loop, let each block's inner swap install its own slot, restore once after. The swap's recomputes need no guard - they are idempotent for whichever grid is installed. Validated on hpcfund (MI250X, amdflang AFAR 23.2.0, OpenMP offload): the two IGR+AMR goldens that exercise this path (6C20B752, 660FFBFE) pass, as does the full 56-case AMR set. precheck clean. Cray not exercised locally, and this routine has a documented OpenACC-only failure history, so the Frontier acc lane is the real gate.
Follow-up to 2bffefd, whose doc hunk did not apply. Records that the IGR sigma save is now depth-guarded while the seed is not, and that guarding only the save is necessary and sufficient because the seed reads sw_jac which still holds the coarse state. All three prerequisites are now resolved, so the note states plainly what is left: hoisting the restore is a change to m_time_steppers alone - open a swap frame across the phase-3 loop, let each block's inner swap install its own slot, restore once after. The swap's recomputes need no guard, being idempotent for whichever grid is installed. Docs only. precheck clean.
…ffect Built the hoist and measured it: added s_amr_swap_hold (save the coarse state without installing a slot) and bracketed the per-tile RHS loop with it, so consecutive tiles ran fine->fine with one coarse restore after the loop instead of one per tile. Output byte-identical. l0_ntile sweep, 2D 256^2, np=1, with vs without: 1.00/1.01/4.32/16.87 vs 1.00/1.01/4.40/16.75. No effect. The coarse round trip between blocks - its sw_* copies, its device sync, its coefficient rebuild - is not a measurable share of the per-block cost even at 16 blocks where the total penalty is ~17x. Reverted rather than carried as unused machinery. This sharpens the note's conclusion: the per-block cost really is kernel launch count inside s_compute_rhs, not swap traffic, so reducing swap traffic in any form should be expected to do nothing on its own. Only the batched-kernel increment addresses the measured cost. Recorded so the hoist is not retried. Docs only. precheck clean.
… cost The note attributed the per-block cost to kernel launch count, but that was reached by elimination, not measurement. Counted it with rocprofv3 --kernel-trace (2D 128^2, 6 steps, np=1, under srun): 381 launches monolithic vs 7619 at l0_ntile=4 (16 blocks), a 20.0x ratio against the 16.75x wall-time ratio. The cost tracks launches directly. Combined with the refuted restore hoist (removing swap traffic changed nothing), this closes the argument: the fixed per-block cost is the launch count, and only batching blocks into single launches addresses it. Also notes each tile advance issues ~450 launches against the monolithic step's 381, since the per-block path adds its own ghost fill and halo work on top of the same RHS kernel sequence. Docs only. precheck clean.
…mented feature Was one combined PROHIBIT covering dynamic regrid, subcycle, and multi-level. Each clause now lifts independently and names the failure it prevents, measured with the gate disabled (2D 64x32 amr_2d_base grid+block, np=2, AMD OMP offload) against the l0_ntile = 0 arm of the same case, which passes in all three modes: static (control) l0_ntile=0 and =2 both 1ae3bfca3521 - byte-identical regrid deadlocks in the first regrid's collectives subcycle runs 6 steps, then NaN at output multi-level segfaults before step 1 Mechanisms: regrid indexes the block pool from slot 1 and sets amr_num_blocks = nboxes, overrunning the level-0 tile prefix in [1..l0_slot_off] (the static path already maps through f_l0_slot; m_amr_regrid uses it zero times). Subcycle applies its Berger-Colella correction as a STATE reflux into L0 with no route to the tile compute-owners, unlike the lock-step rhs reflux. Multi-level is the amr_maxc scratch cap - Track 3's per-block working set, not a coexist bug. Behavior identical: same three conditions, same set of rejected cases.
Four defects, each found by measurement against the l0_ntile = 0 arm of the same case (2D 64x32 amr_2d_base, AMD OMP offload). np=1 coexist regrid is now byte-identical to monolithic over 4 regrids; np>=2 is not yet, and the checker gate is narrowed to it. 1. m_amr_regrid indexed the shared slot pool by BOX index and set amr_num_blocks = nboxes, overrunning the level-0 tile prefix in [1..l0_slot_off]. The static path already maps through f_l0_slot; regrid used it zero times. stash_migrate, rebuild_slots, boxes_unchanged and the block-set commit now work in slot space. Was: deadlock in the first regrid's collectives. 2. s_amr_assign_block_owners cut into amr_owner_cut, then copied it to amr_fine_cut. Harmless at init (it runs before s_l0_tiles_init builds the tile cut) but at REGRID time it clobbers that tile cut, so every level-0 tile then resolves f_amr_owner against the fine cut. Now cuts into amr_fine_cut and mirrors to amr_owner_cut only when there are no tiles. 3. s_l0_tiles_init ZEROED the module-level max_f*/mbuf* slot-extent state before accumulating the tile extents, discarding the fine sizing s_initialize_amr_module had just computed. s_amr_alloc_slot reads those. The static block's slot is allocated before this routine and escapes, but every slot a regrid allocates afterwards is a fine block cut to TILE size - here a block needing 0:23 in y got a tile's 0:15. Was: out-of-range device write in s_l0_copy_block (segfault under __tgt_target_kernel, not a host abort). Now accumulates the max of both. 4. s_amr_regrid both tags off L0 and prolongs each new block's seed from it, but under coexist the tiles own the state and the stage loop refreshes L0 only at the TOP of each stage - so L0 was one stage stale there. Added the same just-in-time s_l0_scatter_tiles_to_coarse that s_save_data already does before it reads L0. Also fixes a latent bug in the parallel-IO AMR restart writer, independent of tiles: the pack loop had no ownership guard (the host-refresh loop in the same routine does), so on a non-owning rank it ran the full extent loop over an unallocated q_cons and overran the 1-element placeholder buffer. Reachable whenever a rank does not own every block; the collective WRITE_AT_ALL still runs everywhere with cnt = 0. Validation: 61/61 buildable AMR + L0 goldens pass (3 chemistry AMR tests fail to execute here - the gpu-mp-chem build variant is not built in this environment - not golden mismatches). New golden 83CC5C6D covers coexist + dynamic regrid at np=1, which no existing test reached: the coexist goldens are all static and every dynamic-regrid golden runs without tiles. np>=2 remains gated. It diverges on the first step after the first regrid (max abs 5.7e-2, total mass off by 1.3e-4) while choosing the SAME boxes as monolithic, so the fault is in the post-regrid cross-rank state, not box selection.
… a strided GPU_UPDATE
s_restrict_fine_to_coarse's np>=2 receiver unpacked the owner's covered-cell
slice on the HOST and then pushed the covered box back with a strided
GPU_UPDATE(device=). AMD flang copies a non-contiguous 3-D array section in an
OpenMP target update as size(section) CONTIGUOUS elements, so only the first
row landed on the cells it named and the remaining cells overwrote their
neighbours with stale host data - silently.
Measured on a 64x32 2D regrid case at np=2: of the 60 covered cells, 10 were
delivered and 50 were not, total mass off by 1.4e-5 and growing per regrid.
The receiver branch only runs when a level-1 block's owner holds NONE of its
covered coarse cells, which no golden produced: static blocks are owned by a
rank that overlaps them, and the one np>=2 regrid golden is 1D, where the
covered box is a single contiguous row. Regrid at np>=2 in 2D creates it.
Unpack on the device instead, via the existing s_l0_pack_unpack_block - its
wire layout (ci fastest, then cj, ck, i) is already s_amr_restrict_pack_device's.
Same defect and same fix in the pb/mv twin s_amr_scatter_pbmv, where the section
is non-contiguous even in 1D (the x sub-range breaks it across nnode/nb).
This was NOT a coexist bug: the monolithic (l0_ntile = 0) arm was equally wrong,
so lift the np>=2 clause of the l0_ntile+amr regrid gate. All four arms of the
probe case - {np=1, np=2} x {monolithic, coexist} - now agree byte-for-byte,
where before np=2 disagreed with np=1 in both arms.
Goldens: 2C46C59A (plain AMR 2D regrid np=2, the routine the defect lives in)
and 33060D84 (coexist 2D regrid np=2, the newly ungated configuration).
Verified: 57 AMR + 5 coexist/regrid goldens pass; the 3 pre-existing coexist
goldens are unchanged, as expected for a fix confined to the cross-rank branch.
The subcycled fine advance needs three things in the L0 frame that coexist did
not maintain, and the run NaN'd at np=1 on the first step:
1. q_cons_ts(stor), its coarse t^n time-lerp bracket, has exactly one writer -
the monolithic RK, which l0_ntile > 0 skips (tiles keep their own per-slot
backup). The fine C/F ghosts were lerping against an unwritten array. Take
the L0-frame backup at stage 1, where the tiles->L0 scatter has just made L0
an exact mirror of the tiles at t^n.
2. q_cons_ts(1) must be t^{n+1} at the subcycle call, but the stage loop
refreshes L0 only at stage TOPS, so it held the stage-3 entry state.
Re-scatter the advanced tiles first.
3. The Berger-Colella correction lands as a STATE reflux on the coarse cells
just OUTSIDE each block (s_amr_apply_reflux_state); the covered-footprint
copy-back (s_l0_restrict_to_tiles) does not carry those cells.
With (2) in place L0 equals the tiles everywhere except the cells the fine fold
deliberately changed, so (3) needs no new delta plumbing: refill every tile
interior from L0 and the restrict and the reflux shell both arrive. That is an
exact copy round-trip on untouched cells. Split the unconditional fill out of
s_l0_copy_coarse_to_tiles, whose seed gate must still fire exactly once.
Lifts the amr_subcycle clause of the coexist gate; only the amr_max_level > 1
clause remains, and that one is a scratch-capacity limit, not a coupling gap.
Verified: coexist is byte-identical to the monolithic (l0_ntile = 0) arm at BOTH
np=1 and np=2 (all four arms 42ea141991e0, where coexist previously NaN'd).
Goldens FD056B71 / 98AA6EDB. 57 AMR + 7 coexist goldens pass.
…t slot 1
s_amr_build_static_multilevel insets its parent to place the nested level-2
box, and read slot 1 directly. Under coexist slot 1 is the first level-0 TILE,
not the level-1 block, so the box was positioned and sized off a tile:
l0_ntile = 1 tile spans the base grid -> box 2x too wide -> the amr_maxc_fit
cap fired, which is what made this look like the scratch-cap
blocker it was documented as
l0_ntile = 2 tile is half the grid -> a plausible-looking box in the wrong
place -> no abort, silent corruption, NaN at a corner cell far
from the block by the final step
Use f_l0_slot(1), and bound the pool-room check by l0_slot_off + amr_max_fine
rather than amr_max_fine (a slot index compared against a fine-only count).
Both are no-ops without tiles, where f_l0_slot is the identity and l0_slot_off
is 0.
Verified byte-identical to the monolithic (l0_ntile = 0) arm at np=1 with
l0_ntile 2 and 4, and at np=2 with l0_ntile 2. Goldens D99F85F8 / 09E0D257.
The coexist multi-level gate is narrowed rather than removed: l0_ntile = 1
still corrupts the heap during init, at any rank count, pool size and block
size. That is NOT the amr_maxc cap (this case clears it) and is undiagnosed, so
it stays gated with a message that says so. l0_ntile >= 2 is now supported.
…uild s_amr_build_static_multilevel ends by restoring amr_cg and the patch frame to 'block 1' and re-gathering its coarse patch. It selected slot 1 - which under coexist is a level-0 TILE, not the first fine block - so the grid globals were left describing tile geometry for the remainder of initialization. s_initialize_weno_module runs after this (m_start_up.fpp:959) and sizes its device-mapped coefficient tables from those globals, so it faulted inside __tgt_target_data_begin_mapper. Why this hid: with several tiles the leftover geometry is a fraction of the subdomain and the tables came out small enough to map; with l0_ntile = 1 the single tile spans the whole base grid, which is the configuration that actually crashed. Fixing the box lookup in e96ca62 removed the amr_maxc_fit abort and exposed this, so the two are one defect class - slot index vs block index - found in two places. Use f_l0_slot(1), the identity without tiles. This was the LAST coexist gate: l0_ntile > 0 .and. amr now has no PROHIBIT at all. Multi-level is byte-identical to the monolithic (l0_ntile = 0) arm at l0_ntile 1, 2 and 4 (np=1, c4f185d15485) and at np=2 (3aaee2b8fed4). Golden 93EFC4F6 covers the single-tile case that failed. Verified: 57 AMR + 10 coexist goldens pass.
The same-rank branch of s_amr_fine_fine_halo routed a purely LOCAL copy through
amr_seambuf_*, a buffer that exists only to cross MPI: four s_amr_fine_slice
calls per seam pair per RK stage, each a kernel plus a blocking device<->host
round trip. Replace them with one fused device kernel doing both directions.
The cross-rank branch is untouched.
Fusing the two directions is safe because all four slabs are disjoint - a
block's seam GHOST lies outside its own interior, and the two reads come from
different slots than the two writes.
Measured, 2D 64x32, l0_ntile = 4 (16 tiles, 24 same-rank seam pairs), AMD MI250X
OMP offload:
kernel launches 10561 -> 7969 (-24.5%, rocprofv3, deterministic)
Time Avg (n=3) 0.8179 -> 0.7121 s (-12.9%; ranges disjoint, 0.8082-0.8348
vs 0.7100-0.7139)
The variance also collapses, 3.3% -> 0.5% spread: the host round trips were a
scatter source, not just a cost.
Byte-identical: it moves the same cells in the same order, and the old wp buffer
only widened and re-narrowed the stp values. All four coexist modes match the
monolithic (l0_ntile = 0) arm at np=1 and np=2 with unchanged checksums, plus
the 16-tile cases; 57 AMR + 10 coexist goldens pass.
Batching further, ACROSS pairs, is not possible without a flat per-slot backing
array: each slot's q_cons(i)%sf is an independent allocation and amr_slots is
not GPU_DECLARE'd, so a runtime slot index in a kernel is a null deref. A pair
has only two slots, which is why they can be dummies here.
s_amr_fill_fine_ghosts issued one GPU_PARALLEL_LOOP per face slab - ns = 2/4/6 in 1D/2D/3D - to prolong the ghost shell. The slabs all belong to ONE block, so q_fine stays a single dummy and they can be fused without the flat per-slot backing store that cross-block batching needs. Fuse with a flat index over the concatenated slabs, decoded per cell (ns <= 6, so a scan beats a stored map). NOT the padded-hull form of s_amr_capture_creg_dense_batch: the x slabs span the full transverse extent, so a hull over all slabs is the whole buffered volume, and masking it would throw away the O(surface) decomposition the slab split exists to get. Safe because s_amr_build_ghost_slabs emits DISJOINT slabs whose union is exactly the ghost shell - verified by construction: x slabs take x outside [0,fm], y slabs restrict x to the interior and take y outside [0,fn], z slabs restrict both. So every ghost cell is still written exactly once and the order does not matter. Measured 4 -> 1 launches per fine block per RK stage (num_fluids = 1; with multiple fluids the alpha-closure loop below is a second, still-unfused 4). On the 16-tile benchmark that is only 10561 -> 7915 vs 7969, because that case has ONE fine block - the 16 tiles are level-0 and get their ghosts from the edge-BC + seam path, not from here. The saving is 3 launches per FINE block per stage, so it scales with fine-block count, which is the regime this targets. Byte-identical: all four coexist modes match the monolithic arm at np=1 with unchanged checksums; 57 AMR + 10 coexist goldens pass.
Same flat-index fusion as the prolongation loop, applied to the sum-to-one closure loop that follows it: ns launches -> 1 per fine block per RK stage. Only runs for num_fluids > 1, so the l0_ntile gate cases (single fluid) cannot reach it - the multi-fluid AMR goldens are the coverage. Two Fypp constraints found the hard way, both reported as a bare FyppFatalError with no location: a Fortran continuation inside a GPU_PARALLEL_LOOP clause only parses when it falls in the LAST argument, and collapse=1 is rejected - a single loop takes no collapse clause. Byte-identical: 57 AMR + 10 coexist goldens pass.
s_amr_build_ghost_slabs has FOUR consumers, all issuing ns launches per fine
block; only s_amr_fill_fine_ghosts was fused. Apply the same flat-index fusion
to the rest:
s_amr_lerp_fine_ghosts the SUBCYCLE time-lerp fill - called per
SUBSTEP, so it runs more often than the
lock-step fill already fused
s_amr_fill_fine_ghosts_pbmv QBMM pb/mv fill
s_amr_lerp_fine_ghosts_pbmv QBMM subcycle lerp
Each goes ns -> 1 launches per fine block (per substep for the lerp pair). The
pbmv two are gated behind qbmm .and. .not. polytropic; goldens DDD79C8B /
BCBA6E74 / B0A5D230 cover them.
Byte-identical: 57 AMR + 10 coexist goldens pass.
s_amr_fine_fine_halo walks the global seam-pair list and exchanges every same-level adjacent pair. The subcycled level-2 child advance needs to reconcile ONLY its own level's seams: it runs inside one of the parent's substeps, when the level-1 blocks are mid-substep and must not be touched. Add an explicit lev_only selector (0 = all levels, as every current caller passes) rather than an optional argument, so each call site states which levels it means. Prerequisite for transposing s_amr_advance_children - today that routine runs each child's ENTIRE subtree before starting the next, so adjacent siblings never see each other, which is why m_amr_regrid clamps subcycle to one capped level-2 child (a wide feature is under-refined, not non-conservative) and s_amr_check_seam_topology aborts on the restart mode-switch route. Byte-identical: 57 AMR + 11 goldens pass, including 00E15144 (multi-level dynamic subcycle), which is the anchor for the transposition to come.
…halo s_amr_advance_children ran each child's ENTIRE subtree before starting the next, so adjacent siblings never saw each other and their shared face carried mismatched fluxes. That is why m_amr_regrid clamps subcycle to ONE capped level-2 child per box (a wide feature is under-refined rather than non-conservative) and s_amr_check_seam_topology aborts on the restart mode-switch route that can produce adjacent L2 blocks anyway. Transpose it into the shape s_amr_advance_fine_subcycle_all already uses at level 1: set every child up, then walk all of them through each substep together - lerp all ghosts, run the level-filtered seam halo, then advance all - recursing into grandchildren per substep and folding back at the end. The halo is filtered to the child level because this runs INSIDE one of the parent's substeps, when the parent's level is mid-substep and must not be touched. s_amr_advance_subtree had no callers left once its body was inlined here, so it is deleted rather than kept as an unused second path. Byte-identical: with one child per parent the transposition is a pure reordering, and 00E15144 (multi-level dynamic subcycle) and B0A5D230 (QBMM nonpolytropic regrid subcycle) both pass unchanged; 57 AMR goldens pass. The multi-child seam this enables is not yet reachable - the regrid clamp still caps it to one child, which is the next commit.
With s_amr_advance_children transposed (2734961), adjacent level-2 siblings exchange their shared face through the level-filtered seam halo, so subcycle no longer needs to keep ONE capped child. Drop the clamp and tile the box exactly as lock-step already does, and drop the s_amr_check_seam_topology clause that aborted on adjacent L2 blocks under subcycle - the halo it warned about now exists. Effect: a level-2 feature wider than amr_maxc_fit/2 was previously UNDER-REFINED under subcycle (the box was truncated); it is now fully covered by adjacent tiled children. Golden A635AA56 exercises it, and was verified to fail without this change: amr_buf = 12 grows the level-2 region past amr_maxc_fit/2 = 16 coarse cells and restoring the clamp moves the answer (a6e3ad6d -> 5c9d3118). Note it is the BUFFER that widens the box, not the tag threshold - a looser amr_tag_eps at amr_buf = 6 leaves the answer bit-identical either way, so an eps-based case would have protected nothing. 58 AMR + 13 goldens pass; every pre-existing golden is unchanged.
…ution The note asked for a multi-block strong-scaling curve and left it unmeasured. Measured now (2048x1024, dynamic regrid, many disjoint tag clusters, np=1..8, with a uniform control arm): AMR does strong-scale with several blocks to distribute, better than the uniform control through np=4, then turns over 25 percent at np=8. That supersedes the single-block section's conclusion. The target for batching is the np=1 column: AMR costs 10.9x the uniform solver on the same grid, growing with problem size. Attribution: wall time tracks launches (12.1x) not kernel work (2.0x), and the shared-solver per-block s_compute_rhs re-invocation is 69 percent of launches and 94 percent of kernel time, while AMR-only kernels are 31 percent of launches but only 6 percent of kernel time - so further AMR-side fusion is worth much less than its launch share suggests. The 89k device copies are kernel-argument marshalling, 18.6 per launch uniform vs 19.2 AMR, not a separable inefficiency; this reconciles the refuted swap-traffic hoist rather than contradicting it. Also records that regrid at amr_regrid_int=2 costs roughly one time step per regrid, comparable to the whole block-advance cost and untouched by any increment here; corrects the enforced block cap to amr_maxc_fit (amr_maxc is the global half, read nowhere else) and notes non-IB tiles rather than aborts; drops m_viscous from the RHS working set (no volumetric scratch) and adds the dominant m_rhs vector_field set. Method cautions included: use Time/step not the running Time Avg, always run the uniform control arm, measure at a realistic size (the 256x128 case reports AMR anti-scaling that does not generalize), and pass --memory-copy-trace or the copy table is silently empty.
…easurement The previous revision said regrid costs about half the runtime, roughly one time step per regrid. That was inferred from an A/B on amr_regrid_int and is wrong: varying the interval does not hold the block set fixed, so it priced the blocks a regrid produces together with the regrid itself. Per-phase system_clock instrumentation inside s_amr_regrid measures it directly (2048x1024, 20 steps, amr_regrid_int=2): regrid is 0.955/0.459/0.602 s of 13.0/5.8/6.2 s runs at np=1/4/8, i.e. 7-10% of runtime. Eight of ten calls hit the boxes-unchanged early-out and cost almost nothing; only the two full rebuilds matter. The packed super-grid therefore remains the dominant lever rather than one of two equal halves. Attributes a full rebuild: at np=1 it is the serial host round trip of fine block state (stash pull, host interpolate, host overlap copy, device push) on data already device-resident; at np>=4 it is per-box synchronization rather than collective cost. Barrier isolation proves the latter - an MPI_BARRIER at the top of the rebuild loop collapses the geometry phase from 0.089 to 0.0011 s (np=4) and 0.119 to 0.0001 s (np=8), so batching its per-box allreduce would buy nothing. Also records that the box set is not rank-invariant: 14 boxes at np=1/2/4 but 21 at np=8, because amr_maxc_fit is the min over ranks of the local half-extent and forces extra tiling. Total boxes rise 50% while per-rank boxes fall 25%, which is a concrete mechanism for the np=8 turnover and ties it to the same scratch cap the packed super-grid addresses. Instrumentation was temporary and is reverted; per-step time under it (0.633-0.659 s) matched the uninstrumented baseline (0.6634 s). precheck: all 7 checks pass.
The block size cap amr_maxc_fit is the min over ranks of the local half-extent, because the fine advance borrows the rank-local solver scratch. That cap shrinks as ranks are added, so a fixed feature is tiled into more and more blocks the further you scale - and per-block cost is roughly fixed regardless of block size, so block count is what costs. Measured on one fixed case (2048x1024, 7 stripes): nbox = 14, 14, 14, 21 at np = 1, 2, 4, 8, where a 624-tall feature needs 2 y-tiles at a cap of 512 and 3 at 256. It also makes the box set, and so the answer within tolerance, depend on the rank count. amr_max_grid_size pins the cap to an absolute number of coarse cells per dimension instead, the AMReX max_grid_size concept. Verified: with amr_max_grid_size = 128 the same case gives maxc_fit = 128x128 and nbox = 35 at np = 1, 2, 4 and 8 - identical box sets at every rank count. Default is 0, which derives the cap exactly as before, so this is opt-in and no existing behaviour moves: the default path still reports 1024/512 -> 14, 256/512 -> 14 and 256/256 -> 21 at np = 1, 4, 8. Confirmed by the goldens - the full AMR suite passes and so do the nine coexist goldens (1F074C5D 8D466A94 83CC5C6D 33060D84 FD056B71 98AA6EDB D99F85F8 09E0D257 93EFC4F6), which --only AMR does not match. The fine advance still borrows rank-local scratch, so a cap larger than half a subdomain per dimension aborts with the limit named rather than writing out of bounds. Lifting that needs the solver working set sized to the cap instead of the subdomain, which is a separate increment; it is not required for the scaling case, where the useful caps are small in absolute terms. precheck: all 7 checks pass.
…lloc) The AMR fine advance points the solver at a block - s_amr_swap_to_fine rewrites m, idwint and idwbuff - but the arrays stay as allocated. Sizing them from idwbuff therefore ties the maximum block size to the rank subdomain, which is why the block cap is the min-over-ranks local half-extent and why it shrinks as ranks are added. Block count is what costs: per-step time is near-linear in total box count (measured exponent 0.68-0.93), so a cap that shrinks with rank count inflates cost as you scale. Introduces idwbuff_alloc, the ALLOCATION bound, distinct from idwbuff, the RUNTIME bound. It equals idwbuff unless amr_max_grid_size pins a cap larger than the subdomain, so this commit changes no behaviour at all - it is a no-op for every non-AMR run and for every AMR run today, since the guard added in 3a71839 still rejects a cap above the derived one. Converted the allocation sites the fine advance actually touches, established by tracing s_compute_rhs rather than by pattern: m_rhs (198 refs), m_igr (24), m_surface_tension (18), m_bubbles_EL (12), m_thinc (12), m_bubbles_EE (6), plus ptil in m_global_parameters and q_T_sf in m_time_steppers - the one array in that module passed through s_amr_fine_stage_advance into s_compute_rhs. Deliberately NOT converted, because the fine advance cannot reach them: the rest of m_time_steppers (coarse solution, time levels, coarse qbmm side state - the fine advance uses amr_slots(k)%q_cons/q_prim/rhs), m_derived_variables (not used by m_rhs), m_acoustic_src (s_amr_swap_to_fine forces acoustic_source and ab_active false), MPI_IO_DATA, m_load_balance, and m_amr's own coarse-sized arrays. STILL REMAINING before the cap guard can be relaxed: the m/n/p-keyed allocation family, which an idwbuff grep does not find - m_riemann_solvers sizes on -1:m,-1:n,-1:p directly and m_weno on is*_weno. Relaxing the guard before those are converted would write out of bounds. precheck: all 7 checks pass. Goldens: full AMR suite 58/58.
Records the agreed target for the phase after the block-size cap work: give each refinement level its own box list AND its own rank mapping, with inter-level coupling by general point-to-point. Documents the measurement it rests on - cost tracks TOTAL box count near-linearly (exponent 0.68-0.93), and the derived cap shrinks with rank count so a fixed feature is tiled into more boxes the further you scale, which is why np=4 -> 8 regresses even though per-rank boxes fall. Explicitly retracts the intermediate conclusion that whole-box ownership must go. AMReX, Chombo and BoxLib all keep it; they scale on a small absolute cap, per-box scratch, and per-level distribution. The MFC ceiling was a cap DERIVED from the subdomain combined with solver scratch SIZED to the subdomain - amr_max_grid_size (3a71839) and idwbuff_alloc (7b1e493) respectively. Also records why the two alternatives were rejected (domain-aligned ownership surrenders load balance to the coarse cut; block sub-decomposition adds a second decomposition layer and keeps block-as-atom), what MFC already has pointing this way, and that Track 2's P2P parent<->child becomes the spine rather than one queue item - tower co-location then disappears as a consequence. Sequencing puts the m/n/p-keyed scratch conversion BEFORE relaxing the cap guard, since the other order is a silent out-of-bounds write, and requires the conservation ladder rather than field diffs for the many-to-many stages.
…factor (MFlowCode#1679) Three conflicts, resolved to keep both sides: - toolchain/mfc/params/definitions.py: upstream reordered the _nv(_SIM_POST) list (same name set as the merge base); our branch added five load-balance params. Resolution is the exact union. NAMELIST_VARS is a dict and every consumer sorts it, so registration order is not semantic. - src/post_process/m_global_parameters.fpp: keep amr = .false.; drop the mixture_err/alt_soundspeed defaults upstream moved into s_assign_common_defaults, which post already calls beforehand. - src/common/m_mpi_common.fpp: both sides rewrote s_mpi_decompose_computational_domain for orthogonal reasons. Ours replaced the inline rem_cells arithmetic with the box layer; upstream hoisted the MFC_POST_PROCESS/MFC_PRE_PROCESS policy blocks into s_apply_decomposition_policies. Take both: the box block supplies m/n/p and start_idx, the helper supplies silo offsets and local-domain adjustment, and rem_cells_by_dim(1) is set after the box block. Rather than reinstate our #ifdef MFC_SIMULATION for the AMR temperature ghost, use upstream's exchange_all_chemistry_temperatures flag (added in MFlowCode#1679 to keep stage guards out of src/common) and pass amr from simulation's call site. Same condition, no stage guard in common. Also fixes a silent semantic conflict git could not flag: upstream added a required third argument to s_initialize_eqn_idx, and the branch-only m_load_balance.fpp calls it. Passes six_eqn_alf_is_advected=.true. to match simulation's own call site, which that call deliberately duplicates. Verified: format clean, precheck 7/7, all three targets build, and the full golden suite passes 703/703 (69 AMR, 16 chemistry, all 8 ppn=2 tests) on AMD MI250X with amdflang OpenMP offload. The 32 convergence cases are the runner's default skip.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An opt-in, default-off family of performance features and the measurement infrastructure they rest on. With all flags at their defaults the only touched production path is
s_mpi_decompose_computational_domain, refactored through the newm_boxmodule (byte-identical; covered by the existing suite).m_box(partition arithmetic),m_load_weight/load_weight_wrt(per-cell load-weight field + imbalance metric),m_sfc_partition/sfc_partition_wrt(Morton-SFC predicted-imbalance diagnostic),m_load_balance/load_balance(weighted static decomposition at init; AMR-fine-work-aware),m_rank_timing/rank_time_wrt(per-rank compute-time diagnostic).m_active_box/active_box: restricts reconstruction/Riemann/RK windows to a light-cone-grown box around non-ambient flow; strict-subset golden-tested.hybrid_wenoandhybrid_riemann(+hybrid_weno_eps,hybrid_smooth_flux): linear-optimal weights / central-or-Rusanov flux in smooth cells, full WENO/HLLC at flagged discontinuities (Jameson sensor, stencil-dilated, per-level under AMR).m_amr+m_amr_registers: two-level 2:1 refined block hierarchy; conservative restriction and conservative-linear prolongation with physics-specific closures; per-stage flux registers with Berger–Colella refluxing; Berger–Rigoutsos multi-block dynamic regrid; optional dt/2 subcycling; multi-rank (single-owner blocks assigned by Morton-SFC work balancing at each regrid, with migration; blocks may span rank seams via P2P coarse↔fine gather/scatter; same-level seam halo; distributed registers); restart (both IO modes, regridded-layout persistence); AMR-aware post-processing (fine blocks visualizable as Silo overlay domains); GPU-resident fine level on both OpenACC and OpenMP offload.Full algorithm and user documentation:
docs/documentation/amr.md(support matrix enforced at runtime by the checker — unsupported combinations abort with named messages, never silently).AMR physics support matrix (abridged; authoritative table in amr.md)
Supported and golden-tested: single- and multi-fluid (5-eq,
mpp_lim) · 6-eq with per-block pressure relaxation · viscous (refluxed) · phase change (relax) · chemistry incl. species diffusion · Euler–Euler bubbles (polytropic/non-polytropic, mono/polydisperse, QBMM incl. non-polytropic with per-blockpb/mvside-state; dynamic regrid + subcycle) · acoustic sources (coarse-grid support with regrid exclusion) · immersed boundaries (multi-body, static or prescribed-motion, incl. dynamic regrid with body-containment expansion and per-substage guards) · 2D axisymmetric (per-block WENO-coefficient recompute) · stretched grids (exact parent-bisection ghost coordinates + per-swap coefficient recompute) · hybrid WENO/Riemann sensors (per-level) · Lagrangian bubbles (cloud excluded from blocks; two-way coupling on the coarse grid; regrid clips around the moving cloud) ·active_box(blocks contained in the growing window; agrees with plain AMR to ~1e-14) · IGR (restriction-only coupling: fine sigma solve seeded/Dirichlet-bounded by the coarse solve; documented truncation-order seam, exact free-stream) · 1D MHD/RMHD (div(B)=0 by construction in 1D; HLL and HLLD, incl. relativistic).Gated with named aborts (documented rationale): surface tension (seam force imbalance is structural — three fixes attempted and diagnosed in amr.md) · 2D/3D MHD (attempted and measured: the c/f seam is a continuous O(1) div(B) source GLM cleaning cannot remove — needs constrained-transport-class B prolongation/reflux) · hyperelasticity · 3D cylindrical (global azimuthal filter) · force-driven IB (
moving_ibm=2) · STL bodies · Riemann-extrapolation BCs (bc=-4) ·amr_subcycleunder IGR · stretched grids with Lagrangian/IB-regrid (uniform-spacing index conversions).Validation evidence
Known issues (all non-gating or in progress)
continue-on-error): an intermittent post-detected NaN on the two Lagrangian+AMR goldens. Exhaustively unreproducible off GitHub's runners — the exact failing stack (NVHPC 24.3 SDK,-tp=px -Kieee, HPC-X MPI, and the CI docker image itself under apptainer) passes elsewhere, as do native/zen2 builds; 24.5+ green. Documented at the golden definitions.Review guide
The commit history is arc-ordered (active-box → load-weight → SFC → weighted decomposition → rank timing → hybrid → m_box → AMR rungs → physics envelope → CI/GPU hardening); reviewing by arc is much easier than by file. The AMR arc builds stepwise: static hierarchy → restriction/prolongation → fine advance → refluxing → regrid → subcycling → multi-rank → GPU → each physics rung with its own validation. Commit messages carry the validation evidence for their change (measured defects, golden UUIDs, repro details for CI fixes).
All parameters ship default-off with
case_validatorentries, runtime checker gates, andcase.md/amr.mddocumentation.