wasm-synth z_impl: sem->count not incremented on give (stays 0, limit=1) on hardware — u64-packed new_count miscompiled; binary sem never signals (v0.11.12)

[avrabe]

Summary

On synth v0.11.12 (3dd70675), the wasm-cross-LTO z_impl_k_sem_give runs on real Cortex-M4 hardware without faulting (thanks — #202 fixed that), but it is functionally wrong: a k_sem_give on a binary semaphore (limit=1) with no waiter does not increment sem->count — it stays 0. The semaphore therefore never signals, and a blocked k_sem_take never wakes → the program hangs.

On-target evidence (NUCLEO-G474RE, gdb via openocd)

z_impl_k_sem_give is the loom-inlined wasm shim (decide folded in), called from the bench's ISR. Breakpointing it:

give entry  (0x08000260): sem=0x20000040  count[+8]=0  limit[+12]=1
give end    (reschedule):                 count[+8]=0     <-- still 0, should be 1

• z_unpend_first_thread(&sem->wait_q) correctly returns NULL (wait_q at 0x20000040 is empty: head==tail==self), so this is the no-waiter → increment path.
• gale's decision for decide(count=0, limit=1, has_waiter=0) is "increment, new_count = min(count+1, limit) = 1".
• The store sem->count = new_count leaves count at 0, not 1.

The WAKE branch is correctly not taken (verified: arch_thread_return_value_set/z_ready_thread never fire; z_reschedule does). So the bug is specifically the new_count value written on the no-waiter path.

gale_k_sem_give_decide returns the decision AAPCS-packed into a u64 (action byte in low bits, new_count in the high 32). The shim extracts du.d.new_count (high word) and stores it. Either the inlined decide computes new_count wrong, or synth's u64→u32 high-word extraction (i64 shift/field access) yields 0. The native rustc-direct build of the identical logic increments correctly, so it's specific to the wasm→loom→synth path's i64 handling.

Reproduction

loom optimize merged.faithful.loom.wasm ... # decide inlined into z_impl (loom v1.1.5)
synth compile <that> --target cortex-m4f --all-exports --relocatable -o z.o   # synth v0.11.12
# link into the engine_control bench on nucleo_g474re, flash, run:
# boots, prints header, then hangs — sem->count never increments (gdb above).

WAT of the loom-inlined module: https://gist.github.com/avrabe/d30f965ac96b8ca4cac7d4fea2832a6a

Impact

Last functional blocker for the wasm-cross-LTO on-silicon measurement: the object compiles, links, flashes, and runs without faulting, but the semaphore give is semantically incorrect (count not incremented), so the bench can't complete a handoff. A unit-level check of the u64-packed-return extraction on the inlined z_impl would catch it (the IR-level Z3 validator evidently doesn't, since this is the encoded i64 high-word extraction).

Environment

• synth v0.11.12 (3dd70675), arm backend, cortex-m4f; input from loom v1.1.5 (95a5f982) --passes inline
• NUCLEO-G474RE; Zephyr SDK 1.0.1

[avrabe]

Investigated deeply but I have not landed a fix yet — being upfront rather than shipping a guess. Here's what I verified and where I'm stuck, so we can pin it together.

Verified CORRECT (byte-traced from your gist's merged.both.loom.wat, compiled v0.11.12 --relocatable)

I compiled your exact gist and hand-decoded z_impl_k_sem_give's no-waiter path. Each of these is correct in the emitted Thumb-2:

1. i64.shl 32 (packing). Large-shift path: rd_hi = rn_lo, rd_lo = 0 → (new_count << 32) lands new_count in the high word. ✓
2. i64.or with the action. {lo: action, hi: new_count}. ✓
3. i64.shr_u 32 (extraction). Large-shift path: LSR rd_lo, rn_hi, #0 → rd_lo = rn_hi = new_count, rd_hi = 0. ✓ (the high-word extraction you suspected is correct)
4. i64.store32. Stores the low word (new_count). ✓
5. Static store/load offsets (offset=8) fold via ADD ip, rN, #8. ✓
6. The isolated decide (count=0,limit=1,waiter=0 → return new_count) compiled standalone returns 1. ✓
7. Local frame slots are consistent at the IR level — local 4 (new_count) maps to the same slot at every local.set/local.get (including the increment and the pack), local 3 (i64) to its own slot. No slot aliasing in the selector output. ✓

Where I'm stuck

I could not reproduce the failure in any minimal local case — every pattern I built (pack/extract/store, decide-with-i64+call pressure, your exact local index structure 3=i64/4=i32) produces correct, consistent IR. The bug only shows on your full on-hardware run. I have no local ARM execution (no qemu/unicorn here), so I can't watch the runtime value.

Could you check two things with gdb? (you have the board + the full link)

1. Is new_count (the i32 working local) actually 1 at the point of the pack? Break right before the i64.extend_i32_u; i64.const 32; i64.shl and read the register/stack slot holding the post-increment count. If it's already 0 there, the bug is upstream of the i64 packing (the select/br_table increment-vs-saturate dispatch picking the wrong value); if it's 1 there but 0 in memory after the store, it's the pack/extract/store.
2. What's the exact faulting store address vs &sem->count? disassemble the final i64.store32 region and confirm the STR lands on &sem->count. If the value is right but the address is off, it's an addressing issue I can chase.

Either answer narrows it to a layer I've already mapped. Sorry I don't have it fixed — #202's branch fix was byte-verifiable locally; this one needs the runtime value, which is on your side of the wire right now.

[avrabe]

Update — root cause confirmed by emulation (not yet fixed). I built a unicorn-engine harness that runs your gist's z_impl_k_sem_give locally (stub z_unpend → NULL, sem.count=0/limit=1) and reproduces count staying 0. Tracing the execution pins it precisely:

The bug (two compounding issues)

1. Store address clobbered (the arm regalloc: select_with_stack allocates param registers (r0-r3) for temps/results before live param reads — i64_lowering fuzz class #193 param-register class). At the i64.store32, the value is correct (new_count = 1) but it's written to [fp + r0] where r0 = 1, not the sem pointer (0x100). local.get 0 (the store's address) is param 0, which lives only in r0 — and select_with_stack's alloc_temp_safe reserves only operand-stack live regs, not the param registers in local_to_reg. So r0 (sem) gets handed out as a temp / overwritten by the k_spin_lock(1024) arg marshalling before the store reads it. This is exactly arm regalloc: select_with_stack allocates param registers (r0-r3) for temps/results before live param reads — i64_lowering fuzz class #193, and it's the direct cause of "sem->count never updates."

2. A control-flow wrinkle underneath. I prototyped the param-reservation fix (keep r0 reserved until param 0's last read). It does preserve sem in r0 to the store region — but then exposes that the no-waiter path branches past the count store (exec goes …→ b → store+4, skipping str.w r8,[fp,r0]), so count is still never written. So there's a second issue in how the increment/store block is placed/branched for this nested-block shape, on top of the clobber.

So #204 isn't a one-line i64 fix — it's the param-clobber (#193) on the store address plus a store-placement/branch issue. I reverted the partial fix (it broke the test build and didn't fully resolve either, fuzz still finds other clobber paths).

Reproducible locally now

unicorn harness (reads your gist, stubs imports, runs z_impl, checks sem.count): I can share it — it makes this a fast local iterate instead of a flash cycle. If you confirm on hardware that the value heading into the store is correct (1) but the address is wrong (your gdb: is the faulting store's base register sem, or a small integer like 1/1024?), that nails issue #1 independently.

Honest status: real root cause now, but the fix is more involved than I can land safely in one pass — it's the #193 allocator rework (reserve param regs) done correctly across all ~58 allocation sites, plus the store-placement issue. Prioritizing getting it right over fast.

[avrabe]

Great root-causing — the unicorn harness + #193 param-clobber-on-store-address + the store-placement/branch issue all line up with what I see on hardware. Independent on-target corroboration (NUCLEO-G474RE, gdb via openocd, your gist build, synth v0.11.12):

1. The value path is fine; the store never reaches &sem->count. Set a hardware watchpoint on &sem->count (0x20000048) and ran. The only write that fires is from compiler_builtins memcpy at 0x08005188 (the k_sem init writing count=0) — z_impl's give store never fires the watchpoint. Combined with my earlier read (count=0 at give entry and at give-end/reschedule), this independently confirms issue #1: the give's count store targets the wrong address, not &sem->count.

2. It's the no-waiter (increment) path. z_unpend_first_thread(&sem->wait_q) returns NULL — wait_q at 0x20000040 is empty (head==tail==self) — so decide should increment, and the WAKE branch correctly never runs (arch_thread_return_value_set/z_ready_thread never fire; z_reschedule does).

So on hardware: value heading to the store is right, the store address is wrong (never &sem->count) → matches your [fp+r0] with r0=1 finding. I couldn't pin the store instruction locally (objdump won't decode synth output w/o $t mapping symbols — #170 — and --only-section=.text extraction came up empty), so the watchpoint was the cleanest hardware angle.

Happy to run your unicorn harness locally (please share — faster than my flash cycle), or do a targeted gdb read if you give me the store PC. I'll re-flash + capture the engine_control handoff-cycle number the moment the #193-rework + store-placement fix lands. Thanks for prioritizing correctness here.

[avrabe]

Fixed in v0.11.13 ✅ (released — please re-flash)

Your watchpoint nailed it: the store address was wrong, the value was right. I reproduced count=0 in a local unicorn harness, and the root cause was a compound of three issues, all now fixed:

1. The decoder silently dropped i64.extend_i32_u / i64.extend_i32_s / i32.wrap_i64 — convert_operator had no arm for them, so your (new_count << 32) pack (local.get N; i64.extend_i32_u; i64.const 32; i64.shl) lost the extend, leaving an i32 as a 64-bit operand with a garbage high half.
2. Latent arm regalloc: i64 consecutive-pair spill / coalescing (root of #168) #171 register-pair spill — every register-resident stack value was spilled as an 8-byte i64 pair, so an i32 temp in R12 hit i64_pair_hi(R12) and the allocator exhausted on the pack. StackVal now tracks i32-vs-i64 width.
3. The store-address param clobber (arm regalloc: select_with_stack allocates param registers (r0-r3) for temps/results before live param reads — i64_lowering fuzz class #193 class) — local.get 0 (your sem pointer) lives in R0, and k_spin_lock(1024)'s AAPCS arg marshalling reclaimed R0 before the store read it. Call-containing functions now frame-back every in-register param (entry spill + reload through a stack slot), so the address survives the call.

Verification

On the cortex-m4 (Thumb-2) build the give now lowers to str.w r_val, [r11, r_semptr] (indexed by the preserved sem pointer), and a unicorn run of your gist sets sem->count = 1 for the no-waiter give — matching the hardware behaviour your watchpoint predicted.

Process note that cost me time, in case it bites you: synth compile defaults to --target arm (ARM32), whose encoder drops the register index on indexed loads/stores. Your Zephyr build uses a Cortex-M (Thumb-2) target, where it's correct — but if you ever compile without --target cortex-m4/-m7, you'll get ARM32 output. (I filed that ARM32-encoder gap separately.)

Released as v0.11.13 (GitHub Release + all 11 crates on crates.io). Please re-flash and capture the engine_control handoff-cycle number — that's the last green light on the calling-convention arc.

One honest caveat: a separate, call-free variant of the #193 param-clobber (an i64 result landing on a live param register in a function with no calls) is still open and tracked for v0.11.14. It does not affect z_impl_k_sem_give (a call-containing function — its params are frame-backed), so it shouldn't block your re-flash. Happy to share the unicorn harness if useful.

[avrabe]

Hardware retest on v0.11.13 (d8712738): the count-store fix works — sem->count now increments 0→1 at the give, and under gdb single-stepping I see it oscillate across gives (give#1 entry=0 → give → main consumes → give#2 entry=0). So the no-waiter increment path is correct now. Thank you!

But the full engine_control bench still doesn't complete on silicon, and it looks like a heisenbug I can't pin from the debugger:

• Under gdb (breakpoints on the give): gives fire repeatedly, count oscillates → handoff works.
• Free-run (no breakpoints): the bench prints its header then stalls. After 3s: CPU idle (__enable_irq), sem wait_q empty (head==self @ 0x20000040), count=0, no E (handoff) rows ever emitted (130s capture).
• gale_w_ready_thread (the WAKE path) never fires in an 8s free-run — so a pended waiter is never woken; the breakpoints evidently shift timing enough that the increment path is always taken instead.

So with the count fix, the increment path is verified correct, but I can't exercise/verify the has_waiter=1 WAKE path on hardware because the debugger perturbs the give/take race. Could you point me at (or share) the unicorn harness you built for #204? If I can drive z_impl with a non-empty wait_q (a stub pended thread) in emulation and check it takes the WAKE branch + calls z_ready_thread with the right thread, that would deterministically confirm whether the WAKE path is correct — without the on-hardware timing noise. Happy to run it and report. (Everything else is in place: builds, links, flashes, runs crash-free, increment path correct — this is the last behavior to verify.)

[avrabe]

🎉 End-to-end success on silicon (v0.11.14, e8488f0c). The br_table/select/SetCond WAKE-path fix did it — the wasm-cross-LTO z_impl_k_sem_give now runs correctly on the NUCLEO-G474RE: the full engine_control sweep completes (150 samples, 5 RPM steps, drops=0, === END ===), and both the no-waiter increment path and the WAKE path work. Measured handoff = 907 cyc median (steady-state, n=148). Thank you for the deep, careful fixes across #202/#204 — the seam-dissolved wasm→loom→synth pipeline now executes correctly on real Cortex-M4 hardware. (907 is ~1.6× native LTO; the gap is synth codegen maturity + my fp=0 trampoline / out-of-line inline-API wrappers, not the architecture.)

[avrabe]

WAKE path fixed in v0.11.14 ✅ (it was three synth bugs, not your logic)

Thanks for the precise hardware report — the "increment works, full bench stalls, z_ready_thread never fires in free-run" symptom was exactly enough to chase it in emulation. I built the harness you asked for and it's now in the repo: scripts/repro/wake_path_differential.py.

Method: a differential test — wasmtime runs your merged.both.loom.wat as the oracle (the WASM semantics), unicorn runs synth's compiled ARM, and we diff the import-call trace + sem->count for both z_unpend → NULL (no-waiter) and z_unpend → 0x5000 (a stubbed pended waiter). Your WAT is correct — wasmtime wakes the waiter and leaves count=0. synth was miscompiling it. Three independent bugs in the --relocatable direct selector, all of which made the give always increment and never wake (so no-waiter looked fine on v0.11.13 by accident):

1. The decoder silently dropped br_table — no index dispatch was emitted, so control fell straight into table arm 0 and has_waiter never selected the WAKE branch.
2. select clobbered its own condition flags — a 16-bit MOVS between the CMP and the conditional move destroyed the comparison (register-pressure-dependent, which is partly why it was a heisenbug).
3. SetCond corrupted a high-register result — the 16-bit MOVS Rd,#imm has a 3-bit Rd field; for Rd = R12 (where the allocator put has_waiter) the register bits overflowed the opcode and turned MOVS into CMP, so the boolean was never written.

Verified end-to-end on the cortex-m4 (Thumb-2) build: synth now matches the WASM oracle on both paths —

• no-waiter → sem->count = 1, no wake;
• waiter → sem->count = 0, arch_thread_return_value_set(thread, 0) + z_ready_thread(thread) with the unpended thread pointer.

Please re-flash v0.11.14 and re-run the engine_control bench in free-run — the WAKE path should now actually hand off, so the bench should complete (and the heisenbug should disappear, since the dispatch is no longer timing-fragile garbage). If you can run the harness locally too, it'll let you confirm any future give/take shape without a flash cycle.

(Unrelated, still open: the call-free #193 fuzz class — does not affect z_impl_k_sem_give. And #206, the ARM32 default-target encoder, only matters if you ever drop --target cortex-m4.)