research/benchmark: bit0 same-machine campaigns (openXC7 vs Vivado 2026.1): raw CSVs, machine files, harness

[cavearr]

As offered in the thread: the raw artefacts of the bit0 (same-machine) half of the openXC7 vs Vivado benchmark, mirrored where the write-up will cite them, under research/benchmark/bit0/.
The README there says which file answers what:

• openxc7/: the clean openXC7 campaign (N=5 per design, toolchain revisions stamped from the built tree at run time), its machine snapshot, REVISIONS.txt (provisioning record; the harness re-reads revisions from the tree and does not trust it), and timing-context/max-frequency-lines.txt: every "Max frequency" line of every stage log with its line number, so the timing column can be re-read from the LAST block (the routed analysis; the mid-log block is the placer's pre-route estimate).

• vivado/: the clean Vivado 2026.1 v2 campaign (two consecutive batches, both v2) with machine snapshots, and contaminated-v1/ archived as evidence with its NOTE (Flexera RUI telemetry making raw connect() calls that ignore the proxy environment, booking 4 to 7 nondeterministic minutes into write_bitstream).

• harness/: both harness scripts, the Vivado tcl, designs.tsv, and vivado-nonet.sh (the rootless unshare -r -n wrapper with a dummy interface carrying the licensed MAC; the fix for the stall).

Known caveats are stated in the README: litex-ddr-arty-s7 (main) fails DRC BIVRU-1 under Vivado as frozen; picosoc's create_clock is commented out in its XDC; every openXC7 timing PASS is against nextpnr's 12 MHz default, because the design's create_clock on the pad net does not propagate through IBUF/BUFG/PLL (issue to be filed).

[gHashTag]

Reviewed. The bundle is in better shape than the write-up it supports — max-frequency-lines.txt carries the log line numbers, and the README already states the estimate-vs-routed trap explicitly ("Read the LAST block per log"). That is the defect I published and it is documented here before I corrected it, which is the right way round.

I verified the numbers rather than reading the summary: litex-ddr-arty-s7 is 65.52 at line 373 (placement estimate) and 68.73 at line 1515 (routed); deephier is 74.21 at 381 and 69.72 at 1512. Identical across all five runs of each — the seed is pinned, so there is no run-to-run spread to report. My published 65.5–69.7 range took the estimate as its floor and the routed value as its ceiling. Corrected on openXC7/nextpnr-xilinx#150.

Two things to add before this becomes the citable record.

1. The 12 MHz caveat says "(issue to be filed)". It is filed: openXC7/nextpnr-xilinx#155. Worth replacing the parenthetical with the link so a reader can follow it.

2. The missing caveat — ten of these rows time a bitstream that cannot run. Both litex-ddr-arty-s7 designs are VexRiscv SoCs, and openXC7/nextpnr-xilinx#150 is the case where the frozen toolchain (05aaa06, which predates f1c77134) emits a bitstream whose console is dead at 16 MHz. Five runs each, so ten of the twenty openXC7 builds produced an artifact that does not work.

The seconds are unaffected — the harness times three subprocesses and none of this touches that. What is affected is any sentence placed near them. "openXC7 built this in N seconds" stands; "…and it met timing" does not, and for these two designs "…and it ran" does not either. I would put that in Known caveats in the same font as the others rather than in a footnote, because the 13.7× is the number people will quote and this is the qualifier that travels worst.

Neither is a blocker on the data. Both are edits to the README.

---

One thing that is not mine to wave through: harness/ ships three .sh files, and this repo has a standing no-shell-scripts rule (.claude/rules/no-shell-scripts.md) with a hook enforcing it. My reading is that the rule targets project tooling — CLI entry points, build steps, deploy — and that an archived record of how an external measurement was produced is a different category; stripping it would leave published numbers with no reproduction path, which is worse than the rule's cost. But the rule also warns specifically against copying patterns from legacy scripts, and that risk is real here.

So: I would merge with a harness/NOTE.md saying these are archived measurement provenance, exempt for that reason, and not to be used as a pattern for anything in src/ or CI — making the exemption explicit and auditable rather than silent. Flagging rather than doing it, since it is a project rule and the call belongs to @gHashTag.

[gHashTag]

Merged as delivered — 91e1443cb. Your branch untouched, so the measurement record is exactly what you handed over.

Follow-up in dd0c9af57 adds the two README items from the review and harness/NOTE.md. On the shell scripts: @gHashTag's call is to keep them, and the reasoning is the one that matters here — rewriting them in Zig after the fact would produce a different harness and break the link between the artefacts and the thing that made them. The rule file and CLAUDE.md now carry the exemption explicitly, including a line saying not to include the path in any sweep that deletes .sh files, so it does not quietly get tidied away in six months.

The caveat that was missing is now in Known caveats in the same font as the others: ten of the twenty openXC7 rows timed a bitstream that does not run. And the routed figures are stated with their log line numbers — 1515 and 1512 — because three corrections to one number is enough.

(Ignore the previous comment; my shell ate every backtick in it. A quoting bug that silently deletes the identifiers and posts the rest is a fair addition to this week's collection.)

[gHashTag]

@cavearr Before you spend a bit0 slot on this: a re-run against a post-#150 toolchain would not move the seconds, and the diff settles it without a campaign.

I was asked to re-measure now that #150 is merged, on the reasoning that ten of our twenty openXC7 rows timed a bitstream that does not run. Read the fix first. f1c771349 touches exactly one file, xilinx/fasm.cc, +49/-1, entirely inside write_bram_width — the FASM backend, downstream of everything the harness times.

stage timed	what #150 touches	effect
synth_ms	yosys — nothing	none
pnr_ms	two integer comparisons per BRAM width parameter, during FASM write	unmeasurable against a P&R of tens of seconds
bit_ms	sets more bits in frames already emitted; frame count is fixed by the part	none

So the wall-clock rows stand as measured and so does the ratio. The fix changes what the bitstream contains, not how long it took to produce.

Which means the caveat I added to the bundle README needed sharpening rather than the numbers needing replacing, and I have done that (8b463ae0c). The honest form is: the seconds were never the thing that was wrong. What was wrong was a sentence next to them.

The re-run that IS worth doing is functional, not temporal — does litex-ddr-arty-s7 now boot with f1c771349 in. That is a silicon test on an Arty-S7 and I do not have that board; we are AX7203 here. If you have one, that closes the loop on #150 with the same directness the 16 MHz dead console opened it.

Two constraints worth stating so nobody proposes the shortcut: the campaign is same-machine by construction, and Vivado does not run on Apple Silicon at all, so the comparison half cannot be reproduced anywhere but bit0. A timing re-run on my hardware would produce numbers that cannot be placed beside the Vivado column — which is a worse outcome than no number.

Separately, while checking this I verified all three commit ids against the upstream tree rather than carrying them forward: 05aaa06bc is our freeze, f1c771349 the fix, d5b2c610a its merge, and the fix is on both main and stable-backports.

[cavearr]

@gHashTag Agreed on all three counts, and thank you for handling the merge and the NOTE.md exemption the way you did — the harness stays what produced the numbers, which is the whole point of shipping it.

On the re-run: right, #150 lives in write_bram_width, downstream of everything the harness times — the seconds stand, the sentence next to them changes, and you have already changed it.

The re-run that matters is functional, and it belongs to @hansfbaier: the Arty S7 is his board and litex-ddr-arty-s7 is his design (he already reports litex-minimal-arty-s7 booting after #134, at 50 MHz).

We do not have an Arty S7 here; if Hans confirms the DDR design boots with f1c77134 in, that closes #150's loop on the silicon that opened it — and then the 100 MHz question is pure QoR, which is our open front.