Skip to content

Use monthly builds from PoB Codes for CI testing - #14

Closed
AdamZ-8113 wants to merge 1 commit into
Paliak:tests-branchfrom
AdamZ-8113:pob-codes-monthly-ci
Closed

Use monthly builds from PoB Codes for CI testing#14
AdamZ-8113 wants to merge 1 commit into
Paliak:tests-branchfrom
AdamZ-8113:pob-codes-monthly-ci

Conversation

@AdamZ-8113

Copy link
Copy Markdown

This hooks up the PoB Codes API to the CI pipeline. The API provides up to 100 builds each month, and this repo keeps a FIFO of up to 500 unique builds. When CI runs, it calculates those saved builds on both versions of PoB and compares the results. The existing test builds run alongside them, so there's some consistent coverage while the monthly builds bring in newer stuff people are playing.

The API is live at https://api.pob.codes/test-builds and doesn't need authentication. CI downloads the monthly batch separately, so every test run can use the saved builds without hitting the API again.

This ended up touching a few different parts: downloading and storing the builds, running the calculations in Docker, comparing the results, and wiring that into GitHub Actions. The existing Busted tests, test builds, and old build-generation code are still there. This doesn't change PoB's calculations or game data.

Testing turned up a couple of issues that needed fixing. The older LuaJIT ran out of memory with current builds, and the comparison script could reject valid results just because the XML attributes were in a different order. Both are fixed here. Failed imports and incomplete calculations also stop the run instead of looking like a successful test.

By default, differences in calculated stats are reported without failing CI, since a change might be intentional. Strict mode makes those differences fail the test too. Import or calculation errors fail either way.

One thing we need to sort out before using this on tests-branch: it's still on the older 3.25 runtime, which can't load the current 3.29 builds. I tested this on PoB v2.67.2. Both versions being compared need to support the builds we're feeding them, so the branch needs a runtime update first. That update isn't included in these CI changes.

I tested this locally using Docker and the actual PoB Codes API:

  • All 100 API builds and the five existing test builds calculated successfully on both sides. Comparing the same calculations gave no differences.
  • I deliberately added one Life to the calculation to make sure it would catch a real change. It caught it in all 105 builds. That test change isn't included here.
  • On v2.67.2, all 517 existing tests passed, along with the 20 new Python and Lua tests. After putting the CI changes onto the current tests-branch, all five fixture comparisons and the 20 new tests passed there too.
  • Downloading the same monthly batch again didn't add duplicates or create another corpus commit. The FIFO tests also cover dropping the oldest builds once it reaches 500.
  • Bad builds fail without reusing results from the previous build. The workflow checks also passed.

Once the runtime is updated, you can try it by manually running Refresh monthly build corpus, then Compare saved builds. Set TEST_BUILD_CORPUS_ENABLED=true to include the saved builds in PR checks. For automatic refreshes, the workflow also needs to be on the default branch. It checks daily, but only adds a new batch once per month.

The remaining checks are running this in your GitHub setup and measuring a full 500-build run. The FIFO behavior is tested at 500, but the actual API currently gives us 100 distinct builds, so that's the live batch I've tested.

@AdamZ-8113 AdamZ-8113 closed this Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant