Use monthly builds from PoB Codes for CI testing - #14
Closed
AdamZ-8113 wants to merge 1 commit into
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This hooks up the PoB Codes API to the CI pipeline. The API provides up to 100 builds each month, and this repo keeps a FIFO of up to 500 unique builds. When CI runs, it calculates those saved builds on both versions of PoB and compares the results. The existing test builds run alongside them, so there's some consistent coverage while the monthly builds bring in newer stuff people are playing.
The API is live at
https://api.pob.codes/test-buildsand doesn't need authentication. CI downloads the monthly batch separately, so every test run can use the saved builds without hitting the API again.This ended up touching a few different parts: downloading and storing the builds, running the calculations in Docker, comparing the results, and wiring that into GitHub Actions. The existing Busted tests, test builds, and old build-generation code are still there. This doesn't change PoB's calculations or game data.
Testing turned up a couple of issues that needed fixing. The older LuaJIT ran out of memory with current builds, and the comparison script could reject valid results just because the XML attributes were in a different order. Both are fixed here. Failed imports and incomplete calculations also stop the run instead of looking like a successful test.
By default, differences in calculated stats are reported without failing CI, since a change might be intentional. Strict mode makes those differences fail the test too. Import or calculation errors fail either way.
One thing we need to sort out before using this on
tests-branch: it's still on the older 3.25 runtime, which can't load the current 3.29 builds. I tested this on PoB v2.67.2. Both versions being compared need to support the builds we're feeding them, so the branch needs a runtime update first. That update isn't included in these CI changes.I tested this locally using Docker and the actual PoB Codes API:
Once the runtime is updated, you can try it by manually running Refresh monthly build corpus, then Compare saved builds. Set
TEST_BUILD_CORPUS_ENABLED=trueto include the saved builds in PR checks. For automatic refreshes, the workflow also needs to be on the default branch. It checks daily, but only adds a new batch once per month.The remaining checks are running this in your GitHub setup and measuring a full 500-build run. The FIFO behavior is tested at 500, but the actual API currently gives us 100 distinct builds, so that's the live batch I've tested.