Repository navigation
fix: start the Robot Framework run once and key tests by id - #523
Merged
Merged
Conversation
start_suite runs for every suite in the hierarchy and started the run each time; in report mode that wiped the results of the suites that had already run. The run is now started, filtered and registered only for the root suite. Tests were matched to their suite path by name and line number, which is not unique across files, so a test could be reported under another file's suite. They are now keyed by the Robot Framework test id. pabot processes waited for the run id in a busy loop with no timeout and hung when the first process failed to create the run; they now poll with a timeout. Keywords tagged with qase.params or qase.fields that run in a suite setup or teardown no longer touch the previous test's result.
The expected report listed only the four tests of the last suite, Steps, because the listener restarted the run for each suite and report mode wiped the results of the suites before it. With the run started once, all 15 tests of integration/robot/tests are in the report. The existing four entries are unchanged.
apetrosyan1613
approved these changes
Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
start_suitefor every suite in the hierarchy, and the listener calledreporter.start_run()each time. Inreportmode thatrmtrees the results directory, so runningrobot tests/witha/one.robot+b/two.robotlost suite A's results. Intestopsmode it was an extracheck_test_runcall per suite. The run is now started, filtered by the execution plan and registered only for the root suite.f"{test.name}:{test.lineno}", soLoginon line 2 ofauth/login.robotand ofbilling/login.robotcollided: one got the other's suite and signature. Keyed bytest.id(s1-s2-t1) now.src.runinwhile Truewith no sleep and no timeout, spinning at 100% CPU forever if process #0 failed to create the run. They now poll every 0.5s for up to 5 minutes, then log an error.end_user_keywordwithqase.params/qase.fieldsraisedAttributeErrorbefore the first test and modified the previous, already-reported test's result in a suite teardown. It now only applies inside a test.runtime.resultis deliberately not reset, soqase.attachin a suite teardown keeps working as before.Bumps qase-robotframework to 7.0.1.
Test plan
test_listener_lifecycle.py, including an end-to-endrobotrun in report mode with two suites that each haveLoginon the same line. 5 of 6 new tests fail on the old code and all pass after.testopsmode, Robot Framework 7.5 (two suites, same-name tests, a failing test, Suite Setup/Teardown keyword withqase.fields):robot: one run with 4 results (3 passed, 1 failed). The twoLogincases landed inTests/Auth/LoginandTests/Billing/Login. The suite keyword's field did not leak into any test.pabot --processes 2: one shared run with the same 4 results, matched to the same cases. Finished in about 10s with no hang.integration/robot/expected/robot-examples.yamlextended from 4 to 15 tests. The old one only described the last suite (Steps) because the earlier suites' results were being wiped — the same bug. The 4 existing entries are unchanged, andreporters-validatorpasses locally.