output.yml schema 1.3: export levels, provenance, atom maps, IRC mapping and correction switches

[calvinp0]

The defect

output.yml left a consumer to infer things ARC knew, or could have recorded, and in several places the inference was wrong.

Formation enthalpies. When Arkane has no atom-energy corrections (AEC) for the level of theory and data/AEC.yml has no entry either, ARC renders the Arkane input with useAtomCorrections = False (arc/statmech/arkane.py, render_arkane_input_template). RMG-Py then sets the atom correction to zero (arkane/statmech.py), so the species' h298_kj_mol, NASA a6/b6 and tabulated H/G are raw electronic energies, about −10⁵ kJ/mol per heavy atom, not formation enthalpies.

output.yml gave a consumer no reliable way to tell. An atom_energy record in energy_corrections looks like proof that AEC was applied, but its absence proves nothing:

• The data/AEC.yml path applies AEC without writing a record.
• Failed recomputations drop records silently.
• Records were written from the key matched for arkane_level_of_theory, regardless of whether Arkane actually ran with corrections. With sp wb97xd/def2tzvp, a frequency level Arkane has no entry for and freq_scale_factor=None, get_arkane_model_chemistry returns None and Arkane runs uncorrected, yet output.yml still reported atom_energy and bond_additivity corrections as applied.

A downstream consumer (the TCKDB adapter) must label an enthalpy formation_298k only when it really is one, so it needs ARC to say so.

Everything else a consumer needs. The same gap exists elsewhere in the document:

• The level of theory of the job behind each exported log was held nowhere. Under adaptive_levels, a reaction-wide override or troubleshooting it differs from the run-level setting, and a consumer had only that setting.
• conformer_energies mixed units without saying so: a force-field energy is kcal/mol and a parsed electronic energy is kJ/mol. The docs also called them relative to the lowest conformer, which they are not.
• The atom map did not state the species order it counts in, or where it came from, and nothing related a reactant or product atom to an atom of the TS geometry. No ARC TS method guarantees that the TS lists its atoms in reactant order.
• reactant_labels and product_labels are sorted and de-duplicated, so HO2 + HO2 <=> H2O2 + O2 exported reactant_labels: [HO2] and a consumer could not balance the reaction.
• The IRC check decided by graph isomorphism and then discarded which fragment of each endpoint geometry was which species.
• TS frequencies were exported only in forms that drop the position of the reaction-coordinate mode, and a normal mode check forced to pass could not be told from a genuine pass.
• No program was stated for conformer jobs, and thermo read from Arkane's output.py stated no standard-state pressure.
• xtb_gsm staged its ograd script with a hard-coded --chrg 0 and no spin, so a charged or open-shell reaction was searched as a neutral closed-shell one.
• The Gaussian adapter rewrote self.level.basis and self.level.method in place, so the level ARC recorded for a job no longer matched the level requested.

The change

OUTPUT_SCHEMA_VERSION goes from 1.1 to 1.3. The schema is closed and every key a consumer relies on is required but nullable, so a consumer can tell "ARC does not know" (null) from "not applicable" without probing for absent keys. ARC states only what it observed or what the run recorded, and never rebuilds a value from a level or from job.args. The new keys are required in records that disallow extra properties, so 1.1 and 1.2 validators reject 1.3 output. docs/output_yml_schema.md and arc/schemas/output_yml_schema.json describe every key.

Correction switches (the 1.2 keys). Each species' thermo gains three required keys:

thermo:
  atom_corrections_applied: true      # true | false | null
  bond_corrections_applied: false     # true | false | null
  atom_corrections_level:             # sp_level-shaped dict, or null
    method: bmk
    basis: cbsb7
    method_type: dft
    software: gaussian

• The switches are the ones ARC actually rendered. The AEC/BAC decision is computed once in render_arkane_input_template and stored as use_aec / use_bac. The same values feed the Arkane template and the record, so they cannot disagree. parse_arkane_thermo_output stamps them only on species whose thermo this Arkane run wrote.
• null means unknown, never guessed. This covers thermo loaded from an Arkane YAML (StatMechJob.load returns before any correction step), thermo not produced by this run, and older output.
• true means only that the switch was on, not that H298 is a formation enthalpy. atom_corrections_level is the level the atom energies came from, so a consumer can compare it with the species' energy level. They differ when arkane_level_of_theory is a stand-in, as in examples/Stationary/bde/input.yml: bmk/cbsb7 corrections on apfd/def2svp energies.
• Correction records follow the switches. atom_energy / bond_additivity records are dropped for a species whose switch was off. They are left unchanged when the switch is unknown.
• The same switches travel with E0 and kinetics. They are stamped on the species' and the TS's E0 (statmech.e0_atom_corrections_applied, e0_bond_corrections_applied) and on the fitted kinetics (kinetics.atom_corrections_applied), since they decide what the E0 means. A species or reaction declared by an Arkane YAML is stamped null.
• Startup warnings. ARC warns at startup when arkane_level_of_theory differs from the energy level: the composite method or sp level, and any adaptive sp/composite level. The comparison normalises method and basis with Arkane's own name handling, and folds dispersion into the method whether it is written in the method string (b3lyp-d3bj) or in the separate dispersion field (gd3bj, EmpiricalDispersion=GD3BJ). ARC also warns when solvated energy levels are paired with a gas-phase arkane_level_of_theory: gas-phase atom energies are then subtracted from solvated energies, so the results are not formation enthalpies.

The JSON schema enforces the couplings with if/then rules:

• atom true requires a level;
• atom false requires bond false and a null level;
• atom null requires both others to be null.

docs/output_yml_schema.md states how a consumer should compare the two levels, and when a match still does not establish a formation enthalpy:

• adaptive levels that vary the sp level;
• a separate dispersion field;
• solvation.

Kinetics are unaffected in the numbers: the same AEC decision applies to reactants, TS and products in one run, so AEC cancels in barriers, and kinetics never uses BAC.

Levels and programs. Each species and TS record carries levels (opt, freq, sp, composite, irc), the level of the job behind each exported log, as recorded by the Scheduler. The header carries adaptive_levels plus the requested scan, IRC, conformer opt, conformer sp, TS guess and GSM levels, each null unless a job of that type could have run at it. ess_software and ess_version are identified from the logs themselves, now for composite, IRC and scan jobs as well.

Conformers. Records export conformer_levels, conformer_energy_kind (force field kcal/mol or electronic kJ/mol), conformer_energy_level and conformer_force_field, and conformer_ess_software / conformer_ess_version, read from the optimization log that produced each conformer geometry (null for a force-field conformer). Conformer energies are absolute. A null energy means the conformer has no energy, and the energy kind, level and force field describe only the other entries.

Atom maps and IRC. A reaction exports atom_map with atom_map_reactant_labels and atom_map_product_labels (the order of the species it counts in), atom_map_source (declared or inferred) and atom_map_method; the map must be a permutation that conserves the elements or it is null. TS records export irc_participant_mapping (which atoms of each endpoint belong to which participant, whether the sides are distinguishable, and atom_order_matches_ts), irc_log_routes, irc_log_directions, irc_log_levels, and IRC endpoint species state irc_endpoint_of and irc_endpoint_direction. atom_order_matches_ts is true only when the chain and element order below are verified, false when one is contradicted, and null when unknown. TS frequencies are exported in ESS order as freq_frequencies_cm1_ess_order, and nmd_forced states whether the normal mode check was forced to pass. Reactions state their species once per occurrence, in get_reactants_and_products order, as reactant_species_labels and product_species_labels; both are null, not [], for a reaction that holds no species objects.

Reaction coordinate mode. reaction_coordinate_mode_index is the 1-based position, in freq_frequencies_cm1_ess_order, of the mode the normal mode check validated. It comes from the NMD record's mode_index, and is stated only when the record's n_modes equals the number of exported frequencies, the log the check parsed is the exported frequency log, and the check genuinely passed. A forced check leaves it null.

TS atom map. A reaction exports ts_atom_map, the TS atom of every reactant and product atom (method irc_endpoint_cgr_isomorphism), or ts_atom_map_unavailable_reason. ARC does not assume an order: it takes a label-preserving isomorphism between two condensed graphs of reaction, one over the reactant atoms with every bond of the reactants or products (labelled by which side it is on) and one over the TS atoms with the bonds of the two IRC endpoints. The isomorphism fixes the atoms whose bonds form or break and leaves only graph automorphisms free, so the reactant and product legs agree with atom_map by construction. The endpoint graphs come from the IRC verdict's own perceived fragments, tied to the endpoint geometry by exact coordinates; no distance limit is applied. The map is a constitutional (2D) correspondence: symmetry-equivalent atoms, diastereotopic ones included, follow a deterministic convention that prefers the identity. ts_atom_order_follows_reactants states whether the identity is a solution.

Taking endpoint atom i as TS atom i is verified from the logs, not assumed:

• The first geometry of every IRC log (Gaussian's first input orientation, read by parse_irc_start_geometry) must be congruent with the TS geometry, atom for atom.
• The optimization log of endpoint k must start from a geometry congruent with the last geometry of IRC log k, and the endpoint geometry the check receives must equal the final geometry of that optimization log.
• Congruence is converter.kabsch with an RSSD of at most 1e-3 Å (print precision), proper rotations only, after the parsed geometry takes the isotopes of the reference. A mirror image is refused.
• Both endpoints must have the element sequence of the TS.

What remains rests on the ESS keeping its input atom order within a log and on ARC's parsing and writing keeping it. The map is also null when a participant's exported geometry is not in the atom order of its molecule (same elements in each position and every bond within ARC's are_coords_compliant_with_graph limit), a check that can only turn a map into null. When no map is stated, the reason is one of: no_ts, irc_not_passed, irc_fallback_path, no_atom_map, endpoint_perception_mismatch, atom_map_contradicts_ts, species_atom_order_mismatch, irc_start_geometry_unavailable, irc_start_geometry_differs, irc_endpoint_geometry_differs, computation_failed or not_recorded.

Parsed properties and routes. The effective route keywords of each job are read from the log in preference to the input deck. A Gaussian composite job (CBS-QB3, G4) exports composite_step_routes, the route of each Link1 step in the log's order; composite_route stays the first entry. A TS exports neb_succeeded: true when the log of a recorded orca_neb guess prints THE NEB OPTIMIZATION HAS CONVERGED, false when such guesses were recorded with readable logs and none prints it, null otherwise, and never inferred from a geometry, neb_level or a success flag. sp_t1_diagnostic is the T1 that parse_t1 reads from the exported sp log, with no gate on the method name. The Gaussian dipole moment with its density and the Gaussian polarizability (read by fixed-width field) are exported.

rigid_rotor_kind. It is read from the exported point group alone, with no moment of inertia and no tolerance. Cubic groups are spherical tops. Groups with a proper C_n axis of n >= 3 are symmetric tops, and so are D2d and S_n with even n >= 4: an improper rotation S_n(θ) is the inversion times the proper rotation C(θ+π), and the inversion acts trivially on the inertia tensor, so S_n acts as a rotation of order 3 or more and forces two moments equal (allene is a prolate symmetric top). C1, Cs, Ci, C2, C2v, C2h, D2 and D2h are asymmetric tops, and C∞v and D∞h are linear. It is null without a point group, and with non-default isotopes, since the point group is that of the unlabelled geometry.

Isotopes. Isotope lists accompany a geometry only when the log of that geometry prints its masses (Gaussian thermochemistry jobs, including each conformer's own log); otherwise they are null. ARC's own geometry dicts are never read for them.

Kinetics and reactions. Kinetics carry Arkane's comment and a ts_validation marker for a rate from a TS that failed its IRC check. Reactions state reversible from the arrow; ARC writes every reaction with <=>, so it is always true.

Arkane identity and treatment. The header exports the identity of the tables Arkane loaded (rmg_database: path_kind, git_commit, version, quantum_corrections_sha256; a mismatch with the file under ARC's RMG_DB_PATH is logged, not exported). arc_aec_yml_sha256 is the SHA-256 of ARC's data/AEC.yml, recorded when an Arkane adapter renders atom energies from it, and carried with each E0 as e0_aec_yml_sha256, so the E0 computed by the TS check is covered. It is exported when exactly one distinct digest exists, and is null otherwise. Petersson components list the bonds Arkane skipped (skipped_components), and statmech states arkane_rotors_applied and arkane_treatment. A rotor's treatment is read from the mode Arkane holds and is null when that is not known. Thermo read from Arkane's output.py states a standard-state pressure of 101325 Pa (1 atm). Applied bond corrections require a bac_type in the schema.

Correction records. An atom_energy or bond_additivity record is kept only when its switch is true; a switch that is off or unknown states no row.

Removed keys. rmg_database.quantum_corrections_path, rmg_database.matches_arc_rmg_db_path and irc_participant_mapping.<side>.endpoint are not exported.

Commits

1. 6337df90 level: record the level of a job as a plain dict. arc/level.py gains level_as_plain_dict, a deep-copied dictionary of plain types (the nested solvation scheme level is converted recursively, or dropped), so a recorded level survives a restart.yml round trip and a YAML dump. adaptive_levels_as_list converts the processed adaptive levels into the restart list form. set_recorded_level, set_recorded_irc_level and set_recorded_irc_log_level keep the per-species levels dictionary and the irc_levels list in step with the logs ARC stores; forward and reverse IRC logs are exported together, so a second log at a different level removes the record instead of stating either level.

2. 10917979 species: track conformer provenance, atom map origin, NMD and IRC participant records. ARCSpecies keeps conformer_levels, conformer_energy_sources ({kind, level, force_field}) and conformer_logs in lockstep with conformers, padded with None for an older restart. conformers.py records which force field and backend produced the energies of a generation, and in which unit, so the name is stated only when exactly one force field, backend and unit did; RDKit's MMFF to UFF fallback is reported as UFF, and the UFF energy path no longer asks for MMFF properties. ARCReaction records the origin of its atom map (inferred with algorithm and family, declared, or None). ARCSpecies keeps e0_atom_corrections_applied, e0_bond_corrections_applied and arkane_rotor_modes next to e0, and copy_e0_from takes the switches with the E0. The NMD check records the position of the mode it analysed (mode_index), and a TS keeps ts_atom_map and its unavailable reason. A queue TS-guess job whose log yields no usable geometry records a failed, coordinate-less orca_neb guess, as the in-core job does.

3. 09e2b03a ts: state IRC participants, endpoint atom order and the TS atom map, stage xtb_gsm. _assign_fragments_to_species replaces the boolean _match_fragments_to_species and returns the matching, which the IRC check uses to build irc_participant_mapping. The IRC check also builds ts_atom_map as described above and verifies the IRC start geometry and the endpoint chain; parse_irc_start_geometry is the new Gaussian parser, and the tolerance is IRC_START_GEOMETRY_TOLERANCE in arc/checks/common.py. A failure to build a record is logged and leaves the verdict, the participant mapping and the reaction's atom map untouched. The xtb_gsm ograd script becomes a template with @CHARGE@ and @UHF@ placeholders and an explicit --gfn 2, and the adapter refuses to stage when the charge or multiplicity is unknown. The Gaussian adapter applies its basis and method rewrites only to the values written to the input file and leaves the Level alone. The linear TS adapter uses the reaction's atom map state accessors.

4. fb697a92 pipe: record the level of ingested tasks and the IRC direction. The ingest functions build a Level from the task spec and record it the way the local scheduler does: the conformer geometry level and its log, the conformer energy source (electronic, kJ/mol), the freq level, and the IRC level and direction of each IRC log. A conf_opt task whose geometry parses but whose energy does not sets the energy to None instead of keeping the force-field energy.

5. 30dd1320 scheduler: record the level behind each job's log, and require a charge for TS search jobs. The Scheduler records the level of the job whose log it stores in output[label]['levels'] and, for IRC, paths['irc_levels'], as an independent plain dict that round-trips through restart.yml. The level comes from the job that ran, not the run-level setting. A composite job's freq level is left unknown, and the sp record is restored after a solvation scheme's extra sp jobs. IRC endpoint species record the direction of their IRC job, and the IRC check receives the IRC logs (irc_log_paths) and the endpoint opt logs (endpoint_log_paths). switch_ts clears the TS's E0 together with its correction switches and Arkane rotor modes. A conformer whose optimization returned no geometry no longer overwrites the stored one or its energy. TS search jobs are not spawned for a reaction whose charge is unknown.

6. 842a96d9 arkane: record whether corrections were applied, and match them on dispersion and solvation. Records the atom- and bond-correction switches ARC rendered into the Arkane input, and the level the atom energies came from, on each species' thermo, E0 and kinetics. ARC now writes the species' bonds into the Arkane species file, records which rotor modes the conformer block of the Arkane output holds, and derives the statmech treatment from them. get_qm_corrections.py reports the quantum_corrections/data.py path Arkane loaded. The adapter records the SHA-256 of data/AEC.yml when it renders atom energies from it, and process_arc_project returns the distinct digests. Correction keys (AEC, PBAC, MBAC) and data/AEC.yml are matched on a level's dispersion and solvation_method as well as method, basis, software and year (BREAKING, see below). Thermo read from output.py records the 1 atm standard state.

7. 82aeb4ab output: export levels, provenance, atom maps, IRC mapping and parsed properties (schema 1.3). Bumps the schema and exports everything listed under "The change". The main driver passes the requested levels to the writer, reports an NEB level only when the run has a TS or a reaction and orca_neb is among the adapters the scheduler uses, and emits the startup level warnings. The schema file, the docs, the parser evidence version and its golden files follow.

8. e1c2126e ARC.py: write the tckdb_arc upload's log records to arc.log. See the section below.

Correction matching: dispersion and solvation (BREAKING)

ARC's Arkane energy-correction matching (AEC, PBAC, MBAC and data/AEC.yml) now reads a level's dispersion and solvation_method. No existing match changes; this was checked against the previous matcher on every key in Arkane's database, with software, year and spelling variants. Two forms that matched nothing now match Arkane's B2PLYP-D3 entry: the Gaussian-style b2plyp-gd3 and b2plyp with dispersion: gd3.

A dispersion correction added to a separately parametrized "-D" functional (B97, ωB97X, ωB97M) matches none of that functional's variants. That holds whether it is given as a separate field or in Gaussian's g-prefixed spelling (b97-gd3). The functionals' own names (b97-d3, wb97x-d3, wb97xd) match as before.

BREAKING: some levels now get no Arkane energy corrections. Previously they silently got the gas-phase, non-dispersion ones. This covers:

• levels with a solvation_method;
• levels with a separate dispersion field for which Arkane has no dispersion-corrected entry, e.g. m062x with gd3.

Arkane's b3lyp2023/def2tzvp set was fitted on B3LYP-D3(BJ) (Wu et al., J. Phys. Chem. A 2024, 128, 4335) but is keyed as plain B3LYP. With the current RMG-database, plain b3lyp matches it and b3lyp + gd3bj gets no corrections; RMG-database #747 relabels it b3lypd3bj2023, after which the two swap. The dispersion-matching tests use a synthetic corrections file, so they pass with either database.

What happens next depends on the run:

• Arkane thermo adapter with compute_thermo on (the default): ARC stops at startup. This includes restarting an existing project that used such a level. The message names the remedies: compute_thermo: false, or an explicit gas-phase / non-dispersion arkane_level_of_theory, which ARC then warns mixes levels. docs/source/advanced.rst describes them.
• compute_thermo: false, e.g. rates only: Arkane runs with atom corrections off. That cancels in rate coefficients, and output.yml carries no correction records for those levels.

Frequency-key matching is unchanged. A missing frequency key drops the whole Arkane model chemistry, and with it the corrections of a correct gas-phase sp level.

arc.log records the TCKDB upload

tckdb_arc logs to its own tckdb_arc logger and attaches no handler, so during the post-run upload its INFO records were dropped and its warnings reached only stderr. A run whose TCKDB server was unreachable therefore ended arc.log at "Uploading ARC results to TCKDB …" without saying the upload failed or giving the tckdb-arc-upload command that recovers it. arc/common.py gains route_logger_to_arc_log(name, level=INFO), a context manager that passes a named logger's records to ARC's console and arc.log handlers for one block. It turns off that logger's propagation for the block so nothing is written twice, and restores the logger's handlers, level and propagation on exit, including on error. Each ARC handler's own level still applies, so a quiet run keeps INFO out of both outputs. run_tckdb_upload wraps the config parsing, adapter and sweep in it.

Behaviour changes outside output.yml

• The Gaussian adapter no longer mutates the level. The def2- to def2 and cbs-qb3-paraskevas to cbs-qb3 rewrites apply only to the values written to the input file, so the Level ARC records for a job is the one requested.
• xtb_gsm is staged with charge and spin, and refuses without them. The script passes --gfn 2, --chrg with the reaction charge and --uhf with multiplicity minus one. Before, every search ran as neutral and closed-shell.
• TS searches need a known charge. The Scheduler spawns no TS search job, for any adapter, when the reaction charge is unknown, as it already did not for an unknown multiplicity. It is no longer only xtb_gsm that refuses.
• switch_ts clears the TS E0, with its correction switches and Arkane rotor modes. Before, compute_rxn_e0 skipped a TS whose e0 was set, so the E0 check of the next guess reused the E0 of the abandoned guess. Resetting a species' E0 also resets the switches and rotor modes recorded with it, and resetting its conformers resets their provenance.
• The pipe conf_opt energy is None when it is not parsed, instead of keeping the force-field energy that was there.
• parse_conformer. A finished conf_opt job with no parseable geometry keeps the stored geometry and the force-field energy. On main it set both to None.
• Conformer force fields. RDKit's MMFF to UFF fallback is reported as UFF, and the UFF path no longer first asks RDKit for MMFF94 molecule properties it does not use.
• Arkane species file. ARC writes the species' bonds = {...} into it, so Arkane's BAC and the exported Petersson components use one bond dictionary.
• Queue-path orca_neb. A queue-path orca_neb job whose log gives no usable geometry is recorded as a failed TS guess without coordinates, as the in-core job does. Before, nothing was recorded.
• IRC check. The IRC check now reads the IRC logs (irc_log_paths) and the endpoint opt logs (endpoint_log_paths), and parse_irc_start_geometry is added to the parsers.

Tests

Results (serial, at e1c2126e):

Module	Result
arc/output_test.py	523 passed
arc/output_schema_test.py	140 passed
arc/job/adapters/ts/orca_neb_test.py	8 passed
arc/checks/nmd_test.py	56 passed
arc/species/species_test.py	111 passed
arc/reaction/reaction_test.py	52 passed
arc/statmech/arkane_test.py	91 passed, 1 skipped (needs rmgpy)
arc/processor_test.py	3 passed
arc/main_test.py	55 passed
arc/checks/ts_test.py	82 passed
arc/parser/parser_test.py	116 passed
arc/parser/adapters/hessian_test.py	15 passed
arc/job/pipe/pipe_run_test.py	76 passed
arc/parser_evidence_test.py	39 passed
arc/level_test.py	23 passed
arc/species/conformers_test.py	70 passed
arc/job/adapters/gaussian_test.py	25 passed
arc/job/adapters/ts/xtbgsm_test.py	15 passed
arc/job/adapters/ts/linear_test.py	137 passed
arc/scheduler_test.py	200 passed
arc/arc_cli_test.py	16 passed
arc/common_test.py	73 passed

arc/output_schema_test.py drives the real writer over real ARCSpecies and ARCReaction objects and validates the document against arc/schemas/output_yml_schema.json; its negative cases assert that the schema rejects a missing key and each inconsistent combination.

Consumers

The TCKDB adapter (tckdb-arc; its 0.9.x branches) reads schema 1.3. It must treat a null species-label key (reactant_species_labels, product_species_labels) as "not stated", and read neb_succeeded. The adapter turns ts_atom_map into TCKDB's per-participant atom_to_ts by slicing it with the participants' atom counts in atom_map_reactant_labels / atom_map_product_labels order; TCKDB's validate_reaction_atom_map accepts the result for H-abstraction, A + A and degenerate reactions.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 68.05%. Comparing base (133b8c1) to head (e1c2126).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1059      +/-   ##
==========================================
+ Coverage   67.09%   68.05%   +0.96%     
==========================================
  Files         123      123              
  Lines       43663    44981    +1318     
  Branches    11152    11399     +247     
==========================================
+ Hits        29295    30613    +1318     
+ Misses      11220    11186      -34     
- Partials     3148     3182      +34     

Flag	Coverage Δ
functionaltests	68.05% <ø> (+0.96%)	⬆️
unittests	68.05% <ø> (+0.96%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.