Skip to content

fix: don't mutate form_details in Node.derive_form - #1844

Open
OnuragDev wants to merge 7 commits into
scikit-hep:mainfrom
OnuragDev:fix-derive-form-mutation
Open

OnuragDev wants to merge 7 commits into
scikit-hep:mainfrom
OnuragDev:fix-derive-form-mutation

Conversation

@OnuragDev

@OnuragDev OnuragDev commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Node.derive_form mutates the node's own form_details. For ListOffsetArray/RegularArray nodes, out = dict(self._form_details) is only a shallow copy, so when the while loop descends into out["content"] and writes the child's form at the bottom, it writes into the node's stored nested dicts.

derive_form is called repeatedly by _discover_forth until the form has no NULLs. On the second call the loop walks into the stale child form and nests another list level each time. Reading an affected branch from a ROOT-written file fails with AttributeError: 'str' object has no attribute 'get'.

When it happens

vector<vector<T>> is not affected: each std::vector level gets its own Node whose form_details has no content, so the loop never descends.

The only case that puts a nested list chain into a single node is AsArray with an inner_shape (ListOffsetArray -> RegularArray[N]). With AwkwardForth enabled, that is a fixed-size array member of a split TClonesArray, e.g. std::vector<std::vector<float>> fVV[2].

Fix

Copy each level as the loop descends, so everything it writes into is a fresh dict and self._form_details is never touched.

Tests

  • tests/test_1844_derive_form_mutation.py: node-level test with the real ListOffsetArray -> RegularArray layout; calls derive_form() three times and checks form_details is unchanged and all results are equal. Fails on main.
  • tests/test_1844_tclonesarray_fixed_size_array.py: end-to-end test that writes a TClonesArray file with ROOT and compares the AwkwardForth read to the pure-Python read. Skipped when ROOT isn't installed. It is a separate file so the node-level test still runs without ROOT.

AI disclosure

See comment below.

@codecov

codecov Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.16%. Comparing base (30b7f1f) to head (3dc768a).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files
Files with missing lines Coverage Δ
src/uproot/_awkwardforth.py 100.00% <100.00%> (ø)

... and 13 files with indirect coverage changes

@OnuragDev OnuragDev changed the title Fix derive form mutation fix: don't mutate form_details in Node.derive_form Nov 16, 2026
@github-actions github-actions Bot added the type/fix PR title type: fix (set automatically) label Nov 16, 2026

@ariostas ariostas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI text below 🤖

Thanks for catching this! The bug is real, but it's triggered by a different case than the description says, and I'd suggest a smaller fix and a different test.

When it happens

vector<vector<T>> isn't affected: each std::vector level gets its own Node whose form_details has no content, so the while loop never descends and only writes into the shallow copy. The only container that puts a nested list chain into a single node is AsArray with an inner_shape (containers.py, ListOffsetArray → RegularArray[N]). With AwkwardForth enabled, that's a fixed-size array member of a split TClonesArray, e.g. std::vector<std::vector<float>> fVV[2].

On the first call, derive_form writes the child's form into the node's own RegularArray dict. If the elements are list-like and derive_form runs again (it's called once per entry in _discover_forth until the form has no NULLs), the loop walks into the stale child form and nests another list level each time. In a file written with ROOT, reading such a branch fails with AttributeError: 'str' object has no attribute 'get'.

Suggested fix

Copying each level as the loop descends is enough, and avoids deep-copying the whole form (including the innermost content, which is replaced right away):

             assert len(self._children) == 1
             returnme = out
             while out.get("content", {}).get("class") in (
                 "ListOffsetArray",
                 "RegularArray",
             ):
+                out["content"] = dict(out["content"])
                 out = out["content"]
             out["content"] = self._children[0].derive_form()
             return returnme

(The deepcopy works too, and I measured no meaningful performance difference, so this is mainly about keeping the fix minimal. The import copy would no longer be needed.)

Suggested tests

The current test only fails because its three nodes share the same dict objects, which uproot doesn't do. I'd replace it with these two, which both fail on main and pass with either fix.

A node-level test with the real layout:

def test_derive_form_fixed_size_array_of_lists():
    # AsArray with an inner shape (e.g. std::vector<float> fV[3] in a split
    # TClonesArray) puts ListOffsetArray -> RegularArray in a single node, whose
    # child is another list node.
    leaf = Node(
        "node3",
        form_details={
            "class": "NumpyArray",
            "primitive": "float32",
            "form_key": "node3",
        },
    )
    vector = Node(
        "node2",
        form_details={
            "class": "ListOffsetArray",
            "offsets": "i64",
            "parameters": {},
            "form_key": "node2",
        },
        children=[leaf],
    )
    array = Node(
        "node1",
        form_details={
            "class": "ListOffsetArray",
            "offsets": "i64",
            "content": {"class": "RegularArray", "size": 3},
            "parameters": {},
            "form_key": "node1",
        },
        children=[vector],
    )
    original = copy.deepcopy(array.form_details)

    # derive_form is called once per entry until the form has no NULLs
    forms = [array.derive_form() for _ in range(3)]

    assert array.form_details == original
    assert forms[0] == forms[1] == forms[2]
    regular = forms[0]["content"]
    assert regular["class"] == "RegularArray"
    assert regular["content"]["form_key"] == "node2"
    assert regular["content"]["content"] == leaf.form_details

And an end-to-end test that writes a matching file with ROOT (skipped when ROOT isn't installed, so it runs in the ROOT CI jobs) and compares the AwkwardForth result to the pure-Python read:

ROOT = pytest.importorskip("ROOT")

ROOT.gInterpreter.Declare("""
#include <vector>
#include "TClonesArray.h"
#include "TFile.h"
#include "TObject.h"
#include "TString.h"
#include "TTree.h"

class Hit1844 : public TObject {
public:
  TString fNames[3];
  std::vector<std::vector<float>> fVV[2];
  ClassDef(Hit1844, 1)
};

void write_hits_1844(const char *filename) {
  TFile f(filename, "RECREATE");
  TTree t("t", "t");
  TClonesArray *hits = new TClonesArray("Hit1844");
  t.Branch("hits", &hits, 32000, 99);
  // entry 0: no hits; entries 1-2: hits with empty vectors; entries 3-4: filled
  for (int entry = 0; entry < 5; entry++) {
    hits->Clear();
    for (int i = 0; i < entry; i++) {
      Hit1844 *hit = new ((*hits)[i]) Hit1844();
      for (int j = 0; j < 3; j++) hit->fNames[j] = TString::Format("e%d_i%d_s%d", entry, i, j);
      if (entry >= 3) {
        for (int j = 0; j < 2; j++)
          for (int k = 0; k <= j; k++)
            hit->fVV[j].push_back(std::vector<float>(k + 1, entry + 0.5f * i));
      }
    }
    t.Fill();
  }
  t.Write();
  f.Close();
  delete hits;
}
""")


@pytest.mark.parametrize("branch_name", ["hits.fNames[3]", "hits.fVV[2]"])
def test_tclonesarray_fixed_size_array_of_lists(tmp_path, branch_name):
    filename = str(tmp_path / "tclonesarray.root")
    ROOT.write_hits_1844(filename)

    with uproot.open(filename) as file:
        branch = file["t"][branch_name]
        assert branch.interpretation._forth
        forth = branch.array(library="ak").tolist()

        # the same branch, read without AwkwardForth
        interp = uproot.interpretation.objects.AsObjects(
            branch.interpretation.model, branch
        )
        interp._forth = False
        python = branch.array(interpretation=interp, library="ak").tolist()

    assert forth == python

The imports needed are copy, pytest, uproot, and from uproot._awkwardforth import Node. Could you also update the PR description to describe the TClonesArray fixed-size array case?

@ariostas

Copy link
Copy Markdown
Member

Also, @OnuragDev could you disclose if you used AI for this (and what model, if possible). Thanks!

@ariostas ariostas added this to the v5.7.8 milestone Nov 16, 2026
@OnuragDev

Copy link
Copy Markdown
Contributor Author

Also, @OnuragDev could you disclose if you used AI for this (and what model, if possible). Thanks!

Hey! Yes, I used AI for this PR. I used Claude Sonnet 5.5 (medium) to scan the file and find the issue, to suggest the fix. Gemini helped write the original test. I applied the changes and ran the tests myself, but the fix and test came from the AI, which is why my description had the wrong trigger case.
Sorry about that! I'm going through your suggestions now and will update the PR.

@OnuragDev

Copy link
Copy Markdown
Contributor Author

I've applied your suggested fix and both tests, and updated the description to cover the fixed-size array / TClonesArray case.

One small change: I moved the ROOT end-to-end test into its own file (tests/test_1844_tclonesarray_fixed_size_array.py), because the module-level importorskip("ROOT") was also skipping the node-level test when ROOT isn't installed. The node-level test fails without the fix and passes with it locally. I don't have ROOT here, so the end-to-end test is only covered by the ROOT CI jobs.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type/fix PR title type: fix (set automatically)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants