Repository navigation
Conversation
…utation of nested form_details
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
|
ariostas
left a comment
There was a problem hiding this comment.
🤖 AI text below 🤖
Thanks for catching this! The bug is real, but it's triggered by a different case than the description says, and I'd suggest a smaller fix and a different test.
When it happens
vector<vector<T>> isn't affected: each std::vector level gets its own Node whose form_details has no content, so the while loop never descends and only writes into the shallow copy. The only container that puts a nested list chain into a single node is AsArray with an inner_shape (containers.py, ListOffsetArray → RegularArray[N]). With AwkwardForth enabled, that's a fixed-size array member of a split TClonesArray, e.g. std::vector<std::vector<float>> fVV[2].
On the first call, derive_form writes the child's form into the node's own RegularArray dict. If the elements are list-like and derive_form runs again (it's called once per entry in _discover_forth until the form has no NULLs), the loop walks into the stale child form and nests another list level each time. In a file written with ROOT, reading such a branch fails with AttributeError: 'str' object has no attribute 'get'.
Suggested fix
Copying each level as the loop descends is enough, and avoids deep-copying the whole form (including the innermost content, which is replaced right away):
assert len(self._children) == 1
returnme = out
while out.get("content", {}).get("class") in (
"ListOffsetArray",
"RegularArray",
):
+ out["content"] = dict(out["content"])
out = out["content"]
out["content"] = self._children[0].derive_form()
return returnme(The deepcopy works too, and I measured no meaningful performance difference, so this is mainly about keeping the fix minimal. The import copy would no longer be needed.)
Suggested tests
The current test only fails because its three nodes share the same dict objects, which uproot doesn't do. I'd replace it with these two, which both fail on main and pass with either fix.
A node-level test with the real layout:
def test_derive_form_fixed_size_array_of_lists():
# AsArray with an inner shape (e.g. std::vector<float> fV[3] in a split
# TClonesArray) puts ListOffsetArray -> RegularArray in a single node, whose
# child is another list node.
leaf = Node(
"node3",
form_details={
"class": "NumpyArray",
"primitive": "float32",
"form_key": "node3",
},
)
vector = Node(
"node2",
form_details={
"class": "ListOffsetArray",
"offsets": "i64",
"parameters": {},
"form_key": "node2",
},
children=[leaf],
)
array = Node(
"node1",
form_details={
"class": "ListOffsetArray",
"offsets": "i64",
"content": {"class": "RegularArray", "size": 3},
"parameters": {},
"form_key": "node1",
},
children=[vector],
)
original = copy.deepcopy(array.form_details)
# derive_form is called once per entry until the form has no NULLs
forms = [array.derive_form() for _ in range(3)]
assert array.form_details == original
assert forms[0] == forms[1] == forms[2]
regular = forms[0]["content"]
assert regular["class"] == "RegularArray"
assert regular["content"]["form_key"] == "node2"
assert regular["content"]["content"] == leaf.form_detailsAnd an end-to-end test that writes a matching file with ROOT (skipped when ROOT isn't installed, so it runs in the ROOT CI jobs) and compares the AwkwardForth result to the pure-Python read:
ROOT = pytest.importorskip("ROOT")
ROOT.gInterpreter.Declare("""
#include <vector>
#include "TClonesArray.h"
#include "TFile.h"
#include "TObject.h"
#include "TString.h"
#include "TTree.h"
class Hit1844 : public TObject {
public:
TString fNames[3];
std::vector<std::vector<float>> fVV[2];
ClassDef(Hit1844, 1)
};
void write_hits_1844(const char *filename) {
TFile f(filename, "RECREATE");
TTree t("t", "t");
TClonesArray *hits = new TClonesArray("Hit1844");
t.Branch("hits", &hits, 32000, 99);
// entry 0: no hits; entries 1-2: hits with empty vectors; entries 3-4: filled
for (int entry = 0; entry < 5; entry++) {
hits->Clear();
for (int i = 0; i < entry; i++) {
Hit1844 *hit = new ((*hits)[i]) Hit1844();
for (int j = 0; j < 3; j++) hit->fNames[j] = TString::Format("e%d_i%d_s%d", entry, i, j);
if (entry >= 3) {
for (int j = 0; j < 2; j++)
for (int k = 0; k <= j; k++)
hit->fVV[j].push_back(std::vector<float>(k + 1, entry + 0.5f * i));
}
}
t.Fill();
}
t.Write();
f.Close();
delete hits;
}
""")
@pytest.mark.parametrize("branch_name", ["hits.fNames[3]", "hits.fVV[2]"])
def test_tclonesarray_fixed_size_array_of_lists(tmp_path, branch_name):
filename = str(tmp_path / "tclonesarray.root")
ROOT.write_hits_1844(filename)
with uproot.open(filename) as file:
branch = file["t"][branch_name]
assert branch.interpretation._forth
forth = branch.array(library="ak").tolist()
# the same branch, read without AwkwardForth
interp = uproot.interpretation.objects.AsObjects(
branch.interpretation.model, branch
)
interp._forth = False
python = branch.array(interpretation=interp, library="ak").tolist()
assert forth == pythonThe imports needed are copy, pytest, uproot, and from uproot._awkwardforth import Node. Could you also update the PR description to describe the TClonesArray fixed-size array case?
|
Also, @OnuragDev could you disclose if you used AI for this (and what model, if possible). Thanks! |
Hey! Yes, I used AI for this PR. I used Claude Sonnet 5.5 (medium) to scan the file and find the issue, to suggest the fix. Gemini helped write the original test. I applied the changes and ran the tests myself, but the fix and test came from the AI, which is why my description had the wrong trigger case. |
|
I've applied your suggested fix and both tests, and updated the description to cover the fixed-size array / One small change: I moved the ROOT end-to-end test into its own file ( |
Problem
Node.derive_formmutates the node's ownform_details. ForListOffsetArray/RegularArraynodes,out = dict(self._form_details)is only a shallow copy, so when thewhileloop descends intoout["content"]and writes the child's form at the bottom, it writes into the node's stored nested dicts.derive_formis called repeatedly by_discover_forthuntil the form has no NULLs. On the second call the loop walks into the stale child form and nests another list level each time. Reading an affected branch from a ROOT-written file fails withAttributeError: 'str' object has no attribute 'get'.When it happens
vector<vector<T>>is not affected: eachstd::vectorlevel gets its ownNodewhoseform_detailshas nocontent, so the loop never descends.The only case that puts a nested list chain into a single node is
AsArraywith aninner_shape(ListOffsetArray -> RegularArray[N]). With AwkwardForth enabled, that is a fixed-size array member of a splitTClonesArray, e.g.std::vector<std::vector<float>> fVV[2].Fix
Copy each level as the loop descends, so everything it writes into is a fresh dict and
self._form_detailsis never touched.Tests
tests/test_1844_derive_form_mutation.py: node-level test with the realListOffsetArray -> RegularArraylayout; callsderive_form()three times and checksform_detailsis unchanged and all results are equal. Fails onmain.tests/test_1844_tclonesarray_fixed_size_array.py: end-to-end test that writes aTClonesArrayfile with ROOT and compares the AwkwardForth read to the pure-Python read. Skipped when ROOT isn't installed. It is a separate file so the node-level test still runs without ROOT.AI disclosure
See comment below.