Skip to content

ccqe_spline_smoothing: model names, electron support, and a streaming replace_splines.py - #568

Open
nusense wants to merge 1 commit into
GENIE-MC:masterfrom
nusense:rhatcher/claude_smoothing_updates
Open

nusense wants to merge 1 commit into
GENIE-MC:masterfrom
nusense:rhatcher/claude_smoothing_updates

Conversation

@nusense

@nusense nusense commented Nov 17, 2026

Copy link
Copy Markdown
Member

These scripts average seeded QE spline runs, fit a smooth representative spline,
and put it back into the full spline file. Using them in a full production
(all six flavors, ~125 isotopes, 100–1000 seeds per pair, several tunes)
turned up the problems fixed here.

construct_representative_spline.py

  • Wrong or missing model names. The smooth spline is matched to the big file
    by name, so a wrong model name means it is never used. An exact models entry
    is now checked first: before, it was checked last, so entries such as
    G18_10a_02_11b were never reached. G18_10a/b/c/d now get
    NievesQELCCPXSec/Dipole, which is what their ModelConfiguration.xml uses,
    instead of LwlynSmithQELCCPXSec/Dipole. Added, each taken from that tune's
    ModelConfiguration.xml:
    • AR23/AR25 20i sub-tunes (01_002, 02_001), N24_20i_02_11b
    • AR23/AR25 20m and their 01_000/001/002 sub-tunes
    • AR23/AR25 22i/22m (HybridXSecAlgorithm/CRPA-QEL)
    • G24_12a (HybridXSecAlgorithm/Martini-QEL)
    • GEM26 (UnifiedQELPXSec/Dipole)
  • Unknown tunes: a tune not covered now stops with a message instead of a KeyError.
  • Electrons (GEM26): spline process EM,QES, and the struck nucleon is given
    with --nucleon 2112|2212, since electron jobs have one spline per nucleon.
  • The Gaussian-process fit:
    • With normalize_y=True, sklearn divides the targets by their standard
      deviation but not alpha, so the noise term was in the wrong units. It is
      now (err / std(mean))**2, with a floor so that err == 0 can't break it.
    • The RBF length scale now has a lower bound of 0.3 in ln E (about 6 knots
      of a 250-knot, 0.01–1000 GeV grid). Without it the optimiser sometimes
      settled on a length scale of a few knots and followed the noise.

average_splines.py

  • Every spline is kept: all non-free-nucleon splines in the runs are kept,
    not only the first one, matched by name. With one spline the output is as
    before; with several (electron jobs) there is one parquet per spline,
    <outbase>_nu<pdg>_tgt<pdg>_N<pdg>.parquet.
  • Mean instead of median: the average is now the mean over runs, zeros
    included. With one nucleon throw per run, many runs are exactly zero near
    threshold. The median there is 0, or biased low, while the mean is the
    unbiased estimate of the cross section.
  • Knot check: knot energies are checked to be the same in every run. Runs
    missing a spline are reported and left out of its mean.

replace_splines.py

The old version parsed the whole base file into ElementTree. For a full neutrino
file (1.3 GB uncompressed) that is far more than 4 GB of memory, enough to stall
a grid job. It also took one revision file per run, re-parsing the base file
each time. It skipped any unmatched spline silently, and left nknots
unchanged, although GENIE reads exactly nknots knots. The rewrite:

  • reads the base file line by line (.xml or .xml.gz; output gzipped if -o
    ends in .gz)
  • takes any number of revision files: -r f1.xml f2.xml ... and/or -l list.
    The old one-file -b/-r/-o usage still works
  • sets nknots to the new knot count and prints a summary (replaced, not
    found, knot count changed)
  • writes no output and exits 1 if a revision spline isn't in the base file
    (--allow-missing to accept that), has no knots, is in two revision files,
    or is for another tune (--ignore-tune)
  • needs only the Python standard library

It drops the unused models dict and the commented-out renaming code.

Testing

  • replace_splines.py: on the G24_12a_00_000 full neutrino file (1.3 GB),
    with a stand-in revision spline, it ran in about 14 s. Its output was
    identical, line for line, to an independent awk implementation of the same
    replacement. The not-found, wrong-tune and changed-knot-count cases behave
    as described.
  • average_splines.py and construct_representative_spline.py: used in full
    AR23_20i, AR25_20i_01_002, G18_02a and GEM26 productions. The smooth
    splines are 4–300× smoother than the standard run away from threshold, and
    differ from it by about the standard run's noise.

Note

Martini-QEL, CRPA-QEL and SuSAv2-QEL are integrated with MECXSec (VEGAS over
q0, q3), not with NewQELXSec's nucleon throws. So the seed and
NumNucleonThrows don't change them: every seeded run gives the same spline
(checked for Martini-QEL), and averaging seeds does nothing for G24_12a,
AR23/AR25 22i/22m and G21_11*. Their model names are added anyway, so the
scripts don't stop on those tunes.

…replace_splines.py

- construct_representative_spline.py: an exact tune entry now wins over the
  prefix rules; G18_10a-d use NievesQELCCPXSec/Dipole (not LwlynSmith);
  add AR23/AR25 20i sub-tunes, 20m, 22i/22m (CRPA), G24_12a (Martini),
  GEM26 (UnifiedQEL, proc EM,QES, --nucleon); unknown tunes stop with a
  message instead of a KeyError. GP: scale alpha for normalize_y and put a
  floor of 0.3 on the RBF length scale so the fit doesn't follow the noise.
- average_splines.py: keep every non-free-nucleon spline (one parquet each,
  for electron jobs with N:2112 and N:2212); mean over runs, zeros included,
  instead of the median; check that knot energies match across runs.
- replace_splines.py: rewritten to stream the base file line by line (.xml
  or .gz), take many revision files (-r ... / -l list), set nknots, and fail
  (no output) on unmatched, empty, duplicated or other-tune splines.
@nusense
nusense requested a review from kjplows November 17, 2026 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant