Skip to content

Add CC-IN2P3 (Lyon) support to the grid production scripts - #557

Open
jtenavidal wants to merge 8 commits into
masterfrom
develop/lyonscripts
Open

jtenavidal wants to merge 8 commits into
masterfrom
develop/lyonscripts

Conversation

@jtenavidal

Copy link
Copy Markdown
Contributor

Add --grid-system LYON to the eA, vA, hadronA and XML grid submitters. Jobs are submitted to the CC-IN2P3 Slurm cluster through a generated lyon_slurm_submit.sh, where each production stage depends on the successful completion of the previous one (sbatch --dependency=afterok), replacing the FNAL jobsub DAG.

  • LyonGridUtils.py: job scripts running in the node local scratch, sbatch options and the stage-chained main submission script
  • setup_LYON.sh / setup_GENIE_LYON.sh: environment from an LCG view on CVMFS, with optional use of a prebuilt GENIE on /sps
  • Spline and event generation command builders gain LYON branches; FNAL output is unchanged
  • Fix XMLGridSubmitter using non-existent NuEMAXSPLINE/EEMAXSPLINE options
  • Document LYON usage in the README

jtenavidal and others added 8 commits October 27, 2026 14:05
Add --grid-system LYON to the eA, vA, hadronA and XML grid submitters.
Jobs are submitted to the CC-IN2P3 Slurm cluster through a generated
lyon_slurm_submit.sh, where each production stage depends on the
successful completion of the previous one (sbatch --dependency=afterok),
replacing the FNAL jobsub DAG.

- LyonGridUtils.py: job scripts running in the node local scratch,
  sbatch options and the stage-chained main submission script
- setup_LYON.sh / setup_GENIE_LYON.sh: environment from an LCG view on
  CVMFS, with optional use of a prebuilt GENIE on /sps
- Spline and event generation command builders gain LYON branches;
  FNAL output is unchanged
- Fix XMLGridSubmitter using non-existent NuEMAXSPLINE/EEMAXSPLINE options
- Document LYON usage in the README

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lyon jobs no longer clone and compile GENIE on the worker nodes, this
is only done at FNAL. They use the compiled GENIE given by
--genie-topdir (default: $GENIE).

- The submitters abort for LYON if no GENIE top dir is given or it
  does not contain a compiled GENIE (bin/gevgen, bin/gmkspl), and warn
  if it is not in an area visible from the worker nodes
- setup_GENIE_LYON.sh only sets up the environment of that build and
  defaults GALGCONF to $GENIE/config; LYON_GENIE_PREBUILT is removed
- INCL tunes abort at LYON
- Builders no longer fail at import time when $GENIE is not set
- Update the README

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- compile_at_Lyon.sh: standalone script to compile GENIE at CC-IN2P3
  from a given git location and version. The configuration follows the
  FSI model of the tune (hA/hN, INCL++ or Geant4). It checks the
  dependencies and saves the environment used, with the Pythia8 path,
  as <genie dir>/setup_LYON.sh
- setup_LYON.sh: use the JUNO externals (ROOT, LHAPDF6, log4cpp) and
  Pythia8 from PYTHIA8PATH instead of the LCG view
- Submitters: the jobs use the environment saved with the GENIE build
  when available, and the Lyon setup scripts are taken from the
  scripts directory, so that any GENIE build can be used
- Document the compilation in the README

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Do not request local scratch with --tmp: the CC-IN2P3 Slurm
  configuration rejects it ("Temporary disk specification can not be
  satisfied"). --subjob-disk is not used at LYON
- lyon_slurm_submit.sh no longer uses set -e, which closed the login
  shell when the script was sourced and a submission failed. On a
  failed submission it now cancels the jobs already submitted and stops
- Update the README

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- At LYON the grouping jobs run gspladd directly in the spline
  directory, which is shared with the worker nodes, instead of copying
  the splines to the node. This also avoids copying the non-existent
  group_v*.xml file, which made the job fail. The intermediate
  per-target files are removed afterwards
- Fix total_xsec.xml when several probes are requested: only the last
  probe of each target was added, and no total was produced for a
  single target. Single probe outputs are unchanged

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FastOnNuclei on a nucleus is the slowest spline job. By default its
processes (CCQE, NCEL, CCRES, NCRES, CCDIS, NCDIS, CCDFR, NCDFR, CharmCCQE,
LambdaCCQE, SingleKaon, NuEElastic, IMD) are now computed in separate jobs.
Fast is still available through --vA-gen-list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only QE and DIS are slow on nuclei, so the rest of FastOnNuclei (RES, DFR,
CharmCCQE, LambdaCCQE, SingleKaon, NuEElastic, IMD) is computed in a single
FastOther job. gmkspl takes one event generator list, so this job runs gmkspl
for each list and adds the splines with gspladd. Fast and the single-process
jobs are still available through --vA-gen-list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--max-parallel-jobs N makes job k of each stage depend (afterany) on job
k-N of the same stage, so that at most N jobs of a stage run at the same
time. The limit can be changed when submitting with the MAX_PARALLEL_JOBS
environment variable. The default, 0, keeps the previous behaviour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant