Skip to content
25 changes: 23 additions & 2 deletions src/scripts/production/python/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@ An example of a Submitter code is `eAScatteringGridSubmitter.py`. This code will
- git-branch: This is the name of the git branch you want to pull from the GENIE Generator repository. Default is master
- cycle : defult 01
- production: default routine_validation
- grid-system: for now only FNAL available
- grid-group: you can specify your grid group here. I.E for FNAL: genie, dune,...
- grid-system: FNAL (default) or LYON (CC-IN2P3, see below)
- grid-group: you can specify your grid group here. I.E for FNAL: genie (default), dune,... For LYON it is passed as the Slurm account (default: your main group)
- softw-topdir: base directory for all your GENIE code.
- genie-topdir: path to your GENIE/Generator code
- jobs-topdir: directory where you want to have your output files and submission commands. For FNAL it must be in pnfs.
Expand Down Expand Up @@ -77,3 +77,24 @@ In some cases, the user might desire to change the model configuration. For inst
For instance:
`python eAScatteringGridSubmitter.py --probe-list 11 --tgt-list 1000060120,1000080160 --energy 1.6,2.2,3 --submit-jobs --starting-point 4 --source-prod-dir MySplines/ --git-branch my_branch_name --jobs-topdir <jobs-topdir> --tune G18_10a_02_11b --conf-dir <jobs-topdir>/conf`

## Running at CC-IN2P3 (Lyon)
Use `--grid-system LYON` to create the jobs for the Slurm batch system of CC-IN2P3. The options and the directory structure are the same as for FNAL, with these differences:

- GENIE is NOT cloned and compiled in each job (this is only done at FNAL). The jobs use the compiled GENIE in `--genie-topdir`, which defaults to `$GENIE`. The submitter aborts if no compiled GENIE (`bin/gevgen`, `bin/gmkspl`) is found there. `--git-location` and `--git-branch` are ignored. If you do not have a GENIE build yet, compile one with `compile_at_Lyon.sh` (see below).
- jobs-topdir must be in an area shared with the worker nodes, i.e. `/sps/<group>/...` (jobs in `/sps` automatically request the `sps` Slurm licence).
- Instead of `fnal_dag_submit.fnal` and `grid_submission.xml`, a `lyon_slurm_submit.sh` script is written. It submits every job with `sbatch`. Each stage (vN splines, grouping, vA splines, grouping, events) only starts when all the jobs of the previous stage succeeded (`--dependency=afterok`); if one fails the remaining stages are cancelled. To submit it manually do `bash <jobs-topdir>/lyon_slurm_submit.sh` (it can also be sourced). If a submission fails, the jobs already submitted are cancelled.
- --max-parallel-jobs N limits the number of jobs of each stage running at the same time: job k of a stage starts once job k-N of the same stage has ended (`--dependency=afterany`). The default, 0, does not limit them; the jobs are then queued by Slurm. The limit can be changed when submitting, without generating the scripts again: `MAX_PARALLEL_JOBS=100 bash <jobs-topdir>/lyon_slurm_submit.sh`.
- Each job runs in the node local scratch (`$TMPDIR`), copies its inputs there with `cp` and copies the outputs back to the jobs directory. The Slurm log of every job is written next to its script as `<job_name>.<job_id>.log`
- GENIE must run with the environment it was compiled with. `setup_LYON.sh` takes ROOT, LHAPDF6 and log4cpp from the JUNO externals in CVMFS, and Pythia8 from `$PYTHIA8PATH`. The submitter copies it to jobs-topdir from, in this order: an existing `<jobs-topdir>/setup_LYON.sh` (so you can edit it), the copy saved with the build by `compile_at_Lyon.sh` (`<genie-topdir>/setup_LYON.sh`), or the default one in this directory.
- --job-lifetime-* options are the Slurm time limit (in hours) of each job; --subjob-memory is the requested memory. --subjob-disk is not used: the local scratch space can not be requested at CC-IN2P3. The main job options (--main-lifetime, --mainjob-memory, --mainjob-disk) are not used.
- INCL tunes are not supported by the Lyon job scripts.

For example:
`python eAScatteringGridSubmitter.py --grid-system LYON --genie-topdir /sps/<group>/<user>/Generator --probe-list 11 --e-tgt-list 1000060120,1000080160 --ebeam-energy 1.6,2.2,3 --jobs-topdir /sps/<group>/<user>/genie_jobs`

### Compiling GENIE at Lyon
`compile_at_Lyon.sh` is a standalone script to compile GENIE once, if you do not have a build yet. It is not run by the jobs. It clones `--git-location` at `--version` (branch or tag), sources `setup_LYON.sh` and configures GENIE for the FSI model of `--tune`: tunes `Gxx_yya_...` (hA) and `Gxx_yyb_...` (hN) use the standard build, `c` (INCL++) needs `--incl-dir` and `--boost-dir`, and `d` (Geant4) needs `--geant4-setup`. Use `--fsi` if the tune name does not follow this convention. Run it with `--help` for all the options.

`./compile_at_Lyon.sh --version R-3_06_02 --tune G18_10a_02_11b --pythia8-dir /sps/<group>/<user>/pythia8315 --topdir /sps/<group>/<user>/genie`

It prints the `--genie-topdir` to use with the submitters, and saves the environment used, with the Pythia8 path, in `<genie-topdir>/setup_LYON.sh` for the jobs.
57 changes: 46 additions & 11 deletions src/scripts/production/python/XMLGridSubmitter.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
import vASplineCommands as vA
import GroupSplineCommands as group
import FNALGridUtils as FNAL
import LyonGridUtils as LYON

sys.path.insert(1, 'event_generation')
import eScatteringGenCommands as eA
Expand All @@ -32,8 +33,8 @@
op.add_option("--cycle", dest="CYCLE", default="01", help="Cycle (default: %default)")
op.add_option("--arch", dest="ARCH", default='SL6.x86_64', help="arch number, default: %default")
op.add_option("--production", dest="PROD", default="routine_validation", help="Production (default: %default)")
op.add_option("--grid-system", dest="GRID", default="FNAL", help="Grid system is %default ")
op.add_option("--grid-group", dest="GROUP", default="genie", help="Grid group to run your jobs. I.E: FNAL groups are genie, dune, etc..")
op.add_option("--grid-system", dest="GRID", default="FNAL", help="Grid system: FNAL or LYON (CC-IN2P3). Default: %default ")
op.add_option("--grid-group", dest="GROUP", default="", help="Grid group to run your jobs. I.E: FNAL groups are genie (default), dune, etc.. At LYON it is the Slurm account (default: your main group)")
op.add_option("--softw-topdir", dest="SOFTW", default=os.getenv('GENIE_MASTER_DIR'), help = "software top dir")
op.add_option("--genie-topdir", dest="GENIE", default=os.getenv('GENIE'), help = "GENIE topdir: %default")
op.add_option("--jobs-topdir", dest="JOBSTD", default=os.getenv('PWD'), help="Top level dir for the job files (default: %default)")
Expand All @@ -43,7 +44,7 @@
op.add_option("--nu-tgt-list", dest="NUTGTLIST", default='all', help = "Comma separated list of Targets. Default: %default.")
op.add_option("--e-tgt-list", dest="ETGTLIST", default='all', help = "Comma separated list of Targets. Default: %default.")
op.add_option("--vN-gen-list", dest="vNList", default='all', help="Comma separated list of event generator list used for the free nucleon spline generation. Can be used to specify electron procecess as well")
op.add_option("--vA-gen-list", dest="vAList", default='all', help="Comma separated list of event generator list used for the nuclei spline generation. Can be used to specify electron procecess as well")
op.add_option("--vA-gen-list", dest="vAList", default='all', help="Comma separated list of event generator list used for the nuclei spline generation. Can be used to specify electron procecess as well. By default FastOnNuclei is run in three jobs: QE, DIS and FastOther (the rest). Use Fast to run it in a single job, or CCQE,NCEL,CCRES,... to run each process on its own")
op.add_option("--e-nu-max", dest="NuEMAX", type="int", default=100, help="Maximum energy for the splines in GeV. Default: %default ")
op.add_option("--e-espline-max", dest="EEMAX", type="int", default=30, help="Maximum energy for the splines in GeV. Default: %default ")
op.add_option("--n-nu-knots", dest="NuKnots", type="int", default=100, help="Number of knots per neutrino spline. Default: %default")
Expand All @@ -60,6 +61,7 @@
op.add_option("--mainjob-disk", dest="MAINDISK", default=20, help="Main job disk usate, default %default")
op.add_option("--subjob-memory", dest="JOBMEM", default="2GB", help="Expected memory usage for each event generation job, default %default. Notice you must specify the units. ")
op.add_option("--subjob-disk", dest="JOBDISK", default="2GB", help="Expected disk space for each event generation job, default %default. Notice you must specify the units. ")
op.add_option("--max-parallel-jobs", dest="MAXPARALLEL", type="int", default=0, help="LYON only: maximum number of jobs of each stage running at the same time, 0 for no limit. Default: %default")
op.add_option("--store-comitinfo", dest="STORECOMMIT", default=False, action="store_true", help="Store command line in jobstopdir directory")
opts, args = op.parse_args()

Expand Down Expand Up @@ -90,21 +92,37 @@
exit()

# Check jobstopdir exists
if not os.path.exists(opts.GENIE) :
print ( "GENIE Path "+opts.GENIE+" doesn't exist. Abort...\n")
if not opts.GENIE or not os.path.exists(opts.GENIE) :
print ( "GENIE Path "+str(opts.GENIE)+" doesn't exist. Specify it with --genie-topdir. Abort...\n")
exit()

if opts.GRID != 'FNAL':
print ("For now only FNAL option is available. Abort... \n")
if opts.GRID != 'FNAL' and opts.GRID != 'LYON':
print ("For now only FNAL and LYON options are available. Abort... \n")
exit()
# In the future we might have other grids

# Check we run from pnfs persistent or scratch for FNAL:
if opts.GRID == 'FNAL':
if 'pnfs' not in opts.JOBSTD :
print ("Not runing from pnfs:"+opts.JOBSTD+" . Jobs top dir must be in pnfs for the submission scrpits to work. Abort ...")
exit()

if opts.GRID == 'FNAL' and opts.GROUP == '' :
opts.GROUP = 'genie'

# At LYON GENIE is not built on the worker nodes: the jobs use the compiled GENIE in --genie-topdir (default: $GENIE)
if opts.GRID == 'LYON':
if not os.path.exists(opts.GENIE+"/bin/gevgen") or not os.path.exists(opts.GENIE+"/bin/gmkspl") :
print ("No compiled GENIE found in "+opts.GENIE+" . At LYON the jobs use an existing GENIE build: specify its top dir with --genie-topdir. Abort ...")
exit()
print ( ' Using compiled GENIE from '+opts.GENIE+' (--git-location and --git-branch are only used at FNAL)' )
if not opts.GENIE.startswith('/sps') and not opts.GENIE.startswith('/pbs/throng') :
print ("Warning: GENIE top dir "+opts.GENIE+" is not in /sps or /pbs/throng. Make sure it is accessible from the worker nodes.")

# Check the jobs top dir is shared with the worker nodes for LYON:
if opts.GRID == 'LYON':
if not opts.JOBSTD.startswith('/sps') and not opts.JOBSTD.startswith('/pbs/throng') :
print ("Warning: jobs top dir "+opts.JOBSTD+" is not in /sps or /pbs/throng. Make sure it is accessible from the worker nodes.")

if opts.CONF :
if not os.path.exists(opts.CONF) :
print ( " GENIE Configuraion dir specified does not exist: " + opts.CONF + " . Abort ..." )
Expand All @@ -120,7 +138,7 @@
version = temp_version[len(temp_version)-1]
print ( ' Setting version to '+ version )

if opts.BRANCH:
if opts.BRANCH and opts.GRID == 'FNAL':
print( ' Cloning GENIE Generator ' + opts.BRANCH )

os.system("source /cvmfs/fermilab.opensciencegrid.org/products/common/etc/setup" )
Expand All @@ -145,6 +163,21 @@
os.system('cp '+GENIE_setup_file+' '+opts.JOBSTD )
grid_setup = opts.JOBSTD+"/setup_FNAL.sh"
genie_setup = opts.JOBSTD+"/setup_GENIE.sh"
elif opts.GRID == 'LYON' :
# The jobs source the setup scripts directly from the jobs top dir. An existing setup_LYON.sh is kept, so that local edits are not lost.
# Otherwise the environment saved with the GENIE build by compile_at_Lyon.sh is used: GENIE must run with the one it was compiled with
scripts_dir = os.path.dirname(os.path.abspath(__file__))
if os.path.exists(opts.JOBSTD+"/setup_LYON.sh") :
print( ' Using existing '+opts.JOBSTD+"/setup_LYON.sh" )
elif os.path.exists(opts.GENIE+"/setup_LYON.sh") :
print( ' Using the environment of the GENIE build '+opts.GENIE+"/setup_LYON.sh" )
os.system('cp '+opts.GENIE+'/setup_LYON.sh '+opts.JOBSTD )
else :
print( ' Warning: '+opts.GENIE+'/setup_LYON.sh not found. Using '+scripts_dir+'/setup_LYON.sh, check it matches the environment used to compile GENIE' )
os.system('cp '+scripts_dir+'/setup_LYON.sh '+opts.JOBSTD )
os.system('cp '+scripts_dir+'/setup_GENIE_LYON.sh '+opts.JOBSTD )
grid_setup = opts.JOBSTD+"/setup_LYON.sh"
genie_setup = opts.JOBSTD+"/setup_GENIE_LYON.sh"
else :
genie_setup = opts.SOFTW+'/generator/builds/'+arch+'/'+version+'-setup'
grid_setup = ""
Expand All @@ -165,7 +198,7 @@
while loop_i < loop_end + 1:
# ID = 0 # vN splines
if loop_i == 0 :
command_dict.update( vN.vNSplineCommands(opts.PROBELIST,opts.vNList,opts.NuEMAXSPLINE,opts.EEMAXSPLINE,
command_dict.update( vN.vNSplineCommands(opts.PROBELIST,opts.vNList,opts.NuEMAX,opts.EEMAX,
opts.NuKnots,opts.EKnots,opts.TUNE,version,opts.GRID,opts.GROUP,
opts.CONF,opts.ARCH,opts.PROD,opts.CYCLE,opts.SOFTW,opts.GENIE,
opts.JOBSTD,grid_setup,genie_setup,opts.vNJOBLIFE,opts.JOBMEM,opts.JOBDISK,
Expand Down Expand Up @@ -207,7 +240,7 @@

loop_i += 1

if total_time > int(opts.MAINJOBLIFE) :
if opts.GRID == 'FNAL' and total_time > int(opts.MAINJOBLIFE) :
print ( "Total time of subjobs requested ("+str(total_time)+") is bigger than the job's expected time ("+str(opts.MAINJOBLIFE)+") ... Abort ..." )
exit()

Expand All @@ -220,6 +253,8 @@
grid_name = FNAL.WriteXMLFile(command_dict, loop_start, loop_end, opts.JOBSTD )

main_sub_name = FNAL.WriteMainSubmissionFile(opts.JOBSTD, opts.GENIE, opts.GROUP, grid_setup, genie_setup, grid_name, opts.MAINJOBLIFE, opts.MAINMEM, opts.MAINDISK )
elif opts.GRID == 'LYON' :
main_sub_name = LYON.WriteMainSubmissionFile(command_dict, loop_start, loop_end, opts.JOBSTD, opts.MAXPARALLEL)

if opts.SUBMIT == True:
# SUBMIT JOB
Expand Down
Loading