Skip to content

Test estimation software on other test cases #12

Open

Description

@ian-bertolacci

For #6
My current implementation of the estimation tool has only been tested on my sinusoidal domain.
I need to try it on other TCL scripts to show that it works on other scripts as well, and doesn't "cause problems"
My main concern is that test scripts often orchestrate their own data distribution, which could generally be annoying at best, and cause issues with later runs at worst.

Done when:

  • I've tested against the following cases:
    • all tests in parflow/tests/ that has a pfrun command
    • our benchmark tests clayl, ru-conus, tfg-conus
    • one or more test cases from hydroframe.org modeling v1.0 subset domain

Deliverables:

  • Report
    • Listing any tests cases that fail and why.
    • Comment on any complications that arise (important for deciding future changes to the tool and to the ReadMe).

Activity

  1. added
    needs reviewTask is complete, needs review before being closed.
    and removed
    needs reviewTask is complete, needs review before being closed.
    on Apr 15, 2020
  2. ian-bertolacci commented on Apr 20, 2020

    @ian-bertolacci
    MemberAuthor

    Running on parflow/tests, observed the following types of failures:
    Key failures:

    1. Model producing invalid predictions (< 0)
    2. non-zero exit codes
      • Some causes:
        • output file blah.pfb not created
        • Failed to open silo file blah.silo (pfload)
        • invalid expressions probably caused by not providing parameters
    3. Bad json output
      • Probably caused by not providing parameters
      • in some cases, multipl json outputs, probably from multiple runs or tests?
    4. Some actually seem to be running something?
      • ./richards_FBy.tcl (tclsh)
      • ./van-genuchten-file.tcl (tclsh)
      • ./richards_FBx.tcl (tclsh)
      • ./overland_flux.tcl (tclsh)
      • ./LW_var_dz_spinup.tcl (tclsh)
      • ./LW_var_dz.tcl (tclsh)
      • ./timing/clm-reuse/clm-reuse.tcl (tclsh)

    Thoughts:

    1. Handle multiple outputs. Report a list of one or more reports.
    2. intercept and no-op the following pfcommands; OR ALTERNATIVE ***
      • pftestFile -> just output true
      • pfload
      • pfsave...
      • pfmdiff
    3. *** Alternatively, could comment out anything not a pfset or set command...
    4. Capture runname and put in configuration
    5. add flag to set whether or not I care about exit code (--ignore-script-exit-code)
  3. ian-bertolacci commented on Apr 20, 2020

    @ian-bertolacci
    MemberAuthor

    added #22 based on thoughts from this issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions