fix(test): make external model fixtures hermetic - #154
Conversation
|
Hi @mttrbrts, the fix follows the Concerto-style cached external model pattern. What changed:
I also verified the full suite locally after the change: npm test -- --runInBand --coverage=false |
Signed-off-by: Rishabh Jain <rishabhj2005@email.com>
61df9ad to
f675fe7
Compare
|
This PR is stale because it has been open with no activity. Remove the stale label or comment to keep it active. Only items with maintainer engagement are auto-closed. |
|
@github unstale |
|
Hey @Rishabh060105, thanks for the PR! One suggestion before merge: take a look at how cicero-template-library handles this same problem in test/render.test.mjs — it might be a cleaner pattern than fetch-mocking here. Instead of intercepting fetch and vendoring copies of the external model files, it forces offline: true at the loader boundary: Template.fromDirectory(path, { offline: true }). That routes cicero-core to mm.validateModelFiles() (a local-only check) instead of mm.updateExternalModels() (the network call), so no HTTP request is ever made in the first place. It goes a step further and globally monkeypatches Template.prototype.validate to always pass offline: true, which protects against any internal validate() call elsewhere in the chain, not just the one call site you know about. Might be worth considering here instead of (or alongside) the mockExternalModelFetches approach , curious what you think. |
4104bd3 to
f675fe7
Compare
Signed-off-by: Rishabh Jain <rishabhj2005@email.com>
Signed-off-by: Rishabh Jain <rishabhj2005@email.com>
|
Hi @devanshi00, This removes the fetch interception entirely and ensures archive models are validated locally through For tests that create a Thank you for pointing it out! |
| async function fromDirectoryOffline(templateDir: string): Promise<Template> { | ||
| const originalValidate = Template.prototype.validate; | ||
| Template.prototype.validate = function validateOffline(options = {}) { | ||
| return originalValidate.call(this, {...options, offline: true}); | ||
| }; | ||
|
|
||
| return originalFetch(input, init); | ||
| }); | ||
| try { | ||
| return await Template.fromDirectory(templateDir, {offline: true}); | ||
| } finally { | ||
| Template.prototype.validate = originalValidate; | ||
| } |
There was a problem hiding this comment.
I worry about the effect of monkey-patching in scenarios where tests are executed in parallel.
Instead, do we understand why the Template.fromDirectory(templateDir, {offline: true}) doesn't work as expected?
There was a problem hiding this comment.
Yes, Template.fromDirectory(..., { offline: true }) uses offline mode initially, but cicero-core later calls template.validate() without forwarding the option. validate() then falls back to updateExternalModels(), causing the network request.
Basically cicero-core is failing to pass offline:true to the template.validate() call.We probably need to fix cicero-core so the loader forwards offline into its final validation call.
About the monkeypatch: it works well for sequential tests but when tests are executed in parallel it modifies the global state as well.
There was a problem hiding this comment.
Are you referring to this line? https://github.com/accordproject/template-archive/blob/main/packages/cicero-core/src/templateloader.ts#L113
Yes, we should fix that, rather than hack it here.
There was a problem hiding this comment.
Yes, that is the line. I also found the same issue in the final validation inside fromDirectory. I can open a template-archive PR to forward offline through both loader paths and add regression coverage.
Closes #153
Summary
This change makes the test suite hermetic for the remaining external-model paths.
It replaces live model downloads in the affected tests with cached
@models...fixtures, following the same naming pattern already used in Concerto tests and existing Accord Project archive fixtures.Changes
test/modelsModelManagerTemplateMarkInterpreter.test.tsto stop callingupdateExternalModels()at runtimeTemplateArchiveProcessor.test.tsto resolve external model fetches from local cached fixturesTest
npm test -- --runInBand --coverage=falseResults: