skip_on_delete, teardown hardening, --on-failure ignore, integration tests (2.2.0) - #57
Merged
Merged
Conversation
…tegration tests, bump to 2.2.0
…ck, fix CI - The exists anchor's postdelete_retries / postdelete_retry_delay were parsed but never used; the post-delete check ran twice within zero seconds, so an asynchronous delete (Cloud Control) was only confirmed by luck of timing. The check now polls per those options after each delete attempt. - callback:delete now runs before the post-delete check, not after it. - delete_and_confirm split into execute_delete + confirm_deleted. - Mock provider: RETURNING rows on DML, asynchronous deletes, canned answers; tests for callback ordering, postdelete polling and unconfirmed deletes. - test_command: drop a reference only compiled on unix (CI compile error). - Give the on-failure ghosts a short postdelete budget so Cloud Control has time to record the failure the troubleshoot anchor reports. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #56.
Tearing down a partially deleted stack could not complete: once an upstream resource (the Databricks workspace) was gone, its exports were set to the
<unknown>placeholder and then interpolated into downstream queries, producing requests tohttps://<unknown>.cloud.databricks.com/..., a fataldial tcperror, and an aborted run on every retry. This PR fixes the root cause, adds theskip\_on\_deleteproperty from the issue, wires--on-failure ignorefor teardown, and adds two layers of integration tests, one of which runs against real providers as a pre-merge gate. Version bumped to 2.2.0.## Features
-
skip\_on\_delete: trueon a resource. Aqueryis not executed during teardown and its exports become<unknown>; aresourceormultistill has its exports collected (downstream deletes may need them) but its own delete is skipped, so it is retained. No effect onbuildortest. Documented on the manifest reference page.-
teardown --on-failure ignore. A delete the provider rejects is logged atwarnlevel, the resource is reported as not confirmed deleted, and the run continues. The default (error) still aborts at the first failure. Fatal errors (network, auth, planner) abort in both modes, and every teardown now ends with a summary of resources whose delete could not be confirmed.rollbackis treated aserrorfor teardown; the build-side rollback aspiration is untouched.- Integration tests, two layers:
- Mock layer (
tests/build.rs,tests/teardown.rs,tests/test\_command.rs). The crate gains a library target so tests driverun\_build,run\_testandrun\_teardownin process against an in-process mock of the stackql server that speaks the PostgreSQL simple-query protocol and records every statement. No binary, registry, network or credentials needed. Runs on everycargo test;ci-scripts/test.shnow actually runs the tests.- Live layer (
tests/live.rs,tests/live\_stacks/). The real binary against AWS SSM Parameter Store (Cloud Control and native API) and GitHub repository labels, all free resources with a per-run name suffix and cleanup on failure. Covers create, update and delete withRETURNING \*,return\_vals,callback:create/callback:update/callback:delete,troubleshoot,statecheck,PatchDocumentupdates,createorupdate,query/command/scriptresources,if,file(),merge, per-environment values, protected masking under--show-queries, stack exports,testpass and fail, dry runs, teardown withskip\_on\_delete, both--on-failuremodes, and idempotent re-runs.#\[ignore]d for plaincargo test; run byci-scripts/integration-test.shand the newIntegration Testsworkflow, which replaces theTest Demoplaceholder. Seetests/README.md.## Fixes
- Teardown no longer executes any
exists,statecheck,exports,deleteor inlinesqlquery whose rendered text contains<unknown>. The resource is skipped with a log line naming the anchor, and its own exports are marked<unknown>so the skip propagates.- Teardown tolerates a non-fatal provider error on an
exportsquery (marks the exports unknown and continues) and no longer aborts on stacks containing ascriptresource.- Inline
sqlis rendered tolerantly during teardown; a missing variable skips the query instead of exiting.-
testnow evaluatesifconditions and runsscriptresources the same waybuilddoes; it previously processed every resource and exited on scripts.- A
--dry-runteardown renders and logs each delete statement; it previously reported every resource as not found.- A
callback:deleteanchor no longer aborts a dry-run teardown, or a teardown whose delete returned noRETURNING \*row; the callback is skipped with a log line, matchingbuild.- The
<evaluated>and<unknown>placeholders are no longer registered for log redaction when an export isprotected; a protected export in a dry run used to mask every other placeholder as\*\*\*\*\*\*\*\*.- Ignored command failures (
multiresources,--on-failure ignore) are logged atwarninstead ofdebug.## Testing
-
cargo fmt --check,cargo clippy --all-targets -D warnings, andcargo testpass on stable 1.97 and on 1.84 (MSRV is 1.75): 103 unit tests and 16 mock integration tests, 3 live tests ignored by default.- The AWS live tests pass end to end against a real account (about four minutes), including the three Cloud Control callbacks and the
troubleshoot:deletediagnostics for an asynchronous Cloud Control failure. The account is left clean.- The GitHub label stack has only been verified up to the request reaching GitHub: the local token is read-only for labels (403), so the first full run is this PR's
Integration Testsjob, which usesGITHUB\_TOKENwithissues: write.- The regression test for the
<unknown>guard was confirmed to fail with the guard disabled.## Repository setup needed
- Repository secrets
AWS\_ACCESS\_KEY\_IDandAWS\_SECRET\_ACCESS\_KEYfor an IAM principal that can manage SSM parameters under/stackql-deploy/(cloudcontrol:\*onAWS::SSM::Parameter,ssm:PutParameter,ssm:GetParameters,ssm:DeleteParameter,ssm:AddTagsToResource,ssm:ListTagsForResource). Thes3:DeleteBucketcall in the on-failure stack is expected to fail and needs no permission.- After merge, add the
Live provider testsjob to the required status checks onmainso it gates future merges. Pull requests from forks skip the job because secrets are unavailable to them.## Notes
- The live layer runs with
target/live/as the binary's working directory for the live suite, so the stackql server's provider cache is pulled once and nothing is written into the source tree.- Two stackql engine behaviours observed while building the live stacks, both worked around in the manifests:
LIKEis not supported against the Cloud Control\_list\_onlyviews (INSTRis used instead), and Cloud Control does not returnDescriptionforAWS::SSM::Parameter, so the statecheck compares value and tags.-
docs/release-process.mdsays to bump the version after merging; this PR carries the bump in the feature change as 2.1.0 and 2.1.1 did. Tagv2.2.0on the merge commit to release.🤖 Generated with [Claude Code](https://claude.com/claude-code)