Skip to content

skip_on_delete, teardown hardening, --on-failure ignore, integration tests (2.2.0) - #57

Merged
jeffreyaven merged 2 commits into
mainfrom
feature/skip-on-delete-teardown-hardening
Sep 10, 2026
Merged

skip_on_delete, teardown hardening, --on-failure ignore, integration tests (2.2.0)#57
jeffreyaven merged 2 commits into
mainfrom
feature/skip-on-delete-teardown-hardening

Conversation

@jeffreyaven

Copy link
Copy Markdown
Member

Closes #56.

Tearing down a partially deleted stack could not complete: once an upstream resource (the Databricks workspace) was gone, its exports were set to the <unknown> placeholder and then interpolated into downstream queries, producing requests to https://<unknown>.cloud.databricks.com/..., a fatal dial tcp error, and an aborted run on every retry. This PR fixes the root cause, adds the skip\_on\_delete property from the issue, wires --on-failure ignore for teardown, and adds two layers of integration tests, one of which runs against real providers as a pre-merge gate. Version bumped to 2.2.0.

## Features

- skip\_on\_delete: true on a resource. A query is not executed during teardown and its exports become <unknown>; a resource or multi still has its exports collected (downstream deletes may need them) but its own delete is skipped, so it is retained. No effect on build or test. Documented on the manifest reference page.

- teardown --on-failure ignore. A delete the provider rejects is logged at warn level, the resource is reported as not confirmed deleted, and the run continues. The default (error) still aborts at the first failure. Fatal errors (network, auth, planner) abort in both modes, and every teardown now ends with a summary of resources whose delete could not be confirmed. rollback is treated as error for teardown; the build-side rollback aspiration is untouched.

- Integration tests, two layers:

- Mock layer (tests/build.rs, tests/teardown.rs, tests/test\_command.rs). The crate gains a library target so tests drive run\_build, run\_test and run\_teardown in process against an in-process mock of the stackql server that speaks the PostgreSQL simple-query protocol and records every statement. No binary, registry, network or credentials needed. Runs on every cargo test; ci-scripts/test.sh now actually runs the tests.

- Live layer (tests/live.rs, tests/live\_stacks/). The real binary against AWS SSM Parameter Store (Cloud Control and native API) and GitHub repository labels, all free resources with a per-run name suffix and cleanup on failure. Covers create, update and delete with RETURNING \*, return\_vals, callback:create / callback:update / callback:delete, troubleshoot, statecheck, PatchDocument updates, createorupdate, query / command / script resources, if, file(), merge, per-environment values, protected masking under --show-queries, stack exports, test pass and fail, dry runs, teardown with skip\_on\_delete, both --on-failure modes, and idempotent re-runs. #\[ignore]d for plain cargo test; run by ci-scripts/integration-test.sh and the new Integration Tests workflow, which replaces the Test Demo placeholder. See tests/README.md.

## Fixes

- Teardown no longer executes any exists, statecheck, exports, delete or inline sql query whose rendered text contains <unknown>. The resource is skipped with a log line naming the anchor, and its own exports are marked <unknown> so the skip propagates.

- Teardown tolerates a non-fatal provider error on an exports query (marks the exports unknown and continues) and no longer aborts on stacks containing a script resource.

- Inline sql is rendered tolerantly during teardown; a missing variable skips the query instead of exiting.

- test now evaluates if conditions and runs script resources the same way build does; it previously processed every resource and exited on scripts.

- A --dry-run teardown renders and logs each delete statement; it previously reported every resource as not found.

- A callback:delete anchor no longer aborts a dry-run teardown, or a teardown whose delete returned no RETURNING \* row; the callback is skipped with a log line, matching build.

- The <evaluated> and <unknown> placeholders are no longer registered for log redaction when an export is protected; a protected export in a dry run used to mask every other placeholder as \*\*\*\*\*\*\*\*.

- Ignored command failures (multi resources, --on-failure ignore) are logged at warn instead of debug.

## Testing

- cargo fmt --check, cargo clippy --all-targets -D warnings, and cargo test pass on stable 1.97 and on 1.84 (MSRV is 1.75): 103 unit tests and 16 mock integration tests, 3 live tests ignored by default.

- The AWS live tests pass end to end against a real account (about four minutes), including the three Cloud Control callbacks and the troubleshoot:delete diagnostics for an asynchronous Cloud Control failure. The account is left clean.

- The GitHub label stack has only been verified up to the request reaching GitHub: the local token is read-only for labels (403), so the first full run is this PR's Integration Tests job, which uses GITHUB\_TOKEN with issues: write.

- The regression test for the <unknown> guard was confirmed to fail with the guard disabled.

## Repository setup needed

- Repository secrets AWS\_ACCESS\_KEY\_ID and AWS\_SECRET\_ACCESS\_KEY for an IAM principal that can manage SSM parameters under /stackql-deploy/ (cloudcontrol:\* on AWS::SSM::Parameter, ssm:PutParameter, ssm:GetParameters, ssm:DeleteParameter, ssm:AddTagsToResource, ssm:ListTagsForResource). The s3:DeleteBucket call in the on-failure stack is expected to fail and needs no permission.

- After merge, add the Live provider tests job to the required status checks on main so it gates future merges. Pull requests from forks skip the job because secrets are unavailable to them.

## Notes

- The live layer runs with target/live/ as the binary's working directory for the live suite, so the stackql server's provider cache is pulled once and nothing is written into the source tree.

- Two stackql engine behaviours observed while building the live stacks, both worked around in the manifests: LIKE is not supported against the Cloud Control \_list\_only views (INSTR is used instead), and Cloud Control does not return Description for AWS::SSM::Parameter, so the statecheck compares value and tags.

- docs/release-process.md says to bump the version after merging; this PR carries the bump in the feature change as 2.1.0 and 2.1.1 did. Tag v2.2.0 on the merge commit to release.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

jeffreyaven and others added 2 commits September 10, 2026 11:57
…ck, fix CI

- The exists anchor's postdelete_retries / postdelete_retry_delay were parsed
  but never used; the post-delete check ran twice within zero seconds, so an
  asynchronous delete (Cloud Control) was only confirmed by luck of timing.
  The check now polls per those options after each delete attempt.
- callback:delete now runs before the post-delete check, not after it.
- delete_and_confirm split into execute_delete + confirm_deleted.
- Mock provider: RETURNING rows on DML, asynchronous deletes, canned answers;
  tests for callback ordering, postdelete polling and unconfirmed deletes.
- test_command: drop a reference only compiled on unix (CI compile error).
- Give the on-failure ghosts a short postdelete budget so Cloud Control has
  time to record the failure the troubleshoot anchor reports.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@jeffreyaven
jeffreyaven merged commit e9e410e into main Sep 10, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEAURE] add skip_on_delete property for query resources

1 participant