skip_on_delete, teardown hardening, --on-failure ignore, integration tests (2.2.0)

.github/workflows/ci.yml

@@ -5,6 +5,8 @@ on:
     branches: [main]
     paths:
       - 'src/**'
+      - 'tests/**'
+      - 'ci-scripts/**'
       - 'build.rs'
       - 'Cargo.toml'
       - 'Cargo.lock'
@@ -13,6 +15,8 @@ on:
     branches: [main]
     paths:
       - 'src/**'
+      - 'tests/**'
+      - 'ci-scripts/**'
       - 'build.rs'
       - 'Cargo.toml'
       - 'Cargo.lock'

.github/workflows/integration-tests.yml

@@ -0,0 +1,60 @@
+name: Integration Tests
+
+# Live provider tests: the real stackql-deploy binary against AWS (SSM
+# Parameter Store) and GitHub (repository labels), using free resources only.
+# Intended as a required status check on pull requests to main.
+#
+# Repository secrets: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY (an IAM
+# principal allowed to manage SSM parameters under /stackql-deploy/ and to
+# call s3:DeleteBucket, which is expected to fail). The GitHub stack uses the
+# workflow's own GITHUB_TOKEN.
+
+on:
+  pull_request:
+    branches: [main]
+    paths:
+      - 'src/**'
+      - 'tests/**'
+      - 'ci-scripts/**'
+      - 'build.rs'
+      - 'Cargo.toml'
+      - 'Cargo.lock'
+      - '.github/workflows/integration-tests.yml'
+  workflow_dispatch:
+
+# One live run at a time: the stacks use a per-run name suffix, but serial
+# execution keeps provider rate limits and the label namespace predictable.
+concurrency:
+  group: integration-tests
+  cancel-in-progress: false
+
+permissions:
+  contents: read
+  issues: write
+
+jobs:
+  live:
+    name: Live provider tests
+    # Secrets are not exposed to pull requests from forks; skip rather than fail.
+    if: github.event_name == 'workflow_dispatch' || github.event.pull_request.head.repo.full_name == github.repository
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    env:
+      AWS_REGION: us-east-1
+      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+      STACKQL_GITHUB_USERNAME: github-actions
+      STACKQL_GITHUB_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
+      GITHUB_OWNER: ${{ github.repository_owner }}
+      GITHUB_REPO: ${{ github.event.repository.name }}
+      STACKQL_DEPLOY_LIVE_RUN_ID: gh${{ github.run_id }}-${{ github.run_attempt }}
+    steps:
+      - uses: actions/checkout@v6
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+      - name: Install stackql
+        uses: stackql/setup-stackql@v2
+      - name: Show stackql version
+        run: stackql --version
+      - name: Run live integration tests
+        run: bash ci-scripts/integration-test.sh

CHANGELOG.md

@@ -1,5 +1,28 @@
 # Changelog
 
+## 2.2.0 (2026-09-10)
+
+### Features
+
+- `teardown --on-failure ignore` is now honoured. A `delete` statement the provider rejects is logged at `warn` level, the resource is reported as not confirmed deleted, and the teardown continues with the next resource; the default (`error`) still aborts at the first failure. Fatal errors (network, auth, planner) abort in both modes. Every teardown now ends with a summary of resources whose delete could not be confirmed. `rollback` is not meaningful for teardown and is treated as `error`.
+- Added a live integration test suite (`tests/live.rs`, `tests/live_stacks/`) that runs the real binary against real providers using free resources only: AWS SSM Parameter Store parameters (Cloud Control and native API) and GitHub repository labels. It covers create with `RETURNING *`, `return_vals`, `callback`, `statecheck`, `PatchDocument` updates, `createorupdate`, `query`/`command`/`script` resources, conditions, `file()`, `merge`, per-environment values, protected masking, stack exports, `test` pass and fail, `teardown` with `skip_on_delete`, `--on-failure`, and idempotent re-runs. The tests are `#[ignore]`d for `cargo test`; `ci-scripts/integration-test.sh` runs them, and the new `Integration Tests` workflow runs them on pull requests to `main` as a merge gate (replacing the `Test Demo` placeholder workflow). See `tests/README.md`.
+- Added `skip_on_delete: true` for resources ([#56](https://github.com/stackql/stackql-deploy-rs/issues/56)). A `query` resource with this flag is not executed during `teardown` and its declared exports are set to `<unknown>`; a `resource` or `multi` resource with this flag still has its exports collected (a downstream `delete` may need them) but its `delete` query is not executed, so the resource is retained. The flag has no effect on `build` or `test`.
+- Added an integration test suite (`tests/`) that drives the `build` and `teardown` flows against an in-process mock stackql server speaking the PostgreSQL wire protocol, with fixture stacks under `tests/fixtures/`. Tests assert on the exact statements sent and need no stackql binary, provider registry, network access, or cloud credentials. `ci-scripts/test.sh` now runs `cargo test`, and the CI workflow triggers on `tests/**` and `ci-scripts/**` changes. To support this, the crate now has a library target alongside the binary; it is not a public API.
+
+### Fixes
+
+- Teardown no longer executes queries that interpolate the `<unknown>` export placeholder. When an upstream resource had already been deleted (for example by an earlier, partially successful teardown), its exports were set to `<unknown>` and then substituted into downstream `exists`, `exports`, `delete`, and inline `sql` queries. For a Databricks workspace this produced `https://<unknown>.cloud.databricks.com/...`, a fatal `dial tcp ... no such host` error, and an aborted teardown that could never complete. Such queries are now skipped with a log line naming the resource and anchor, and the dependent resource's exports are themselves marked `<unknown>` so the skip propagates consistently.
+- Teardown now tolerates a non-fatal provider error on an `exports` query (fatal network and auth errors still abort). The resource's exports are marked `<unknown>` with a warning and the teardown continues; previously the error aborted the run.
+- Teardown no longer aborts on stacks that contain a `script` resource. Export collection tried to load a `.iql` file for the script and exited when it was not found; script exports are now marked `<unknown>` (scripts are never executed on teardown).
+- Inline `sql` on `query` resources is rendered tolerantly during teardown: a missing template variable skips the query instead of exiting the process.
+- The `test` command now evaluates `if` conditions (it previously processed every resource regardless) and runs `script` resources the same way `build` does (it previously exited with `unknown resource type: script`).
+- A `--dry-run` teardown now renders and logs each `delete` statement. Previously the dry-run exists check reported every resource as not found, so the run only showed "skipping delete".
+- The `<evaluated>` and `<unknown>` export placeholders are no longer registered for log redaction when the export is `protected`. Previously a protected export in a dry run registered `<evaluated>` as a secret, which then masked every other placeholder in the run as `********`.
+- Command failures that are ignored (`multi` resources, and now `--on-failure ignore`) are logged at `warn` level instead of `debug`.
+- A `callback:delete` (or generic `callback`) anchor no longer aborts a `--dry-run` teardown, or a teardown whose delete returned no `RETURNING *` row. Callbacks poll the handle returned by `RETURNING *`, so they are now skipped with a log line when there is nothing to poll, matching `build`.
+- The `postdelete_retries` and `postdelete_retry_delay` options on the `exists` anchor are now honoured, as documented (defaults 10 and 5). They were parsed but never used: the post-delete check ran once immediately and once more after the `delete` anchor's `retry_delay`, which defaults to 0, so an asynchronous delete (Cloud Control, most SaaS APIs) could only be confirmed by luck of timing. After each delete attempt the exists query now polls until the resource is gone, up to `postdelete_retries` times, `postdelete_retry_delay` seconds apart; only then is the delete re-issued, up to the `delete` anchor's `retries`.
+- The `callback:delete` anchor now runs before the post-delete check rather than after it, so a provider's asynchronous delete is polled to completion before the resource is checked for absence.
+
 ## 2.1.1 (2026-08-24)
 
 ### Fixes

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "stackql-deploy"
-version = "2.1.1"
+version = "2.2.0"
 edition = "2021"
 rust-version = "1.75"
 description = "Infrastructure-as-code framework for declarative cloud resource management using StackQL"
@@ -12,11 +12,23 @@ keywords = ["stackql", "infrastructure", "iac", "cloud", "devops"]
 categories = ["command-line-utilities", "development-tools"]
 readme = "README.md"
 
-# docs.rs: this is a binary-only crate — no public library API to document.
+# docs.rs: the library target exists only so the integration tests in
+# tests/ can drive the command runner in-process; it is not a public API.
 # The README (above) is what crates.io renders on the package page.
 [package.metadata.docs.rs]
 no-default-features = true
 
+# Module-level doc comments contain illustrative snippets that reference
+# crate-internal paths; they are not compilable examples, so doctests are off.
+[lib]
+name = "stackql_deploy"
+path = "src/lib.rs"
+doctest = false
+
+[[bin]]
+name = "stackql-deploy"
+path = "src/main.rs"
+
 [dependencies]
 clap = { version = "4.3", features = ["derive"] }
 colored = "2.0"

ci-scripts/integration-test.sh

@@ -0,0 +1,37 @@
+#!/bin/bash
+set -euo pipefail
+
+# Runs the live integration tests: the real stackql-deploy binary against
+# real providers, using free resources only (AWS SSM Parameter Store standard
+# parameters and GitHub repository labels). See tests/README.md.
+#
+# Required in the environment:
+#   AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY   (AWS stacks)
+#   STACKQL_GITHUB_USERNAME, STACKQL_GITHUB_PASSWORD  (GitHub stack; the
+#     password is a token with issues:write on GITHUB_OWNER/GITHUB_REPO)
+# Optional:
+#   AWS_REGION (default us-east-1), GITHUB_OWNER, GITHUB_REPO,
+#   STACKQL_DEPLOY_LIVE_RUN_ID (unique suffix for resource names; a local
+#     timestamp is used when unset), STACKQL_DEPLOY_LIVE_FILTER (run only
+#     tests whose name contains this string).
+#
+# The stackql binary must be on PATH (or in the working directory).
+
+echo "==============================================="
+echo "  Running live integration tests"
+echo "==============================================="
+
+if ! command -v stackql >/dev/null 2>&1 && [ ! -x ./stackql ]; then
+  echo "error: the stackql binary was not found on PATH" >&2
+  echo "       install it (https://stackql.io/downloads) or run 'stackql-deploy upgrade'" >&2
+  exit 1
+fi
+
+export STACKQL_DEPLOY_LIVE_RUN_ID="${STACKQL_DEPLOY_LIVE_RUN_ID:-local-$(date +%s)}"
+echo "run id: ${STACKQL_DEPLOY_LIVE_RUN_ID}"
+
+# Each live stack starts its own stackql server on a dedicated port, but the
+# provider cache under ~/.stackql is shared, so run the stacks one at a time.
+cargo test --test live -- --ignored --test-threads=1 --nocapture "${STACKQL_DEPLOY_LIVE_FILTER:-}"
+
+echo -e "\n✅ Live integration tests passed"

ci-scripts/test.sh

@@ -6,16 +6,14 @@ echo "==============================================="
 echo "  Running Tests for stackql-deploy"
 echo "==============================================="
 
-# Run unit tests
+# Run unit tests (in-module #[cfg(test)] tests in src/)
 echo "Running unit tests..."
-# cargo test --lib
+cargo test --lib
 
-# Run integration tests if they exist
+# Run integration tests (tests/*.rs). These drive the build and teardown
+# flows against an in-process mock stackql server - no stackql binary,
+# provider registry, network access, or cloud credentials required.
 echo -e "\nRunning integration tests..."
-# cargo test --test '*'
+cargo test --test '*'
 
-# Run doc tests
-echo -e "\nRunning documentation tests..."
-# cargo test --doc
-
-echo -e "\n✅ All tests passed successfully!"
+echo -e "\n✅ All tests passed successfully!"

docs/flows.md

@@ -194,7 +194,21 @@ graph LR
     H -->|still there| I
 ```
 
-During teardown export collection, missing exports are set to `<unknown>` rather than failing — the stack may be partially deployed.
+During teardown export collection, exports that cannot be collected are set to `<unknown>` rather than failing - the stack may be partially deployed. An export ends up `<unknown>` when its query returns no rows, returns a non-fatal provider error, cannot be rendered, belongs to a `script` resource (never executed on teardown), or belongs to a resource with `skip_on_delete: true`.
+
+Any teardown query (`exists`, `statecheck`, `exports`, `delete`, or inline `sql`) whose rendered text contains `<unknown>` is skipped rather than executed. Running it would at best match nothing and at worst put the placeholder into a hostname or identifier (for example `https://<unknown>.cloud.databricks.com/...`), which fails with a fatal `dial tcp` error and aborts the whole teardown. The resource is logged as skipped and processing continues with the next one.
+
+After each `delete` (and its `callback:delete`, if any) the `exists` query is polled until the resource is gone, up to `postdelete_retries` times `postdelete_retry_delay` seconds apart (options on the `exists` anchor, defaults 10 and 5). Only then is the delete re-issued, up to the `delete` anchor's `retries`.
+
+A `delete` statement the provider rejects aborts the teardown by default (`--on-failure error`). With `--on-failure ignore` the failure is logged, the resource is reported as not confirmed deleted, and the next resource is processed; fatal network, auth and planner errors abort in both modes. Every teardown ends with a summary of resources whose delete could not be confirmed.
+
+`skip_on_delete: true` opts a resource out of teardown explicitly:
+
+| Type | Effect on teardown |
+|------|--------------------|
+| `query` | Query not executed; declared exports set to `<unknown>` |
+| `resource` / `multi` | Exports still collected (downstream deletes may need them); `delete` not executed, resource retained |
+| `command` / `script` | No change (never executed on teardown) |
 
 ---