docs: explain snapshot artifacts and deployment diffs - #405
Conversation
Add a hand-authored guide alongside the generated API reference that explains how deployment_diff in the get snapshot response is computed: what "previous" means, why there is one answer per flow, when it is null, and the caveats on previous_running, compliance state and diff_url. Cross-link from the glossary, how_kosli_works, and the querying tutorial. Refs #403
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
|
Claude encountered an error after 5m 18s —— View job Doc review
4 findings, all Improvement. No Critical. Details are inline; summarized here because each one is a cross-file claim.
On placement (raised since a new page was added): Reference ▸ API Reference is the right call — the page is explanation-shaped but its subject is a response field, and the reader arrives from Verified, not re-reported: the nav entry exists (core rule 2); all six internal targets in the new page resolve to real files; every link is root-relative; the four-flow table matches the generated sample in Couldn't run locally (sandbox blocks network and these invocations): Verdict: 4 Improvements, 0 Critical. Mergeable; findings 1 and 3 are the two that can mislead a reader, and both are small edits. |
| curl -s "$BASE/-1" | jq -r '.artifacts[] | "\(.name) -> \(.deployment_diff.diff_url // "no diff")"' | ||
| ``` | ||
|
|
||
| `-1` is the latest snapshot. The other snapshot expressions (`#N`, `~N`, `@{...}`) are listed on the [get snapshot](/api-reference/snapshots/get-snapshot) page. For a private organization, authenticate with your [API token](/administration/authentication/api_authentication_methods). |
There was a problem hiding this comment.
Improvement — the other snapshot expressions don't work in a URL the way this reads.
#N, ~N and @{...} are the CLI's ENVIRONMENT-NAME-OR-EXPRESSION grammar, where they attach to the environment name (prod#42, prod~5, prod@{2.hours.ago} — see client_reference/kosli_get_snapshot.md:13-28). Here the only example is a URL path segment ($BASE/-1), so a reader naturally tries $BASE/#42 — and # starts a fragment, so curl never sends it and the request silently returns the latest snapshot instead of snapshot 42. @{...} has the same encoding problem with {/}.
Recommend showing the API form explicitly rather than deferring to the CLI grammar — e.g. a plain numbered id ($BASE/5352) alongside -1, and if #N really is accepted on this path, say it must be percent-encoded as %23N.
| The public `cyber-dojo` organization shows this. Its `web` image is reported into four flows, and the latest `aws-prod` snapshot returns four different answers for the same running image: | ||
|
|
||
| | Flow | Previous fingerprint | `diff_url` compares | | ||
| |---|---|---| | ||
| | `web-ci` | `36ad0020` | `cyber-dojo/web` `cbe481c...236898f` | | ||
| | `production-promotion` | `36ad0020` | `cyber-dojo/aws-prod-co-promotion` `7494758...7494758` | | ||
| | `snyk-aws-beta-per-artifact` | `36ad0020` | `cyber-dojo/snyk-scanning` `ed3c81d...30111f1` | | ||
| | `snyk-aws-prod-per-artifact` | `29c69c2f` | `cyber-dojo/snyk-scanning` `00c4797...30111f1` | |
There was a problem hiding this comment.
Improvement — the example is pinned to a snapshot the page never names.
The four rows are verifiable today: they match the generated sample in client_reference/kosli_get_snapshot.md:1668-1810 exactly (36ad0020 for web-ci, production-promotion and snyk-aws-beta-per-artifact; 29c69c2f for snyk-aws-prod-per-artifact; cbe481c...236898f, 7494758...7494758, ed3c81d...30111f1, 00c4797...30111f1).
The problem is the framing: "the latest aws-prod snapshot", plus $BASE/-1 in See it yourself (line 100). cyber-dojo deploys continuously, so the next web deploy makes -1 disagree with this table, and a reader who follows the instructions and gets four different fingerprints concludes the page is wrong rather than stale. The PR description says these came from snapshot 5352 — say so on the page, and use that id in the walkthrough so the numbers keep reproducing.
Suggested: "the aws-prod snapshot 5352 (the latest at the time of writing) returns four different answers", and $BASE/5352 in the first curl, keeping -1 for the "show me current state" example.
|
|
||
| - **`previous_running` is about this snapshot, not history.** `false` means the predecessor is gone, the normal steady state after a deploy completes. `true` means both versions are present in this snapshot at once: a rolling deploy caught mid-flight, a canary, or a partially drained service. | ||
| - **`previous_artifact_compliance_state` is evaluated when you make the request**, not frozen at snapshot time. If someone attests evidence against the older artifact tomorrow, this field can change for a snapshot taken today. Compliance is a live judgment about an artifact, not a historical record of the deploy. | ||
| - **`diff_url` can be `""`.** It is built by matching the commit URL against GitHub, GitLab, Bitbucket, and Azure DevOps. Any other git host produces an empty string rather than a broken link. `previous_git_commit` and the artifact's own `git_commit` are still there, so you can build the link yourself. |
There was a problem hiding this comment.
Improvement — previous_git_commit_url is missing, and it's the field this caveat should point at.
The response carries a ninth sub-field the page never mentions: previous_git_commit_url (54 occurrences in the generated sample, e.g. client_reference/kosli_get_snapshot.md:1675). It is a ready-made link to the predecessor's commit.
Two consequences:
- This bullet tells the reader to fall back to
previous_git_commitand the artifact's owngit_commitand "build the link yourself" — when for the previous commit specifically they already haveprevious_git_commit_url. The genuinely missing piece whendiff_urlis""is only the compare URL. - Line 84's nullability enumeration walks the sub-fields and skips this one, so the page reads as a complete field inventory while being one short.
Recommend naming previous_git_commit_url in the line 84 sentence, and in this bullet saying what is actually absent: the compare link, which you can construct from previous_git_commit and git_commit (or note whether previous_git_commit_url is also empty for an unrecognized host — worth checking, since it's reported rather than pattern-matched).
| "groups": [ | ||
| { | ||
| "group": "Guides", | ||
| "pages": [ | ||
| "api-reference/snapshot_artifacts_and_deployment_diffs" | ||
| ] | ||
| }, | ||
| { | ||
| "group": "Endpoints", | ||
| "openapi": "https://app.kosli.com/api/v2/openapi.json" | ||
| } | ||
| ] |
There was a problem hiding this comment.
Improvement (verify on the preview) — this changes the shape of the API Reference item that ~14 existing links depend on.
openapi moved from being the item's direct value to sitting inside a groups entry. The auto-generated endpoint URLs (/api-reference/<tag>/<operation>) should be unaffected, since Mintlify derives page paths from the item, not the group label — but nothing in CI will tell you if that assumption is wrong: mint broken-links does not resolve auto-generated OpenAPI pages, and the link-rot check is unreliable per CLAUDE.md.
If the prefix did change, these break silently:
understand_kosli/what_is_kosli.md:12→/api-reference/actions/list-actionsgetting_started/enforce_policies.md:127,143→/api-reference/asserts/assert-artifacttutorials/rotating_api_keys.mdx:99,127,148-150andadministration/authentication/api_key_rotation.md:23-25→/api-reference/service-accounts/...user/default_organization.md:25→/api-reference/user/set-default-organizationchangelog/index.mdx:1068config/redirects.json:84— whose destination is/api-reference/actions/list-actions, so a broken destination turns a working redirect into a dead end- this PR's own page, which links
/api-reference/snapshots/get-snapshotfour times
The PR notes "a generated endpoint page renders (200)". Worth confirming specifically that one of the pre-existing URLs above still resolves on the Mintlify preview, not just that some endpoint page exists.
Refs #403
What
Adds a hand-authored guide,
api-reference/snapshot_artifacts_and_deployment_diffs.md, explaining thedeployment_diffobject in the get snapshot response:kosli diff snapshotsvsdeployment_diff) and when to use whichtemplate_reference_name, most recent start before this one — and why it is not the previous snapshot and not matched on namewebartifact in publiccyber-dojo/aws-prod; the top-level field is a copy offlows[0]null, and which sub-fields can benullor""previous_runningis about this snapshot, compliance state is evaluated live,diff_urlis empty for unrecognized git hostsCross-links added from
understand_kosli/glossary.md(new "Deployment Diff" term),understand_kosli/how_kosli_works.md(Monitoring runtime), andtutorials/querying_kosli.md(Compare snapshots).Where
Reference ▸ API Reference ▸ Guides, alongside the OpenAPI-generated Endpoints group. This follows the precedent of
client_reference/overview.mdandoutput_and_verbosity.md, which are hand-authored pages inside the generated CLI Reference. It was not placed under Understand Kosli: every sibling there (what_is_kosli,risks,controls,how_kosli_works) is one abstraction level up, and a page about per-flow predecessor resolution would read as an outlier. Readers of this page arrive fromget-snapshot, so it sits one click away.scripts/audit_navigation.pyflagsGuidesas a single-child group. That is advisory only (--checkexits 0), and the group is structurally required: Mintlify needs a group container to mix a hand page with anopenapigroup.Relationship to #403
kosli-dev/server#6852 is merged and adds descriptions to all nine
deployment_difffields in the OpenAPI schema. The API Reference pullshttps://app.kosli.com/api/v2/openapi.jsonlive, so those descriptions appear onget-snapshotautomatically once the server deploys — no docs change needed for that half. This PR covers the narrative half (what "previous" means, when it is null). Close #403 once the deploy lands and the field descriptions render.Verification
mint dev: the new page and a generated endpoint page both render (200); the sidebar shows bothGuidesandEndpoints.mint broken-links: one failure,tutorials/working_with_controls.mdx→/getting_started/service-accounts, pre-existing onmainand unrelated to this change.python3 -m pytest tests/: 44 passed (navigation integrity included).diff_urls,flows[0]equality) verified against the livecyber-dojo/aws-prodsnapshot 5352.