Repository navigation
feat(gatekeeper-constraints): daily workload policy report (MAPCO-11867) - #187
Merged
Merged
Conversation
files/daily-report/report.py (stdlib only, python3 -I) reads the live workload ConstraintTemplates, Constraints, Namespaces and the workloads those Constraints match, writes a projection of them (kind/apiVersion set, one object per file, every Constraint once and as warn), runs the pinned gator on it and routes each namespace to a team. It posts one summary per team (counts per namespace and rule, capped names, dashboard link) and pushes the run to Loki. Every failure goes to the default team and exits 1. A run is "skipped" on an API error, a 410 on a list continue, a missing Constraint kind, a gator download or sha256 failure, or a gator error/warning/timeout. It is "partial" on a team whose webhook is missing or whose post failed, on an audit-count mismatch, or on a failed Loki push. dryRun suppresses Slack only. MAPCO-11867. Tests (wired into make test): - tests/daily-report/test_report.py: unittests on raw API fixtures. - tests/daily-report/parity.sh: gator on tests/objects vs gator on report.py's projection of the same objects; it also fails on stale fixtures. Claude-Session: https://claude.ai/code/session_019uxkhSxKXFuSBsYBAmh8rB
dailyReport (off by default) renders, in dailyReport.namespace: - a ConfigMap with report.py and teams.json (namespaceTeams in matching order, globs as regexes); - a ServiceAccount; - a CronJob (Sun-Thu 07:30 UTC, Forbid, backoffLimit 0, hardened Python Pod with CPU at 1:4, Pod labels and annotations that satisfy this chart's rules, one optional secretKeyRef per team webhook). It also renders a get/list-only ClusterRole and binding. In the release namespace it renders two Grafana rules for deployAlerts.defaultTeam: - gk-daily-report-missing: Loki run count plus a Prometheus time-of-day gate, so it only checks while the stack is up after the run; - gk-daily-report-failed: a run with status partial or skipped. gator v3.22.0 linux-amd64 is pinned by sha256 (checked against the release's sha256sums.txt). Rendering fails on missing or malformed required values. MAPCO-11867. Tests (wired into make test): - tests/daily-report.sh: render assertions. - tests/daily-report/job-pod.sh: gator on the job's own Pod and Pod template. - tests/render-failures/daily-report-*: one case per validation. Claude-Session: https://claude.ai/code/session_019uxkhSxKXFuSBsYBAmh8rB
…s a webhook - Any exception in a run (not only ReportError) is now a "skipped" run: infra is told and the run line still reaches Loki; the traceback goes to the Pod log. http.client errors (IncompleteRead, RemoteDisconnected, InvalidURL) are caught on API GETs, the gator download, posts and pushes. - Webhook and Loki posts share post_json(): the Request is built inside the try, a malformed URL gives the fixed reason "invalid URL", and every other error text goes through hide(), which also removes the URL's path. A scheme-less or whitespace webhook value fails that team only. - Audit check is per Constraint and no longer depends on list order: a stale or unaudited Constraint is skipped (listed under "skipped"), the fresh ones are still compared; the rule is "skipped" only when none was fresh. - A failed Loki push now also updates the logged run line (status partial, reason "loki push failed: ..."). Tests: 3d-dev fixture workload so 3d = 3d-dev + dem-dev is checked; violation stream labels; a failed post to infra itself; malformed webhook values; Loki HTTP 400/429/302; API 403/500/URLError/IncompleteRead/bad JSON; unexpected errors (KeyError, unwritable /work); sorted names whatever the gator order; audit order; gator stdin checked with a pipe on fd 0 (not the runner's stdin); tarball with extra members; redacted skipped reason. Claude-Session: https://claude.ai/code/session_019uxkhSxKXFuSBsYBAmh8rB
- daily-report.sh: CronJob wiring (script and teams.json in the ConfigMap mount, --work on an emptyDir, bound ServiceAccount, restartPolicy Never, activeDeadlineSeconds 900); every value reaches the job env with non-default values, and every env var report.py reads is rendered; ClusterRole follows constraintKinds; defaultTeam gets its webhook when namespaceTeams does not name it; gk-daily-report-failed threshold (> 0) and states; a mirror gator URL renders. - render-failures: no schedule, no webhooksSecret, non-numeric maxNamesPerRule, negative maxMessageChars, bad lookback, empty checkDays, constraintKinds as a string, non-http readmeUrl and gator.url. - gk-daily-report-missing: execErrState Error (design-final §3) instead of Alerting, so a datasource error outside the check window is reported as a DatasourceError, not as a missing run. - gator.url must contain gator.version anywhere (a mirror may carry it only in the file name); the sha256 pin guarantees the content. Deliberate deviations from design-amendments.md, kept on purpose: - dead-man lookback is 9h, not [3h]: with [3h] the 07:30 run leaves the window at 10:30 and the rule would fire until 16:00 every working day. - the check window is whole hours (checkFromHour/checkToHour), not hour()*60+minute(). Claude-Session: https://claude.ai/code/session_019uxkhSxKXFuSBsYBAmh8rB
A dryRun deploy is the rollout gate before going live, but a missing
<TEAM>_WEBHOOK Secret key only showed as "NOT SET" in the Pod log: the run
stayed ok. Now a team with no webhook fails in a dry run too (partial,
failed=[team], infra told, gk-daily-report-failed fires); only the post
itself is suppressed. parity.sh gives its dry run a dummy webhook.
A skipped run's auditCheck is now {"<rule>": "skipped"} per configured
rule, as the Loki contract says, instead of {}.
Tests that pin what review mutations showed unpinned:
- DRY_RUN="true" from the env alone (no --dry-run, as in the CronJob)
posts nothing; env_config parses DRY_RUN and POST_WHEN_CLEAN
- hide(): Slack and HTTP error bodies that echo the webhook path without
a scheme never reach the run line, infra, Loki or the log
- a StatefulSet with a Deployment's name is a separate workload (R7 key
has the kind; new fixture raster-dev sts/api)
- routing matches the whole namespace name, not a prefix
- a failure after counting (bad status.totalViolations) pushes the run
line only
- the audit tolerance boundary (diff = max(5, 10%) is within) and the 5
floor
- 410 Gone keeps its own reason
- daily-report.sh renders a non-default webhooksSecret
README: dry-run failure semantics, the rollout-day dead-man note (a
deploy inside the check window fires gk-daily-report-missing unless a run
lands within 15 minutes), skipped auditCheck in the contract.
MAPCO-11867
Claude-Session: https://claude.ai/code/session_019uxkhSxKXFuSBsYBAmh8rB
…t and exact
- gk-daily-report-missing: execErrState KeepLast instead of Error. In Grafana
12.3.1 Error raises a DatasourceError alert at once, past the check window
and `for`, whenever Loki or Prometheus is not ready at stack start.
Checked on a local grafana:12.3.1: the Error copy went to Error with an
alert_team=infra instance, the KeepLast rule stayed "Normal (Error, KeepLast)".
- write_inputs: remove the files a previous run wrote to <work>/in, and skip
the run on any other file there, so a reused --work never feeds gator stale
objects.
- auditCheck in the run line keeps the Loki contract's shapes
({"gator","audit"} or "skipped"); Constraints without a fresh audit are
named in the Pod log instead of a "skipped" list.
- Tests: every network call (webhook and Loki posts, kube API GETs, gator
download) must set a finite timeout; in-cluster ServiceAccount token and CA
are used (and missing ones skip the run); a reused work dir; the
constrainttemplates rule is get on constraintKinds by resourceNames; render
failures for a bad team name with dailyReport alone, a dashboardUrl with
"|", and checkToHour 25.
Claude-Session: https://claude.ai/code/session_019uxkhSxKXFuSBsYBAmh8rB
report.py reads namespaces, Deployments, StatefulSets and the Constraints only as collections (a GET on the collection path, the list verb). Its only single-object GET is on the report's ConstraintTemplates, which keep their own get rule limited by resourceNames. Drop get from the other three rules; the test pins list per rule. Claude-Session: https://claude.ai/code/session_019uxkhSxKXFuSBsYBAmh8rB
…s are off Grafana keeps a provisioned rule that only disappears from the file, and dev/prod Grafana store rules in Postgres. Turning the report off (or a rollback to a chart without it) left both rules behind, and with no runs in Loki gk-daily-report-missing fired every check day with no way to clear it from the chart. With dailyReport.enabled and alerts.enabled false the rule ConfigMap now holds only deleteRules for gk-daily-report-missing and gk-daily-report-failed (the blackbox-lib pattern). The default render (report off) is unchanged. The README and values say to deploy with alerts.enabled false first, then turn the report off or roll back. Claude-Session: https://claude.ai/code/session_019uxkhSxKXFuSBsYBAmh8rB
…ints are recreated A chart upgrade recreates the ConstraintTemplates and so the Constraints; their first audits read totalViolations 0 before the templates load. A report run in that window (seen on dev 2026-10-26: gator 157/139 vs audit 0/0) was marked partial and fired the failed-run alert. An audit now counts only if it started at least 10 minutes after the Constraint's creationTimestamp. Claude-Session: https://claude.ai/code/session_019uxkhSxKXFuSBsYBAmh8rB
CI (Helm 4.3.1) keeps the trailing newline of a ConstraintTemplate's rego block where Helm 3 drops it, so the committed fixtures were "stale" in CI only. The generator now writes rego and libs with exactly one trailing newline. Claude-Session: https://claude.ai/code/session_019uxkhSxKXFuSBsYBAmh8rB
ronenkapelian
previously approved these changes
Oct 28, 2026
…#188) * feat(gatekeeper-constraints): provision a workload policy dashboard Add an opt-in Grafana dashboard "Gatekeeper workload policy" (uid gk-workload-policy) for MAPCO-11868, shipped alloy-style as a ConfigMap with the dashboard sidecar label and folder annotation, rendered into dashboards.namespace (the namespace Grafana's sidecar watches). The dashboard reads the Loki lines the daily report pushes (summary, team_total, violation and run lines under dailyReport.loki.job): stats, per-team and per-namespace tables, workloads and messages for the selected run, mclabels split into missing and invalid, silenced shown as a team, a 14-day per-team trend from the team totals (max of the previous 24 h, so reruns never add up and teams at 0 show 0), distinct deploy-time violations and the run status. dashboards.enabled is false by default; rendering fails without a namespace. tests/dashboards.sh and render-failure cases run in make test. Claude-Session: https://claude.ai/code/session_019uxkhSxKXFuSBsYBAmh8rB * fix(gatekeeper-constraints): dashboard review fixes and per-query tests - namespace variable: All is .+, so New at deploy shows admission violations in namespaces no report run has seen (before the first run it showed nothing). - Trend: highest run of the previous 12 h instead of 24 h. Runs only happen 04:00-16:00 UTC, so each point is that run day's highest run and a manual rerun never carries into the next morning. The README and the panel text now say a lower same-day rerun does not replace an earlier, higher one. - Selected-run stats show "no run" (not a green 0) when no run is selected, and 0 only for a selected run with nothing in the filter. - Last run shows status, age and failed teams. - tests/dashboards.sh pins each report query per panel and refId: selector (run, team, namespace filters), line type, rule, unwrapped field, outer aggregation, window, step and legend; the table column names; the Last run stat's reducer, fields and overrides; and the namespace variable's allValue. Claude-Session: https://claude.ai/code/session_019uxkhSxKXFuSBsYBAmh8rB * fix(gatekeeper-constraints): count admission violations per rule, widen tables, pin dashboard contract - New at deploy: group by constraint_kind instead of constraint_name, so a workload flagged by both <rule>-warn and <rule>-enforced (a namespace in both lists) counts once, as the panel says. Checked on a local Loki 3.7.3: 4 warn+enforced lines for one Deployment gave 2 stacked series before, 1 after; the new query also runs on dev Loki. - Per team and Per namespace tables are full width, one under the other, so the mclabels missing/invalid columns are visible without scrolling (checked in Grafana 12.3.1 at 1600px). Per namespace fills a rule with no summary line in a namespace with 0 in the query (noValue does not fill cells left empty by the merge transform). - tests/dashboards.sh now also pins: the Loki datasource on every query panel and target; the 7-day default range and the 14-day trend/admission panels (no timeFrom elsewhere); team/namespace multi-select with All; the exact admission query, legend and 1h step; the Workloads fields, columns, extractFields transform and maxLines >= 5000; the Violation messages line_format; the run status colours on Last run and Runs, the Last run direction and the Runs fields; the unstacked dashed totals; the table width; and the Per namespace 0 fill. - render-failures/dashboards-blank-folder.yaml covers a whitespace-only dashboards.folder. Claude-Session: https://claude.ai/code/session_019uxkhSxKXFuSBsYBAmh8rB * fix(gatekeeper-constraints): keep the trend and old run links working, pin dashboard structure - team variable: All is .+, so the 14-day trend still draws when the dashboard range holds no run (All otherwise expands to no options). - Default time range is now-14d, the Loki retention: the Slack link carries only var-team and var-run, so a run older than a 7-day default read "no run" although Loki still held it. - dashboards.sh now asserts the Loki datasource and refresh of every query variable, a single-value run variable, every panel's type, the bar chart's x field, merge+organize on the summary tables and bar chart with their key columns shown, extractFields on labels before organize for Last run, Runs and Workloads, and the Runs columns and line cap. Claude-Session: https://claude.ai/code/session_019uxkhSxKXFuSBsYBAmh8rB * docs(gatekeeper-constraints): document the Loki contract once The daily report's "Loki contract" is the single description of the line format. The dashboard section links to it and keeps only which lines feed which panels. Claude-Session: https://claude.ai/code/session_019uxkhSxKXFuSBsYBAmh8rB * fix(gatekeeper-constraints): Last run no longer claims to show a failed Loki push When the push fails report.py does not push again, so that run exists only in the Pod log: the dashboard shows the previous run or no run, never a push-failure partial. The panel description now says so. Claude-Session: https://claude.ai/code/session_019uxkhSxKXFuSBsYBAmh8rB * feat(gatekeeper-constraints): trend panels as daily bars of each day's last run The two trend panels now show one bar per UTC day with a report run, stacked per team, as amendment R8 asked. Each bar is the last run of that day per team: last_over_time(... [1d]) by (team) at a 1d step. Every run writes its own run_id stream, so grouping on the range aggregation makes a same-day rerun replace the earlier number instead of keeping the larger one. Loki aligns 1d steps to UTC midnight and the point at D+1 covers day D, so barAlignment -1 puts each bar on its own day. Totals are outlined, unstacked bars. Today's point lies after now, so a "latest run" series (instant last_over_time over the panel range, split per team, drawn as stacked dots) shows today's run before its bar appears at 00:00 UTC. tests/dashboards.sh pins the query shape, the 1d step and interval, the bar style and alignment, and the latest-run series. Claude-Session: https://claude.ai/code/session_019uxkhSxKXFuSBsYBAmh8rB * fix(gatekeeper-constraints): hide the derived traceID column in log tables The dev Loki datasource's derived field adds an empty traceID column to the Workloads and Runs tables (seen on grafana-dev 2026-10-26). Claude-Session: https://claude.ai/code/session_019uxkhSxKXFuSBsYBAmh8rB * fix(gatekeeper-constraints): show current-hour admissions, document the folder race New at deploy steps hourly, so admissions from the current hour showed only after the next hour started. An instant query of the same 24 h count adds "now" dots at the right edge, split per namespace and rule like the trend panels' latest-run dots. README: on a Grafana without the folder yet, the dashboard sidecar and the alert rule provisioning can each create it at once and leave an empty duplicate; how to avoid or clean it up. Review: #188 Claude-Session: https://claude.ai/code/session_019uxkhSxKXFuSBsYBAmh8rB
ronenkapelian
approved these changes
Oct 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a daily report of workload policy violations to the gatekeeper-constraints chart. A CronJob runs
report.py(Python stdlib only) at 07:30 UTC, Sun-Thu. It reads the live workload ConstraintTemplates and Constraints, plus every Deployment, StatefulSet and Namespace. It then runs gatorv3.22.0(a pinned release tarball, checked by sha256) on a projection of those objects. No Rego is copied, so exemptions behave exactly as deployed.Each team gets one Slack message for its namespaces, with counts per rule, up to 10 names with "+N more", and a link to the dashboard. Every run is also pushed to Loki for the dashboard (MAPCO-11868,
feat/gatekeeper-constraints-dashboard). Two Grafana rules tell infra when a run is missing or did not endok.The report is off by default (
dailyReport.enabled: false). If it is enabled with a required value missing or malformed, rendering fails with a clear error.Decisions (confirmed by the requester; posted on MAPCO-11867)
schedule: "30 7 * * 0-4"(07:30 UTC, Sun-Thu).(empty), an owner outside the allowed list,environment=undefined, and any unknown problem text. A unittest pins the rego message formats.team="silenced"and appear only on the dashboard.gator.urlcan be pointed at a mirror.Deviations from the design, for the record (approved by the requester)
lookback: 9h(configurable), not[3h]. The amendment pairs Lokicount_over_time(...[3h])with a 09:00-16:00 gate. The run is at 07:30, so a 3h window stops covering it at 10:30, and the rule would then fire falsely every day from about 10:30.lookbackmust reach back to the run fromcheckToHour(16:00 - 07:30 = 8.5h) and stay shorter than the gap to the previous run, so the default is 9h. The gate is built from whole hours (hour(),day_of_week()), nothour()*60+minute(), because both ends of the window are whole hours./api/ds/query): A=0, B=1, C=1, all status 200. The rule's two-datasource math model is accepted.execErrState: KeepLast, notError. In Grafana 12.3.1,Errorraises a DatasourceError alert (alert_team=infra) as soon as a query errors, skipping the check window andfor: 15m. This happens every morning while Loki or Prometheus is not yet ready after the 04:00 stack start. The live state history of other Error-mode rules shows exactly this pattern.KeepLast, a query error keeps the previous state.partial, and infra is told by the job itself.Errorcopy went toErrorand created an alert instance withalert_team=infra. TheKeepLastrule stayedNormal (Error, KeepLast).for: 15m. No design document gives afor; the rule needs 15 minutes of "no run" inside the check window before it fires. Side effect: on the day the rules are first deployed no run is in Loki yet, so a deploy inside 09:00-16:00 UTC fires the rule 15 minutes later. Deploy W4 outside 09:00-16:00 UTC, or start the manual Job within 15 minutes of the deploy (a dry run counts).postedlists the teams that would have been posted (the run line carriesdryRun: true), and a team with no<TEAM>_WEBHOOKkey still fails (statuspartial, exit 1,gk-daily-report-failedfires to infra) even though nothing is posted. This is deliberate, so the W4 dry run catches a missing Secret key before W5, but the dashboard's Runs table and the failed-run rule see these meanings during W4.auditCheckkeeps the contract's two shapes. These are{"gator":N,"audit":M}and"skipped". When only some Constraints of a rule have a fresh audit, the counts cover those Constraints only, and the Pod log names the others. No third shape is added.Acceptance gates (R10/R11), in this order
report.py --from-cluster --dry-runthroughkubectl proxy(no Loki push, nothing posted).ok.--workgave the same counts.auditCheck.dryRun: true:kubectl -n infra-dev create job --from=cronjob/gatekeeper-constraints-daily-report gk-report-manual-1(with consent).gator.urlto a mirror.ok. In a dry run a missing<TEAM>_WEBHOOKkey turns the statuspartial.{job="gatekeeper-workload-report", team="_run"}.placing ... in: /tmp/dashboards/Gatekeeper./d/gk-workload-policyrenders.dryRun: false. Confirm the posts in the 4 channels at the next 07:30 UTC run.Definition of done (R12)
dryRun: false) release is deployed in azure-dev and the first 07:30 UTC posts arrive.Turning it off or rolling back
Grafana keeps a provisioned rule that only disappears from the file (dev/prod store rules in Postgres), and with no runs
gk-daily-report-missingwould then fire every check day. WithdailyReport.alerts.enabled: falsethe rule ConfigMap holds onlydeleteRulesfor both uids. So: first deploy withdailyReport.alerts.enabled: false, then setdailyReport.enabled: false. Ahelm rollbackto a revision before this feature needs that intermediate deploy first.RBAC
The ClusterRole has
liston Namespaces, Deployments, StatefulSets and the Constraints ofconstraintKinds(report.py reads only collections), andgeton those kinds' ConstraintTemplates byresourceNames. No Secrets, nopods/log, no write verbs.Shared-release hazard
prod pins gatekeeper-constraints 1.0.0 on the same release. A prod-side sync of that release would remove everything this PR adds in dev. This is the same hazard that already exists for deployAlerts. Do not sync prod from this branch.
Merge order
dryRun: true, thendryRun: false.Tests
make -C charts/gatekeeper-constraints testrunstests/daily-report.sh, thetests/render-failures/daily-report-*.yamlcases, thetests/daily-report/unittests,parity.sh(gator on the original objects = gator on the projection) andjob-pod.sh(the job's own Pod passes every policy). Every assertion was first shown failing against a temporary mutation, then the mutation was reverted:initContainers'app' != 'raster')CPU limit must be exactly 4xmapcolonies.io/criticalityMissing required labelpods/logresourceNamesdropped from constrainttemplatesconstrainttemplates must be get-only and limited ...execErrState: Error[get, list]ClusterRole verbs must be list onlyalerts.enabled=falsedeleteRules lose a uidalerts.enabled=false must render deleteRules for both uidsnetwork call without a finite timeout: None<work>/infiles keptgot 24 files)auditCheckgets askippedlist again|/ nocheckToHour <= 24expected rendering to failJira notes (posted on MAPCO-11867; records D1, D6, D7 and D8)
https://claude.ai/code/session_019uxkhSxKXFuSBsYBAmh8rB