feat(dir): add a dry-run RPC to try a candidate content policy on any node #2290

Background

PR #2275 adds a dry run for content policies: it evaluates a candidate policy on a node's real records and reports how many it would exclude and why, without storing anything. It has two entry points:

• dirctl daemon policy dry-run, for a local daemon, which opens the daemon's database and data directory directly.
• reconciler dry-run, run inside the reconciler pod with kubectl exec.

Review feedback on #2275 (#2275 (comment)): a single command would be better, through a dry-run RPC.

Proposal

Add a server-side dry-run RPC and one command, dirctl policy dry-run, that calls it on the node in the user's context. It would work the same on a local daemon and on Kubernetes. The two entry points from #2275 would then be removed.

• Request: the candidate (the same validators list as the configuration, entries with op: ["evaluate"]), the contents of any OPA files it names (in place of --policy-dir), the policy to try (optional) and the number of samples.
• Response: the same report as today: evaluated, would pass, would exclude (rejected, could not evaluate) and a sample of excluded records with reasons. Whether to stream progress is an open question (see below).
• Keep from feat(dir): try a candidate content policy before deploying it #2275:
  • It cannot write: it reads through an interface with no write method, so it can't store a verdict or register a policy version.
  • It uses the same evaluator and the same batch size and per-record timeout as the policy task.
  • A record it can't read or judge counts as excluded.

Security

• Same privilege as the audit service. A dry run reads every record, including ones the enforced policies already hide, and reports their CIDs and reasons. A candidate policy can also be used to probe the contents of hidden records. So the RPC must be an identity-only method like PolicyAuditService: granted only by a rule naming the caller's SPIFFE ID, offered only when authorization is on, and audit-logged. Add it to identityOnlyMethods in server/authz/authorizer.go.

• Caller-supplied policy code. Today policies come from the operator's configuration. The validators now:

  • build OPA with the default built-ins (server/validators/opa.go), so a caller's policy could call http.send or other network built-ins;
  • set no cost limit on CEL (server/validators/cel.go).

  Before this ships, restrict OPA's capabilities for this path (no network, no runtime access), set a CEL cost limit, and keep the per-record timeout.

• Request size. Bound the size of the candidate and of the OPA files.

Performance

• A dry run reads and evaluates every record, so it must not starve normal traffic: limit concurrent dry runs per node (one at a time may be enough), apply an overall deadline, and stop work when the client disconnects.
• Keep the paged reads and bounded memory of the current implementation. Add metrics for dry-run count, duration and records evaluated.

What changes

• Move the evaluator and dry-run code out of the reconciler module (reconciler/tasks/policy, reconciler/dryrun) into a package the server can import, since the reconciler imports the server and not the other way round.
• New proto and service, server implementation, client method, dirctl policy dry-run.
• Remove dirctl daemon policy dry-run and reconciler dry-run.
• Update the runbook section "Trying a Policy Before You Deploy It", the CLI reference and the E2E test.

Done when

• dirctl policy dry-run --candidate ./candidate.yaml gives the same report on a local daemon and against a Kubernetes deployment.
• A caller not named by a SPIFFE ID rule is refused, and every call is audit-logged.
• A candidate OPA policy that calls a network built-in is refused, and a CEL expression over the cost limit is rejected.
• A dry run changes nothing a read returns, and this is tested.
• Concurrent and oversized requests are limited.

Open questions

• Unary response or a server stream with progress? A node with many records may take minutes.
• A new PolicyService or an addition to PolicyAuditService? A separate service keeps the authorization boundary clear.
• Should the RPC be limited to one policy per call?