Store hardening: auth, TLS and snapshot integrity

[jhrozek]

Redis dials with no auth, no TLS and no keyspace scoping, and sessnap is plain JSON with no MAC.
It should land before any multi-tenant deployment, because it buys the same defence a signed chain
would buy against one specific adversary: someone holding a leaked store credential cannot forge
a MAC. It buys nothing against a compromised pod, and that qualifier is load-bearing.

mecatl, deployment work. Unblocked, independent of everything else in #377.

Hazards

Facts that were expensive to find and that nobody re-derives on the way to implementing. Each is verified against code.

• The MAC key inherits Spike: can a model-spawned shell reach the key? #375. The MAC is applied where the snapshot bytes are produced and
  consumed, which is the per-turn hot path inside the harness process, and that process is the parent
  of the Bash tool (osfs.go:1167-1187 is a plain exec.CommandContext(ctx, r.shell, "-c", command),
  same uid, no namespace, no path allowlist; the hardened runners differ only in cmd.Env and
  cmd.Dir). The key does not have to be read out of memory to be stolen: a flag lands in
  /proc/<pid>/cmdline, an env var is readable at the parent's /proc/<harness-pid>/environ because
  envscrub strips names from the child's environment only, and a mounted Secret file is a fixed
  path that cat reaches under a read-only-by-verb classification with no path predicate, which an
  isolated child auto-approves. Only /proc/pid/mem needs PTRACE_MODE_ATTACH, and no manifest here
  sets ptrace_scope.
• workspace is the field that escalates, not the owner label. The persisted string flows
  verbatim into the workspace factory at run entry (service.go:2311) and becomes the
  osfs.NewWorkspace root for Read/Edit/Write/Grep/Glob, the Bash cmd.Dir, and the base the escape
  classifier is constructed relative to (build.go:6784-6811). A stored "workspace":"/" is a
  session rooted at the filesystem root with no escape left to detect. The same string pins the child
  permission resolver (build.go:1757). profile is trusted the same way in rehydrateSession
  (service.go:2403-2423), so flipping it off no-fs escalates a deliberately file-less session onto
  the full FS tool set plus Bash.
• The event log is a second authority path. approvalreplay.go:33-82 runs at every
  loadAndReopen: it reads the event log, filters EvApproval with AllowAlways, correlates each to
  a ToolCall in the loaded conversation, and re-drives policy.Learn. Both inputs live in the same
  open store. A writer forges a Bash call in the snapshot's messages plus a matching approval event,
  and at the next resume the harness learns a durable permission allow the user never granted. The
  replay code is fail-safe against every accident and defenceless against a writer.
• A payload MAC does not survive the grpcdriver round trip. The driver server decodes and
  re-encodes (sessnap.Unmarshal on Save at server.go:65, a fresh sessnap.Marshal on Load at
  server.go:93), and sessnap.Of() rebuilds from the live session, so a stored MAC has nowhere to
  ride. Either the MAC is an envelope in the store adapter, in which case that backend has no MAC at
  all, or the driver protocol carries it as an opaque field. One caller cannot reach another's work #368 already carries the adjacent
  sentence about labels being dropped at the process boundary.
• Auth, TLS and keyspace scoping stop the same adversary at the door and fix the confidentiality
  half a MAC cannot touch. If this issue gets split, those go first.
• The event log stays uncovered by decision, and one attack survives because of it. A per-record
  MAC would have closed it, and the call is to ship the snapshot MAC first and close the event log
  separately in Event-log integrity: a forged approval record still grants a durable permission #385. What remains open: an existing benign tool call in the conversation plus a forged
  EvApproval{AllowAlways} naming it is enough for approvalreplay to run policy.Learn, because
  only the snapshot half is authenticated. Do not describe this issue as closing the approval-replay
  path.

Acceptance criteria

• Auth, TLS and keyspace scoping are supported and used by the shipped manifest.
• A shared store with no auth produces a startup warning.
• The MAC covers the whole snapshot, not a chosen subset of fields. Anything a resume trusts is
  inside it, workspace, profile and mode included.
• The MAC is an envelope in the store adapter, around the serialized snapshot. The remote driver
  backend is therefore not covered, and that is a decided exclusion rather than an oversight: the
  driver server decodes and re-encodes, so a payload MAC cannot survive it. The exclusion rests on
  One caller cannot reach another's work #368's deployment requirement that the driver endpoint is never tenant-reachable, which puts it out
  of reach of the same adversary this MAC defends against.
• A verification failure reads as an integrity error, distinguishable from a transient one.
• The boundary is stated where an implementer reads it, in these words or equivalent: this MAC
  detects tampering by an adversary who can write the store but cannot execute code in a mecatl pod.
  It provides no integrity against a compromised pod or a prompt-injected model with Bash, because the
  verifying key lives in the agent-loop process and is reachable from a child shell. It does not cover
  the event log or the tool-call audit log. It is not confidentiality either; the snapshot stays
  readable to anyone who can read the store.

Proof

The same properties, adversarially, with an actor and an ordering. A property-level criterion can go green while being false; these are what stop that.

• Modify a stored row directly, then resume: refused. An unmodified snapshot still loads, so
  "refuse everything" cannot pass.
• Rewrite workspace to / in the store, then resume: refused, rather than a session whose file
  tools are rooted at the filesystem root.
• Flip profile off no-fs in the store, then resume: refused, rather than a file-less session
  that comes back holding Bash.
• Forge a tool call in the snapshot's messages, then resume: refused, so the snapshot half of the
  approval-replay path is closed.
• The same tampering is detected on both local backends. The remote driver backend is expected to
  fail this and the test says so, rather than the gap being discovered later.

Slices

Decomposition is the implementer's job; these are the natural seams, not a mandate.

• Auth, TLS and keyspace scoping, with the manifest that uses them.
• The MAC envelope over the whole snapshot, and where its key comes from.

Reasoning

The design this comes from, both on main:

• docs/agent-identity-model.md — the inbound side: who is acting, and how mecatl would identify them.
• docs/agent-identity-outbound.md — the outbound boundary: acting for a user against an external system.

Every hazard above is stated in full here rather than linked, so this issue stands on its own.

[jhrozek]

Ran a security pass over this one against the real code, and it comes back with the MAC still worth building but scoped wrong in three places. Findings below, with file and line so they can be checked.

The key is reachable, and not only through memory. The MAC has to be applied where the snapshot bytes are produced and consumed, which is the per-turn hot path inside the harness process. That process is the parent of the Bash tool: osfs.go:1167-1187 is a plain exec.CommandContext(ctx, r.shell, "-c", command), same uid, no namespace, no path allowlist, and the hardened runners differ only in cmd.Env and cmd.Dir. But nobody needs to read the key out of memory, because every channel this repo uses to deliver a secret is readable by a cat. A flag lands in /proc/<pid>/cmdline. An env var is readable at the parent's /proc/<harness-pid>/environ, and envscrub strips names from the child's environment only, so it doesn't help here. A mounted Secret file is a fixed path, and cat is classified read-only by verb with no path predicate, so IsolationApprovable auto-approves it for an isolated child with no human in the loop. Only /proc/pid/mem needs PTRACE_MODE_ATTACH, and no manifest in this repo sets ptrace_scope.

So the MAC key inherits #375 verbatim. That is worth saying out loud, because #375 is currently written as a question about the SPIFFE key.

It is still worth building. The store-credential adversary is the likeliest one here precisely because it needs no code execution in the pod: a stale connection string in a CI log, a shared Redis, a misconfigured NetworkPolicy, an offline backup. Against unauthenticated Redis it is a one-command attack. And it is cheap, crypto/hmac plus SHA-256, no new dependency.

Worth noting the ordering inside this issue, though: auth, TLS and keyspace scoping stop the same adversary at the door and fix the confidentiality half a MAC cannot touch. If this gets split, those go first.

Scope problem 1: workspace matters more than the owner label. The criterion says the MAC covers the owner and the label. The field that actually escalates is workspace. The persisted string flows verbatim into the workspace factory at run entry (service.go:2311, ws = s.cfg.Workspaces(sess.Workspace)) and becomes the osfs.NewWorkspace root for Read/Edit/Write/Grep/Glob, the Bash cmd.Dir, and the base the escape classifier is constructed relative to (build.go:6784-6811). A stored "workspace":"/" is a session whose file tools are rooted at the filesystem root with no escape left to detect. The same string pins the child permission resolver (build.go:1757), so project rules resolve from the attacker's chosen root. And profile is trusted the same way in rehydrateSession (service.go:2403-2423), so flipping it off no-fs escalates a deliberately file-less session onto the full FS tool set plus Bash. The fix costs nothing: MAC the whole snapshot blob rather than two fields.

Scope problem 2: the event log is a second authority path. approvalreplay.go:33-82 runs at every loadAndReopen. It reads the event log, filters EvApproval with AllowAlways, correlates each to a ToolCall in the loaded conversation via ev.Approval.Call, and re-drives policy.Learn. Both halves live in the same open store, the conversation in mecatl:session:<id> and the events in mecatl:events:<id>. A store writer forges a Bash call in the snapshot's messages plus a matching EvApproval{AllowAlways} in the events, and at the next resume the harness learns a durable permission allow the user never granted. The replay code is carefully fail-safe against every accident and defenceless against a writer, because it takes both inputs from the same untrusted place. A snapshot-only MAC leaves this open, and arguably makes it more attractive by closing the neighbouring door.

Scope problem 3, smaller: mecatl:tools:<id> is unauthenticated too. Nothing re-derives authority from it, but it is the trail an incident responder reads, and a writer can rewrite or truncate it.

One implementation trap. The grpcdriver server decodes and re-encodes: sessnap.Unmarshal on Save (server.go:65), a fresh sessnap.Marshal on Load (server.go:93). And sessnap.Of() rebuilds from the live session, so there is nowhere for a stored MAC to ride. A MAC over the payload bytes is destroyed by the round trip. Either it is an envelope in the store adapter, in which case a shipped backend silently has no MAC at all and this issue's criteria quietly exclude it, or the driver protocol carries it as an opaque field. #368 already has the adjacent sentence about labels being dropped at the process boundary; the MAC needs the same one.

And the summary line at the top of this issue overreaches. "It absorbs the one property the deferred SPIFFE issuer was going to provide" is true for the adversary named in the same sentence and generalises past it. The version in the long-form doc carries the qualifier, "against that adversary", and that is the one to keep. Nothing anywhere claims the MAC survives pod compromise, to be clear. The gap is that the MAC key's custody was never named as an instance of the reachability question.

Editing the body to match.

[jhrozek]

The "MAC key inherits #375" hazard is now verified rather than reasoned, and every channel it predicted works. Run against binaries built from this repo, in a kind cluster, driven through mecatl's own tool surface at --posture=auto. Detail and tiering on #375.

A canary standing in for a MAC/signing key was seeded into mecated's process environment; PID 1 in the pod is mecated.

• Parent's environment — Bash{cat /proc/1/environ} returned it along with the provider API key. Exactly the predicted mechanism: envscrub strips names from the child's environment only.
• Mounted file — a fixed-path Secret came out through Read, not Bash, because --workspace defaults to cwd (/ on the ko image). Filed as mecak8s: default workspace is the entire container filesystem — an injected agent reads the service-account token #448.
• Process memory — dd if=/proc/1/mem returned ELF magic at mecated's text segment with CapEff: 0000000000000000. /proc/sys/kernel/yama/ptrace_scope does not exist on the kind node's kernel, so the hazard's "no manifest here sets ptrace_scope" is worse than stated for that platform: YAMA is absent entirely, and the open succeeded with no capabilities at all.
• argv — demonstrated in the spike's throwaway broker, where a capability reference was recovered from /proc/*/cmdline at the same uid and then spent.

One detail worth adding to the hazard, because it is the reason this is Bash-specific rather than general: internal/app/escapeclassifier.go's classify switches on Read/Write/Edit and returns escapeInRoot for everything else, so Bash is never classified at all. Read{/proc/1/environ} is refused with "pseudo-filesystem … never relaxed at any posture"; the same path via Bash is served. The FS tools have os.Root plus the pseudo-fs guard; the command runner has a working directory and a scrubbed environment, and no OS isolation whatsoever — no chroot, unshare, seccomp, SysProcAttr or Credential anywhere in osfs.

What closes it, verified: --no-bash and --shell="" each remove the tool entirely (unknown tool "Bash"), with a positive control confirming the harness still functions. The shipped distroless mecak8s image also has no Bash — but accidentally, because there is no /bin/sh. --no-bash defaults to false and --shell to /bin/sh in every binary, so a base-image change re-opens all three channels with no flag flipped and no test failing.

Proposed acceptance criterion, since the hazard currently records the exposure without anything closing it:

• The harness refuses to hold MAC key material when a shell is reachable — checked at startup, not documented as guidance. Either the MAC key is absent and integrity is off, or Bash is absent; never both present. A deployment that wants both must supply an out-of-process signer.

The value of making it a startup refusal rather than a deployment note is that today's safe configuration is an accident of a Dockerfile, and nothing fails if someone changes the base image.

Recorded for whoever implements this: an alternative that preserves both is real OS isolation on the command runner — a distinct uid via SysProcAttr.Credential, user/PID/mount namespaces, or Landlock plus a seccomp filter denying ptrace/process_vm_readv. Any of the three closes all three channels. Untested here, and note buildSandboxedCommandRunner's "sandbox" is a git worktree: it contains writes to the repo, not reads of the host.