refactor(velero): remove privileged node-agent by moving Longhorn data movement to Longhorn backupTarget→R2 #2301

🤖 Generated by the Daily AI Assistant

Goal

Eliminate the privileged Velero node-agent DaemonSet (runs runAsUser: 0 + hostPath /var/lib/kubelet/{pods,plugins}), which is the source of Kubescape C-0057 (privileged), C-0048 (hostPath), C-0045 (writable hostPath), currently suppressed by the velero entry in k8s/bases/infrastructure/security-exceptions/infrastructure-privileged.yaml. Removing node-agent lets us delete that exception entirely.

Why this is NOT a simple toggle (investigation findings)

CSI snapshots are already enabled and live (features: EnableCSI, Longhorn VolumeSnapshotClass longhorn-snapshot-vsc labeled velero.io/csi-volumesnapshot-class: "true", snapshot-controller running). But node-agent has two essential roles that block its removal:

1. Kopia data-mover — the daily schedule uses snapshotMoveData: true, so CSI snapshots are uploaded off-site to R2 by node-agent's Kopia data-mover (live: 119 DataUploads). Drop node-agent → snapshots stay in-cluster only → no DR copy.
2. FSB fallback for OpenBao — OpenBao runs on hcloud block storage (csi.hetzner.cloud, no snapshot capability); the volume-policy + defaultVolumesToFsBackup: true route it to Kopia filesystem backup, which requires node-agent (live: 138 PodVolumeBackups).

So the privilege node-agent needs is inherent to filesystem-level data movement. The real fix moves that data movement into Longhorn (already privileged + exception-covered) — a net reduction in privileged components, achieved via re-architecture.

Proposed migration

1. Configure Longhorn backupTarget → R2. Set Longhorn's backup-target (S3 to the same R2 bucket/prefix or a sibling) + backup-target-credential-secret (via external-secrets). Add a Longhorn VolumeSnapshotClass with parameters.type: bak (off-site backup) alongside the existing type: snap.
2. Switch Longhorn volumes from data-mover to Longhorn backup. Point the velero volume-policy / VolumeSnapshotClass at type: bak so the snapshot is exported to R2 by Longhorn, and set snapshotMoveData: false (no Kopia data-mover).
3. Re-home OpenBao's backup. Either move OpenBao's PVC to a snapshot-capable StorageClass (Longhorn), or confirm the existing OpenBao raft-snapshot CronJob is the sufficient backup of record and drop OpenBao's PVC from Velero FSB. (Decision needed — raft snapshot is logical, PVC backup is physical; pick one.)
4. Remove node-agent. deployNodeAgent: false; drop defaultVolumesToFsBackup: true; remove the node-agent configmap/extraArgs.
5. Delete the exception. Remove the velero entry (C-0057/0048/0045) from infrastructure-privileged.yaml.

Restore validation loop (MUST pass before promotion)

Data protection is silent-failure-prone, so each step is gated on a real restore:

1. Create a throwaway namespace + Longhorn PVC + pod writing a known sentinel value.
2. velero backup create (CSI snapshot → Longhorn type: bak → R2). Confirm the backup is Completed and the snapshot landed in R2 (not just in-cluster).
3. Delete the namespace (PVC + data gone).
4. velero restore create from the backup.
5. Verify the PVC is recreated and the sentinel value is intact.
6. Repeat for the OpenBao volume under whatever path step 3 chooses.
7. Only after a clean restore of both a Longhorn volume and the OpenBao path: disable node-agent and promote.

Risks / rollback

• Silent DR loss if step 1–2 don't actually move snapshots off-site — the restore test gates this.
• Longhorn backupTarget credentials/endpoint must be correct (reuse the R2 external-secret pattern).
• Rollback: re-enable deployNodeAgent + snapshotMoveData: true + defaultVolumesToFsBackup: true (revert this change) restores the current working (if privileged) path.

Acceptance criteria

• Longhorn backupTarget→R2 configured + a restore test of a Longhorn volume passes.
• OpenBao backup re-homed + its restore/recovery path validated.
• node-agent DaemonSet gone; nightly schedule Completed for ≥2 nights.
• velero entry removed from infrastructure-privileged.yaml; C-0057/0048/0045 no longer need the exception.

Out of scope / separate

The current nightly-backup failures (06-25/26/27 Failed, velero server restarting mid-backup) are a separate live issue — see the diagnosis: node-agent crashloop (06-20→06-23, fixed by the v0.3.142 pin) + VPA InPlaceOrRecreate on a spiky backup tool. Fix/verify that before starting this migration so the validation loop runs against a healthy baseline.