feat(dr): consume KSail for opt-in monthly recovery and controlled platform replacement #5631

🤖 Generated by the Agentic Engineer

User Story

As a platform owner,
I want an explicitly enabled monthly recovery exercise that uses KSail to build and validate a replacement platform,
So that recovery is demonstrated with real application data while the serving platform remains available, and a proven replacement can take over safely.

Owning capability

The reusable capability is tracked in KSail #8093. That issue owns the opt-in backup/restore integration and evaluation of Velero. This issue owns the consuming configuration and external automation; do not duplicate the backup engine or put autonomous disaster detection and monthly platform replacement inside KSail.

Acceptance Criteria

• Configure an explicit opt-in monthly exercise using the supported KSail capability. Routine runs are autonomous and require no maintainer PR approval or per-run consent. Activation follows demonstrated readiness, not merely installation of KSail.
• Run recovery orchestration with the required bootstrap material and backups available independently of the platform being recovered.
• Restore and verify the declared applications and their data in a distinct replacement environment. Enforce isolation before restored controllers or jobs can publish production DNS, claim shared resources, consume production queues, or perform external writes.
• Validate backup coverage and application-level recovery, including database-native mechanisms where needed. A running cluster, successful resource restore, or partially completed backup is insufficient.
• Use explicit handover policy and proven KSail operations to catch up changes made after the backup, transfer write/traffic/integration ownership, and validate the replacement. Preserve the functioning source if pre-handover checks fail; do not silently relax availability or data-loss requirements.
• Retire the previous platform only after the replacement's application read/write checks and backup coverage pass and a data-preserving failure/rollback procedure exists. Preserve shared infrastructure and recovery points.
• Record recovery time, recovered-data freshness, actual coverage, handover impact, and cleanup results. Separately prove bootstrap/recovery without access to the old cluster, since a planned replacement alone does not prove total-loss recovery.

Agents own implementation and challenge unsupported assumptions. Reuse existing recovery work and retain platform-specific application policy here.

Triage follow-through

Native Feature type, an appropriate Epic, a native dependency on KSail #8093, and project-board placement remain to be applied with repository tooling. The available issue connector cannot set those native fields; the link above is a reference, not a native dependency.

Agreed recovery identity boundary — 2026-10-31

• Establish isolated identity and authorization before restored workloads or controllers start. Identical namespace, ServiceAccount and application names must not grant a validation cluster production access.
• Account for credentials, issuer/signing material, secret-store policies and trust configuration copied by a restore. A new cluster name, DNS name or address alone does not establish isolation. Restored Secrets or a restored broker must not make production write privileges usable.
• Prove denial of production writes for applicable DNS, GitHub, backup/storage and other external integrations while the original platform remains active. Provide only the explicitly required recovery access; protect existing backups against modification or retention cleanup by the test environment.
• Grant or transfer production authorization only through the validated handover, alongside data catch-up, traffic ownership and single-writer enforcement. Retire the old environment's authorization while preserving the defined data-safe rollback/recovery path.
• Demonstrate bootstrap and backup access when the original cluster and its identity services are unavailable, without giving an isolated test restore production privileges.

This remains opt-in, externally orchestrated monthly recovery. KSail supplies reusable operations and integration; this consuming automation owns scheduling, recovery decisions and takeover. Related: KSail #8093, #2664 and #2831.