Fix fluentd lifecycle hook BackoffLimitExceeded on deploy_to_doks_release

[circleci-app]

Problem

Job 30973 (deploy_to_doks_release) failed in the Install Codacy step with:

Error: UPGRADE FAILED: release codacy-release failed, and has been rolled back due to atomic being set: post-upgrade hooks failed: job failed: BackoffLimitExceeded

This happened on both the initial attempt and the automatic retry added in #838.

Root cause

codacy/templates/fluentd/lifecycle-police-job.yaml is a post-upgrade Helm hook that sets a MinIO bucket lifecycle policy. It was configured with:

• restartPolicy: Never + backoffLimit: 100 → Kubernetes creates a new pod for every failure (no reuse)
• No wait logic → each pod attempts aws s3api put-bucket-lifecycle-configuration immediately after the upgrade, while MinIO is still restarting
• Each pod fails in ~9 s (MinIO unreachable) → 100 pods × 9 s ≈ 15 minutes before BackoffLimitExceeded
• hook-delete-policy: hook-succeeded only → failed job remains in the cluster unless Helm's atomic rollback cleans it up

Because both the first attempt and the retry exhaust all 100 pods against an unavailable MinIO, neither succeeds.

Fix

codacy/templates/fluentd/lifecycle-police-job.yaml

1. Add a readiness wait loop inside the container: polls aws s3 ls every 10 s for up to 10 minutes before attempting the lifecycle operation. This tolerates MinIO restarting during the Helm upgrade.
2. Reduce backoffLimit from 100 → 3: retry logic is now inside the container, so only 3 pod-level retries are needed as a safety net.
3. Add before-hook-creation to hook-delete-policy: ensures any stale failed hook job from a previous run is deleted before the next upgrade creates a new one, preventing conflicts if Helm's atomic rollback misses cleanup.

https://app.circleci.com/agents/gh/codacy/chat/82879e75-8ec3-430b-bf8d-0bab85c5a7a4

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

AI Reviewer: first review requested successfully. AI can make mistakes. Always validate suggestions.

_{TIP This summary will be updated as you push new changes.}

[codacy-production]

Pull Request Overview

This PR effectively addresses the BackoffLimitExceeded issue by implementing a readiness wait loop and optimizing the Job configuration. While the implementation aligns with the core requirements (polling interval, backoff reduction, and cleanup policy), the readiness check itself introduces a potential race condition.

The current loop only checks for the availability of the S3 endpoint. Since this job configures a specific bucket created by a preceding hook, the script may proceed before the bucket is actually available, leading to failures despite the wait loop. Additionally, the embedded shell logic lacks automated verification, increasing the risk of regressions in the deployment environment.

About this PR

• The shell script logic added to the Helm template lacks automated tests. This makes the hook susceptible to regressions if CLI versions or shell environments change in the deployment pipeline.

Test suggestions

• Verify the shell script waits for the full 600 seconds and then exits with code 1 if MinIO remains unreachable.
• Verify the shell script proceeds to apply the lifecycle configuration immediately after 'aws s3 ls' succeeds.
• Verify the 'backoffLimit' and 'hook-delete-policy' are correctly rendered in the Kubernetes Job manifest.

Prompt proposal for missing tests

Consider implementing these tests if applicable:
1. Verify the shell script waits for the full 600 seconds and then exits with code 1 if MinIO remains unreachable.
2. Verify the shell script proceeds to apply the lifecycle configuration immediately after 'aws s3 ls' succeeds.
3. Verify the 'backoffLimit' and 'hook-delete-policy' are correctly rendered in the Kubernetes Job manifest.

Low confidence findings

• The wait loop relies on 'aws s3 ls' which might fail for reasons other than connectivity (e.g., IAM permission errors). This could cause the job to wait for the full 10-minute timeout before reporting a failure that was identifiable immediately.

_{TIP Improve review quality by adding custom instructions
TIP How was this review? Give us feedback}

[codacy-production]

_{🟡 MEDIUM RISK}

Suggestion: The current check only verifies that the MinIO endpoint is reachable. Since the script's primary action is to configure a specific bucket, the wait loop should verify the bucket's existence to avoid a race condition where the service is up but the target bucket has not yet been initialized.

Suggested change

	until aws s3 ls --endpoint-url "${ENDPOINT}" > /dev/null 2>&1; do
	until aws s3 ls "s3://${BUCKET}" --endpoint-url "${ENDPOINT}" > /dev/null 2>&1; do

[codacy-production]

_{⚪ LOW RISK}

Nitpick: It is safer to quote the annotation value to ensure consistent parsing, especially when it contains special characters like commas.

Suggested change

	"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
	"helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded"