Skip to content

Fix clean_jobs missing --namespace flag causing BackoffLimitExceeded - #837

Merged
ricardobernardino2024 merged 1 commit into
release-17.0.0from
chunk/fix-clean-jobs-namespace
Aug 31, 2026
Merged

Fix clean_jobs missing --namespace flag causing BackoffLimitExceeded#837
ricardobernardino2024 merged 1 commit into
release-17.0.0from
chunk/fix-clean-jobs-namespace

Conversation

@circleci-app

@circleci-app circleci-app Bot commented Aug 31, 2026

Copy link
Copy Markdown

Problem

deploy_to_doks_release (job 30881) failed at the "Install Codacy" step with:

Error: UPGRADE FAILED: release codacy-release failed, and has been rolled back due to atomic being set: post-upgrade hooks failed: job failed: BackoffLimitExceeded

The previous fix (#836) added a clean_jobs Makefile target to delete stale hook jobs before the helm upgrade. However, that target ran:

kubectl delete jobs -l app.kubernetes.io/instance=codacy-release --ignore-not-found=true

…without a --namespace flag, so kubectl searched the default namespace instead of codacy-release. The command returned "No resources found" and the stale hook jobs (DB migrations, etc.) in codacy-release were never deleted. On the next helm upgrade attempt, helm found the existing BackoffLimitExceeded job and immediately failed again.

Fix

Add --namespace ${NAMESPACE} to the kubectl delete jobs command in the clean_jobs target so it targets the correct namespace for every environment (dev, sandbox, release).

clean_jobs:
    -kubectl delete jobs -l app.kubernetes.io/instance=${RELEASE_NAME} --namespace ${NAMESPACE} --ignore-not-found=true

Test

Re-running deploy_to_doks_release should now successfully delete stale hook jobs in the codacy-release namespace before the helm upgrade proceeds.

https://app.circleci.com/agents/gh/codacy/chat/53169a49-798b-45fd-a1cc-9721760e5036

The clean_jobs Makefile target ran kubectl delete jobs without specifying
the --namespace flag, so it searched the default namespace instead of the
release namespace (e.g. codacy-release). This meant stale hook jobs (DB
migrations etc.) from prior failed deployments were never cleaned up, and
helm's next attempt would find the existing BackoffLimitExceeded job and
immediately fail again.

Add --namespace ${NAMESPACE} to the kubectl delete command so it targets
the correct namespace for every environment (dev, sandbox, release).

AI-Generated: true
@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

AI Reviewer: first review requested successfully. AI can make mistakes. Always validate suggestions.

Run reviewer

TIP This summary will be updated as you push new changes.

@codacy-production codacy-production Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

The PR addresses a deployment blocker by adding the missing --namespace flag to the clean_jobs Kubernetes deletion command, ensuring stale jobs are cleared from the correct environment. Codacy analysis indicates that the changes are up to standards with no new quality issues or complexity regressions.

While the logic appears sound, there is a gap in automated verification. A required test scenario to confirm the correct population of the namespace and label selector variables in the kubectl command is currently missing. Implementing this verification is recommended to prevent future regressions in environment-specific cleanup tasks.

Test suggestions

  • Verify that clean_jobs executes kubectl delete with the correct namespace and label selector variables
Prompt proposal for missing tests
Consider implementing these tests if applicable:
1. Verify that clean_jobs executes kubectl delete with the correct namespace and label selector variables

TIP Improve review quality by adding custom instructions
TIP How was this review? Give us feedback

@ricardobernardino2024
ricardobernardino2024 merged commit e4c8bdd into release-17.0.0 Aug 31, 2026
9 checks passed
@ricardobernardino2024
ricardobernardino2024 deleted the chunk/fix-clean-jobs-namespace branch August 31, 2026 22:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant