CI: parallel jobs, lint/format gate, security scanning, coverage (#155, #156)

[ritiksah141]

Summary

Restructures CI from a single serial job into parallel, independent gated jobs, adds the shift-left security scanning suite, brings the codebase under a ruff lint/format gate, and adds full test coverage reporting. Addresses #155 (INFRA 2) and #156 (INFRA 3) together, phase-wise, since both reshape the same pipeline.

Before merging (requires repo admin)

1. Turn OFF CodeQL "default setup" (Settings → Code security → Code scanning) before this merges. An advanced codeql.yml workflow errors out if default setup is still enabled — the two cannot coexist. This is a hard merge-blocker for the CodeQL jobs.
2. Branch protection is not part of this PR and needs admin to configure — the workflows here run but only become blocking once an admin marks them as required status checks on dev and main. Until then, checks are advisory. Exact check names and ready-to-run gh commands are in the "Admin follow-ups" section below.

What changed

CI restructure (#155)

• Split the monolithic ci-checks job into parallel jobs: lint, rule-validation, secret-scan, sast-bandit, sca-pip-audit, sbom, container-scan, backend-tests, frontend, enforce-source-branch, aggregated by ci-summary. Wall-time is now the slowest job, not the sum.
• One pytest invocation (the previous three overlapping test steps are collapsed) running the full tests/ suite against an ephemeral Postgres service, with --cov and a --cov-fail-under=25 floor.
• pip + npm caching, a concurrency block that cancels superseded runs, and permissions: contents: read (least privilege).
• Frontend is now linted and built in CI (npm ci → eslint → vite build); a broken dashboard build can no longer merge.
• New pyproject.toml ruff config (line-length = 120, E,F,W, py311). Existing violations across the codebase are fixed or explicitly # noqa/# nosec with justification so the gate lands green — this is the bulk of the file count.

Security scanning (#156)

• CodeQL (codeql.yml, python + javascript) on every PR and a weekly schedule.
• Gitleaks secret scan, Bandit SAST, pip-audit SCA (with a documented --ignore-vuln mechanism), Syft CycloneDX SBOM artifact.
• Trivy container scan scaffolded but dormant until INFRA 1 ([INFRA 1] Containerise the stack: Dockerfiles for API and worker, docker-compose local dev, split requirements #154) adds a Dockerfile — it skips cleanly and is not a required check yet.
• SBOM attached to releases: a separate sbom-release.yml workflow (triggered on: release: published) generates a CycloneDX SBOM from the tagged code and uploads it to the GitHub Release assets — satisfying [INFRA 3] Supply-chain security: CodeQL, pip-audit, Dependabot for pip/Actions, SBOM, refresh vulnerable pins #156's "attached to releases" line (the repo already publishes releases, e.g. v0.1.0).
• dependabot.yml for pip + github-actions + npm.
• Refreshed all stale pins from [INFRA 3] Supply-chain security: CodeQL, pip-audit, Dependabot for pip/Actions, SBOM, refresh vulnerable pins #156 (requests, cryptography, flask, pyjwt, gunicorn) plus three more that pip-audit surfaced (flask-cors, azure-identity, python-dotenv). Full suite still passes; gunicorn boots and serves /health.
• All third-party actions SHA-pinned with # vX.Y.Z comments (trivy-action specifically because its tags were compromised in 2026).

Branch-flow enforcement

• enforce-source-branch job blocks any PR into main whose source is not dev, enforcing feature/* → dev → main. No-op (skipped) on dev PRs.

Docs

• docs/ci-pipeline.md rewritten for the new parallel pipeline: per-job local-run commands, the branch-protection model, and tooling-decision rationale.

Notes for reviewers

• On the "pytest runs 3×" premise in [INFRA 2] Restructure CI: parallel jobs, lint/format gate, coverage reporting, frontend build check #155: confirmed accurate against current dev — there were three overlapping pytest steps. This PR collapses them into one full-suite run with coverage.
• Dependabot is notify-only (open-pull-requests-limit: 0): it still raises alerts but does not auto-open bump PRs, per maintainer direction. This intentionally diverges from [INFRA 3] Supply-chain security: CodeQL, pip-audit, Dependabot for pip/Actions, SBOM, refresh vulnerable pins #156's "Dependabot opens PRs" acceptance line — flagging for visibility.
• pip-audit ignores 3 transformers CVEs (transitive via sentence-transformers, no non-breaking fix yet) — tracked for a coordinated ML-stack bump, not silently dropped.

Admin follow-ups (cannot be done from a PR)

1. Verify CodeQL "default setup" is OFF (Settings → Code security → Code scanning) — a merge-blocker for the CodeQL jobs, since advanced codeql.yml cannot coexist with default setup.
2. Branch protection on dev and main making the new jobs required status checks — these run now but only block once marked required. Required check names:
   Lint (ruff), Rule & Compliance Validation, Secret Scan (Gitleaks), SAST (Bandit), SCA (pip-audit), SBOM (Syft), Backend Tests (pytest + coverage), Frontend (lint + build), CI Summary, Analyze (python), Analyze (javascript) — plus Enforce dev to main source on main only. (Container Scan (Trivy) intentionally excluded until [INFRA 1] Containerise the stack: Dockerfiles for API and worker, docker-compose local dev, split requirements #154.)
   Set dev = 1 approval + "up to date before merging"; main = stricter review + enforce_admins. Ready-to-run gh commands and the full model are in docs/ci-pipeline.md.
3. GitHub Environments development / staging / production (production: required reviewers + prevent self-review).
4. Enable Dependabot alerts + secret-scanning push protection — free here because the repo is public (org ownership does not restrict native secret scanning; that only affected the gitleaks-action, which is why this PR uses the Gitleaks CLI instead).

Deferred (out of scope here)

• Alembic migration CI step → belongs with [INFRA 5] Adopt Alembic for database schema migrations #158 once Alembic is adopted (no alembic.ini/migrations/ exist yet).
• Trivy container scan activation → after [INFRA 1] Containerise the stack: Dockerfiles for API and worker, docker-compose local dev, split requirements #154.

Local verification (all green)

ruff check/format, bandit -ll, pip-audit, gitleaks, actionlint, and the full pytest suite (109 passed, 2 skipped, 31% coverage) all pass locally; frontend eslint (0 errors) + vite build pass.

---

Files to review (the rest is mechanical formatting)

Of the ~147 changed files, only these need a human read — the remainder is one-time ruff format / eslint normalization (unused-import removal, whitespace, quote style) with no behavior change:

File	Why
.github/workflows/ci.yml	The core restructure — parallel jobs, gates, caching, summary
.github/workflows/codeql.yml	New CodeQL workflow
.github/dependabot.yml	New dependency-update config (notify-only)
.github/workflows/sbom-release.yml	New release-triggered SBOM upload
pyproject.toml	New ruff configuration
requirements.txt	8 security pin bumps
docs/ci-pipeline.md	Rewritten pipeline documentation

Spot-check (small, non-mechanical edits):

• api/models/finding.py, api/routes/findings.py, scanner/engine.py, scanner/nvd_client.py, scanner/rules/az_db_004.py, api/app.py — a few # nosec justifications and one SQL/import cleanup each.
• tests/conftest.py — import ordering only.
• frontend/ .jsx/.js — unused-import removal to pass eslint (mechanical).

[github-advanced-security]

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

• The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
• Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
• You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

[Vishnu2707]

async execution is the right call. one thing to keep in mind @ritiksah141 is that the in-memory job store will lose state on Render restart, lets track that as a future improvement to persist job state in the db. approving this for now but please do raise a github issue for the next PR to fix that.