Add Azure Network Layer assurance and routing checks

[ritiksah141]

Summary

Adds Azure public-cloud Network Layer assurance with complete addressing, routing, transit, protection, and observability coverage plus customer-actionable Layer 3 routing checks.

What changed

• Added a validated catalog covering all 20 Layer 3 domains and five functional subdomains.
• Added one explicit assurance control for every Layer 3 domain.
• Added authenticated GET /api/assurance/network-layer reporting with separate catalog coverage and evidence freshness.
• Audited every AZ-NET-* scanner rule by its actual OSI behavior.
• Preserved port-specific NSG checks as Layer 4, DNS and WAF controls as Layer 7, and ExpressRoute Direct MACsec checks as Layer 2.
• Added AZ-NET-016 for network interfaces with IP forwarding enabled.
• Added AZ-NET-017 for explicit IPv4 or IPv6 default user-defined routes using the direct Internet next hop.
• Added safe remediation playbooks, all four framework mappings, and offline compliant, non-compliant, empty-inventory, and API-failure tests for both rules.
• Preserved empty inventory as not applicable and Azure API or permission failures as indeterminate without false findings.
• Added closed validation for domains, subdomains, controls, responsibility, applicability, evidence, verification state, and rule classifications.
• Resolved bug: scanner engine loads rule modules that CI validation and doc counts never see #244 by aligning runtime rule discovery with CI validation and adding a regression guard against misnamed rule modules.
• Made no frontend changes and did not score provider-owned controls.

CIS mapping decisions

All CIS mappings were audited against CIS Microsoft Azure Foundations Benchmark 2.0.0. AZ-KV-006 now maps to the genuine RBAC recommendation 8.6, while AZ-KV-004 maps to recoverability recommendation 8.5. Rules without a direct benchmark recommendation use rule-specific N/A-* identifiers instead of unresolved TBD-* placeholders.

N/A is an explicit reviewed decision: it prevents OpenShield from claiming that a rule implements an unrelated CIS recommendation and preserves the repository invariant that each numbered CIS control maps to one OpenShield rule. This includes the Layer 3 routing checks, Layer 2 MACsec checks, application identity checks, supply-chain checks, the overlapping Key Vault soft-delete prerequisite, and the proactive 30-day certificate-renewal check.

Verification

• Backend suite: 556 passed, 2 skipped.
• Layer 3 assurance and network rule regression tests pass.
• Ruff check passes.
• Bandit reports no high-severity issues.
• DCO verification passes for all commits.
• Python compilation, compliance JSON validation, and playbook shell syntax pass.
• Microsoft evidence links resolve successfully.

Type of change

• New scan rules
• Remediation playbooks
• API endpoint
• Assurance catalog
• Rule classification audit
• Scanner discovery fix
• Documentation
• Compliance mappings

Testing

• Tested against a real Azure free trial subscription
• Returns correct JSON output
• Local CI-equivalent backend checks pass
• No hardcoded credentials or secrets

Checklist

• Every commit includes a DCO Signed-off-by trailer
• Every Layer 3 domain and subdomain is covered
• Every existing and new AZ-NET-* rule is classified
• Runtime and CI use the same rule-discovery convention
• Non-Layer 3 rules retain their correct OSI classification
• Empty inventory and API failures do not create findings
• No speculative provider-fabric findings were introduced
• No real Azure credentials are committed
• Branch name follows the project convention

Related issues

• Closes Add complete OSI Network Layer assurance and Azure Layer 3 checks #248 — complete OSI Network Layer assurance and Azure Layer 3 checks.
• Resolves bug: scanner engine loads rule modules that CI validation and doc counts never see #244 — align scanner runtime discovery with CI validation and rule counts.

Files to review

The main implementation can be reviewed in this order:

1. compliance/assurance/network_layer.json — complete Layer 3 catalog, controls, evidence, automation decisions, and OSI classifications.
2. api/services/network_layer_assurance.py — closed validation and report generation.
3. scanner/rules/az_net_016.py and scanner/rules/az_net_017.py — new actionable Layer 3 checks.
4. scanner/azure_client.py — authoritative NIC and route-table inventory accessors.
5. tests/test_network_layer_assurance.py and tests/test_rules_network.py — catalog, endpoint, rule, empty-inventory, and failure-path coverage.
6. playbooks/cli/fix_az_net_016.sh and playbooks/cli/fix_az_net_017.sh — confirmation-based remediation.
7. compliance/frameworks/*.json — CIS, NIST, ISO 27001, and SOC 2 mappings.
8. scanner/engine.py, .github/workflows/ci.yml, and tests/test_engine_integration.py — bug: scanner engine loads rule modules that CI validation and doc counts never see #244 rule-discovery fix and regression guard.
9. docs/network-layer-assurance.md and CHANGELOG.md — responsibility boundary, limitations, and release entry.

[Vishnu2707]

@ritiksah141 - One thing, can you confirm nothing downstream pattern matches on the old TBD- prefix specifically, since this rename touches almost every rule's mapping and that logic isn't shown in the diff. If get_compliance_score() just checks key existence we're fine, but worth a quick check before i go ahead and merge it.

[ritiksah141]

@Vishnu2707, I verified that:
get_compliance_score() matches rules by rule ID, not by TBD-* and also No runtime code depends on the old prefix.

[Vishnu2707]

Good, thanks for confirming that. No other changes from my side, this looks good to merge.