From de371f43ab78d9ca9ea8f23de73b3c43bd82cef2 Mon Sep 17 00:00:00 2001 From: Vishnu Ajith <86302373+Vishnu2707@users.noreply.github.com> Date: Sat, 25 Apr 2026 15:07:13 +0100 Subject: [PATCH 01/15] chore: add skeleton files and requirements --- openshield | 1 + 1 file changed, 1 insertion(+) create mode 160000 openshield diff --git a/openshield b/openshield new file mode 160000 index 00000000..647f74b6 --- /dev/null +++ b/openshield @@ -0,0 +1 @@ +Subproject commit 647f74b69888891b39d3af8aa77aacfcfae83770 From dd24ce0ae354da904351e6434022fca69c91caa0 Mon Sep 17 00:00:00 2001 From: Vishnu Ajith <86302373+Vishnu2707@users.noreply.github.com> Date: Sat, 25 Apr 2026 15:07:30 +0100 Subject: [PATCH 02/15] fix: remove embedded git repo --- openshield | 1 - 1 file changed, 1 deletion(-) delete mode 160000 openshield diff --git a/openshield b/openshield deleted file mode 160000 index 647f74b6..00000000 --- a/openshield +++ /dev/null @@ -1 +0,0 @@ -Subproject commit 647f74b69888891b39d3af8aa77aacfcfae83770 From e87207471e21ffefe240647b44dd34b52f706012 Mon Sep 17 00:00:00 2001 From: Vishnu Ajith <86302373+Vishnu2707@users.noreply.github.com> Date: Sat, 25 Apr 2026 15:14:45 +0100 Subject: [PATCH 03/15] Core Structure Created --- .github/CODE_OF_CONDUCT.md | 10 ++++++++++ .github/ISSUE_TEMPLATE/bug_report.md | 15 ++++++++++++++ .github/ISSUE_TEMPLATE/new_rule.md | 23 ++++++++++++++++++++++ .github/PULL_REQUEST_TEMPLATE.md | 29 ++++++++++++++++++++++++++++ openshield | 1 + 5 files changed, 78 insertions(+) create mode 100644 .github/CODE_OF_CONDUCT.md create mode 100644 .github/ISSUE_TEMPLATE/bug_report.md create mode 100644 .github/ISSUE_TEMPLATE/new_rule.md create mode 100644 .github/PULL_REQUEST_TEMPLATE.md create mode 160000 openshield diff --git a/.github/CODE_OF_CONDUCT.md b/.github/CODE_OF_CONDUCT.md new file mode 100644 index 00000000..31cd3af9 --- /dev/null +++ b/.github/CODE_OF_CONDUCT.md @@ -0,0 +1,10 @@ +# Code of Conduct + +OpenShield is an open, welcoming project. + +- Be respectful in all interactions +- No harassment, discrimination, or offensive language +- Constructive feedback only — critique code, not people +- All contributions welcome regardless of experience level + +Violations can be reported to the maintainer directly via GitHub. \ No newline at end of file diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 00000000..086751fc --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,15 @@ +--- +name: Bug report +about: Something is broken +labels: bug +--- + +## What happened? + +## What did you expect? + +## Steps to reproduce? + +## Environment +- Python version: +- Azure SDK version: \ No newline at end of file diff --git a/.github/ISSUE_TEMPLATE/new_rule.md b/.github/ISSUE_TEMPLATE/new_rule.md new file mode 100644 index 00000000..ab8c850e --- /dev/null +++ b/.github/ISSUE_TEMPLATE/new_rule.md @@ -0,0 +1,23 @@ +--- +name: New scan rule +about: Propose a new Azure misconfiguration rule +labels: new-rule, good-first-issue +--- + +## Rule proposal + +**Rule ID:** AZ-XXX-000 +**Rule name:** +**Severity:** HIGH / MEDIUM / LOW +**Category:** Storage / Network / Identity / Database / Compute + +## What misconfiguration does it detect? + +## Why is it a security risk? + +## Which frameworks does it map to? +- CIS: +- NIST: +- ISO 27001: + +## Remediation (how to fix it)? \ No newline at end of file diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 00000000..af474c80 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,29 @@ +## What does this PR do? + + +## Type of change +- [ ] New scan rule +- [ ] Remediation playbook +- [ ] Bug fix +- [ ] Frontend component +- [ ] API endpoint +- [ ] Documentation + +## Rule details (if applicable) +- Rule ID: AZ-XXX-000 +- Severity: HIGH / MEDIUM / LOW +- Category: Storage / Network / Identity / Database / Compute +- Frameworks mapped: CIS / NIST / ISO 27001 + +## Testing +- [ ] Tested against a real Azure free trial subscription +- [ ] Returns correct JSON output +- [ ] No hardcoded credentials or secrets + +## Related issue +Closes # + +## Checklist +- [ ] My code follows the rule template in CONTRIBUTING.md +- [ ] I have not committed any real Azure credentials +- [ ] My branch name follows the convention: feat/description \ No newline at end of file diff --git a/openshield b/openshield new file mode 160000 index 00000000..647f74b6 --- /dev/null +++ b/openshield @@ -0,0 +1 @@ +Subproject commit 647f74b69888891b39d3af8aa77aacfcfae83770 From ee773771c6f5cec3788f6b76782ec680708f432d Mon Sep 17 00:00:00 2001 From: Vishnu Ajith <86302373+Vishnu2707@users.noreply.github.com> Date: Sat, 25 Apr 2026 15:45:00 +0100 Subject: [PATCH 04/15] =?UTF-8?q?feat:=20build=20complete=20core=20?= =?UTF-8?q?=E2=80=94=20scanner=20engine,=2010=20rules,=20API,=20playbooks,?= =?UTF-8?q?=20compliance=20mappings,=20docs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- api/__init__.py | 0 api/app.py | 127 +++++++ api/models/__init__.py | 0 api/models/finding.py | 311 ++++++++++++++++++ api/routes/__init__.py | 0 api/routes/compliance.py | 39 +++ api/routes/findings.py | 44 +++ api/routes/scans.py | 60 ++++ api/routes/score.py | 27 ++ .../frameworks/cis_azure_benchmark.json | 57 ++++ compliance/frameworks/iso27001.json | 57 ++++ compliance/frameworks/nist_csf.json | 57 ++++ docs/adding-a-rule.md | 216 ++++++++++++ docs/architecture.md | 187 +++++++++++ docs/azure-setup.md | 205 ++++++++++++ openshield | 1 - playbooks/cli/fix_az_cmp_001.sh | 33 ++ playbooks/cli/fix_az_db_001.sh | 27 ++ playbooks/cli/fix_az_db_002.sh | 36 ++ playbooks/cli/fix_az_idn_001.sh | 45 +++ playbooks/cli/fix_az_idn_002.sh | 69 ++++ playbooks/cli/fix_az_kv_001.sh | 39 +++ playbooks/cli/fix_az_net_001.sh | 49 +++ playbooks/cli/fix_az_net_002.sh | 48 +++ playbooks/cli/fix_az_stor_001.sh | 24 ++ playbooks/cli/fix_az_stor_002.sh | 24 ++ requirements.txt | 16 + scanner/__init__.py | 0 scanner/azure_client.py | 188 +++++++++++ scanner/engine.py | 110 +++++++ scanner/rules/__init__.py | 0 scanner/rules/az_cmp_001.py | 77 +++++ scanner/rules/az_db_001.py | 48 +++ scanner/rules/az_db_002.py | 60 ++++ scanner/rules/az_idn_001.py | 57 ++++ scanner/rules/az_idn_002.py | 84 +++++ scanner/rules/az_kv_001.py | 55 ++++ scanner/rules/az_net_001.py | 68 ++++ scanner/rules/az_net_002.py | 69 ++++ scanner/rules/az_stor_001.py | 42 +++ scanner/rules/az_stor_002.py | 42 +++ 41 files changed, 2697 insertions(+), 1 deletion(-) create mode 100644 api/__init__.py create mode 100644 api/app.py create mode 100644 api/models/__init__.py create mode 100644 api/models/finding.py create mode 100644 api/routes/__init__.py create mode 100644 api/routes/compliance.py create mode 100644 api/routes/findings.py create mode 100644 api/routes/scans.py create mode 100644 api/routes/score.py create mode 100644 compliance/frameworks/cis_azure_benchmark.json create mode 100644 compliance/frameworks/iso27001.json create mode 100644 compliance/frameworks/nist_csf.json create mode 100644 docs/adding-a-rule.md create mode 100644 docs/architecture.md create mode 100644 docs/azure-setup.md delete mode 160000 openshield create mode 100755 playbooks/cli/fix_az_cmp_001.sh create mode 100755 playbooks/cli/fix_az_db_001.sh create mode 100755 playbooks/cli/fix_az_db_002.sh create mode 100755 playbooks/cli/fix_az_idn_001.sh create mode 100755 playbooks/cli/fix_az_idn_002.sh create mode 100755 playbooks/cli/fix_az_kv_001.sh create mode 100755 playbooks/cli/fix_az_net_001.sh create mode 100755 playbooks/cli/fix_az_net_002.sh create mode 100755 playbooks/cli/fix_az_stor_001.sh create mode 100755 playbooks/cli/fix_az_stor_002.sh create mode 100644 requirements.txt create mode 100644 scanner/__init__.py create mode 100644 scanner/rules/__init__.py create mode 100644 scanner/rules/az_cmp_001.py create mode 100644 scanner/rules/az_db_001.py create mode 100644 scanner/rules/az_db_002.py create mode 100644 scanner/rules/az_idn_001.py create mode 100644 scanner/rules/az_idn_002.py create mode 100644 scanner/rules/az_kv_001.py create mode 100644 scanner/rules/az_net_001.py create mode 100644 scanner/rules/az_net_002.py create mode 100644 scanner/rules/az_stor_001.py create mode 100644 scanner/rules/az_stor_002.py diff --git a/api/__init__.py b/api/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/api/app.py b/api/app.py new file mode 100644 index 00000000..b36605f4 --- /dev/null +++ b/api/app.py @@ -0,0 +1,127 @@ +"""Flask application factory for the OpenShield REST API.""" + +import logging +import os + +import jwt +from dotenv import load_dotenv +from flask import Flask, g, jsonify, request +from flask_cors import CORS + +load_dotenv() + +logging.basicConfig( + level=logging.INFO, + format="%(asctime)s %(levelname)s %(name)s: %(message)s", +) +logger = logging.getLogger(__name__) + +# Paths that do not require a JWT token +_PUBLIC_PATHS = {"/health", "/"} + + +def create_app() -> Flask: + """Create and configure the Flask application. + + Returns a fully wired Flask app with: + - CORS enabled for all origins + - JWT authentication middleware on all non-public routes + - Blueprints for findings, scans, score, and compliance + - JSON error handlers for 400, 401, 403, 404, and 500 + """ + app = Flask(__name__) + app.config["JWT_SECRET"] = os.environ.get("JWT_SECRET", "change-me-in-production") + + # ------------------------------------------------------------------ # + # CORS # + # ------------------------------------------------------------------ # + CORS(app, resources={r"/api/*": {"origins": "*"}}) + + # ------------------------------------------------------------------ # + # JWT middleware # + # ------------------------------------------------------------------ # + + @app.before_request + def verify_jwt() -> None: + """Validate the Bearer token on every non-public, non-OPTIONS request.""" + if request.method == "OPTIONS": + return None + if request.path in _PUBLIC_PATHS: + return None + + auth = request.headers.get("Authorization", "") + if not auth.startswith("Bearer "): + return jsonify({"error": "Missing or malformed Authorization header"}), 401 + + token = auth.split(" ", 1)[1] + try: + payload = jwt.decode( + token, + app.config["JWT_SECRET"], + algorithms=["HS256"], + ) + g.user = payload + except jwt.ExpiredSignatureError: + return jsonify({"error": "Token has expired"}), 401 + except jwt.InvalidTokenError as exc: + return jsonify({"error": f"Invalid token: {exc}"}), 401 + + return None + + # ------------------------------------------------------------------ # + # Blueprints # + # ------------------------------------------------------------------ # + from api.routes.compliance import compliance_bp + from api.routes.findings import findings_bp + from api.routes.scans import scans_bp + from api.routes.score import score_bp + + app.register_blueprint(findings_bp) + app.register_blueprint(scans_bp) + app.register_blueprint(score_bp) + app.register_blueprint(compliance_bp) + + # ------------------------------------------------------------------ # + # Health check (public) # + # ------------------------------------------------------------------ # + + @app.get("/health") + def health(): + return jsonify({"status": "ok"}) + + # ------------------------------------------------------------------ # + # Error handlers # + # ------------------------------------------------------------------ # + + @app.errorhandler(400) + def bad_request(exc): + return jsonify({"error": "Bad request", "detail": str(exc)}), 400 + + @app.errorhandler(401) + def unauthorized(exc): + return jsonify({"error": "Unauthorized"}), 401 + + @app.errorhandler(403) + def forbidden(exc): + return jsonify({"error": "Forbidden"}), 403 + + @app.errorhandler(404) + def not_found(exc): + return jsonify({"error": "Not found"}), 404 + + @app.errorhandler(500) + def internal_error(exc): + logger.error("Unhandled exception: %s", exc) + return jsonify({"error": "Internal server error"}), 500 + + logger.info("OpenShield API created — %d blueprints registered", len(app.blueprints)) + return app + + +if __name__ == "__main__": + application = create_app() + application.run( + host="0.0.0.0", + port=int(os.environ.get("PORT", 5000)), + debug=os.environ.get("FLASK_DEBUG", "false").lower() == "true", + ) diff --git a/api/models/__init__.py b/api/models/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/api/models/finding.py b/api/models/finding.py new file mode 100644 index 00000000..90b86623 --- /dev/null +++ b/api/models/finding.py @@ -0,0 +1,311 @@ +"""Finding dataclass and PostgreSQL-backed DatabaseManager.""" + +import json +import logging +import os +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Dict, List, Optional + +import psycopg2 +import psycopg2.extras + +logger = logging.getLogger(__name__) + +FRAMEWORKS_DIR = Path(__file__).parent.parent.parent / "compliance" / "frameworks" + +SEVERITY_WEIGHTS = {"HIGH": 10, "MEDIUM": 5, "LOW": 2, "INFO": 0} + +FRAMEWORK_FILE_MAP = { + "cis": "cis_azure_benchmark.json", + "nist": "nist_csf.json", + "iso27001": "iso27001.json", +} + + +@dataclass +class Finding: + """Represents a single security misconfiguration finding.""" + + rule_id: str + rule_name: str + severity: str + category: str + resource_id: str + resource_name: str + resource_type: str + description: str + remediation: str + frameworks: Dict[str, str] + detected_at: str + scan_id: Optional[str] = None + playbook: Optional[str] = None + metadata: Dict[str, Any] = field(default_factory=dict) + id: Optional[int] = None + + def to_dict(self) -> Dict[str, Any]: + return { + "id": self.id, + "rule_id": self.rule_id, + "rule_name": self.rule_name, + "severity": self.severity, + "category": self.category, + "resource_id": self.resource_id, + "resource_name": self.resource_name, + "resource_type": self.resource_type, + "description": self.description, + "remediation": self.remediation, + "frameworks": self.frameworks, + "detected_at": self.detected_at, + "scan_id": self.scan_id, + "playbook": self.playbook, + "metadata": self.metadata, + } + + +class DatabaseManager: + """Manages PostgreSQL persistence for scans, findings, and scoring. + + All public methods open a new connection on first use. Call connect() + explicitly if you want to pre-warm the connection. + """ + + def __init__(self, dsn: Optional[str] = None) -> None: + self.dsn = dsn or os.environ["DATABASE_URL"] + self.conn: Optional[Any] = None + + # ------------------------------------------------------------------ # + # Connection # + # ------------------------------------------------------------------ # + + def connect(self) -> None: + """Open a persistent database connection.""" + self.conn = psycopg2.connect(self.dsn) + self.conn.autocommit = False + logger.info("Database connection established") + + def _get_conn(self) -> Any: + if self.conn is None or self.conn.closed: + self.connect() + return self.conn + + # ------------------------------------------------------------------ # + # Schema # + # ------------------------------------------------------------------ # + + def create_tables(self) -> None: + """Create the findings, scans, and rules tables if they do not exist.""" + conn = self._get_conn() + with conn.cursor() as cur: + cur.execute(""" + CREATE TABLE IF NOT EXISTS scans ( + scan_id UUID PRIMARY KEY, + subscription_id TEXT NOT NULL, + started_at TIMESTAMPTZ NOT NULL, + completed_at TIMESTAMPTZ, + total_findings INTEGER DEFAULT 0 + ); + """) + cur.execute(""" + CREATE TABLE IF NOT EXISTS findings ( + id SERIAL PRIMARY KEY, + scan_id UUID REFERENCES scans(scan_id), + rule_id TEXT NOT NULL, + rule_name TEXT NOT NULL, + severity TEXT NOT NULL, + category TEXT, + resource_id TEXT, + resource_name TEXT, + resource_type TEXT, + description TEXT, + remediation TEXT, + playbook TEXT, + frameworks JSONB, + metadata JSONB, + detected_at TIMESTAMPTZ NOT NULL + ); + """) + cur.execute(""" + CREATE INDEX IF NOT EXISTS idx_findings_scan_id + ON findings(scan_id); + CREATE INDEX IF NOT EXISTS idx_findings_severity + ON findings(severity); + CREATE INDEX IF NOT EXISTS idx_findings_rule_id + ON findings(rule_id); + """) + conn.commit() + logger.info("Database tables created / verified") + + # ------------------------------------------------------------------ # + # Write # + # ------------------------------------------------------------------ # + + def save_scan(self, scan_result: Dict[str, Any]) -> None: + """Persist a full scan result (scan header + all findings).""" + conn = self._get_conn() + with conn.cursor() as cur: + cur.execute( + """ + INSERT INTO scans (scan_id, subscription_id, started_at, completed_at, total_findings) + VALUES (%s, %s, %s, %s, %s) + ON CONFLICT (scan_id) DO NOTHING + """, + ( + scan_result["scan_id"], + scan_result["subscription_id"], + scan_result["started_at"], + scan_result["completed_at"], + scan_result["total_findings"], + ), + ) + for f in scan_result.get("findings", []): + cur.execute( + """ + INSERT INTO findings + (scan_id, rule_id, rule_name, severity, category, + resource_id, resource_name, resource_type, + description, remediation, playbook, + frameworks, metadata, detected_at) + VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s) + """, + ( + f.get("scan_id"), + f.get("rule_id"), + f.get("rule_name"), + f.get("severity"), + f.get("category"), + f.get("resource_id"), + f.get("resource_name"), + f.get("resource_type"), + f.get("description"), + f.get("remediation"), + f.get("playbook"), + json.dumps(f.get("frameworks", {})), + json.dumps(f.get("metadata", {})), + f.get("detected_at"), + ), + ) + conn.commit() + logger.info("Saved scan %s with %d findings", scan_result["scan_id"], scan_result["total_findings"]) + + # ------------------------------------------------------------------ # + # Read # + # ------------------------------------------------------------------ # + + def get_findings(self, filters: Optional[Dict[str, Any]] = None) -> List[Dict[str, Any]]: + """Return findings, optionally filtered by severity, category, or rule_id.""" + filters = filters or {} + clauses: List[str] = [] + params: List[Any] = [] + + if "severity" in filters: + clauses.append("severity = %s") + params.append(filters["severity"].upper()) + if "category" in filters: + clauses.append("LOWER(category) = LOWER(%s)") + params.append(filters["category"]) + if "rule_id" in filters: + clauses.append("rule_id = %s") + params.append(filters["rule_id"]) + if "scan_id" in filters: + clauses.append("scan_id = %s") + params.append(filters["scan_id"]) + + where = "WHERE " + " AND ".join(clauses) if clauses else "" + sql = f"SELECT * FROM findings {where} ORDER BY detected_at DESC LIMIT 1000" + + conn = self._get_conn() + with conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor) as cur: + cur.execute(sql, params) + return [dict(row) for row in cur.fetchall()] + + def get_finding_by_id(self, finding_id: int) -> Optional[Dict[str, Any]]: + """Return a single finding by its integer primary key.""" + conn = self._get_conn() + with conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor) as cur: + cur.execute("SELECT * FROM findings WHERE id = %s", (finding_id,)) + row = cur.fetchone() + return dict(row) if row else None + + def get_scans(self) -> List[Dict[str, Any]]: + """Return all scan records ordered by most recent first.""" + conn = self._get_conn() + with conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor) as cur: + cur.execute("SELECT * FROM scans ORDER BY started_at DESC LIMIT 100") + return [dict(row) for row in cur.fetchall()] + + # ------------------------------------------------------------------ # + # Scoring # + # ------------------------------------------------------------------ # + + def get_score(self) -> int: + """Return a 0–100 security posture score based on open findings. + + HIGH findings deduct 10 points each, MEDIUM 5, LOW 2. + Score floors at 0. + """ + conn = self._get_conn() + with conn.cursor() as cur: + cur.execute( + "SELECT severity, COUNT(*) FROM findings GROUP BY severity" + ) + rows = cur.fetchall() + + deduction = sum( + SEVERITY_WEIGHTS.get(sev.upper(), 0) * count for sev, count in rows + ) + return max(0, 100 - deduction) + + def get_compliance_score(self, framework: str) -> Dict[str, Any]: + """Return pass/fail breakdown against a compliance framework. + + Args: + framework: One of 'cis', 'nist', or 'iso27001'. + + Returns: + dict with keys: framework, total_controls, passed, failed, + score_percent, controls (list of control detail objects). + """ + filename = FRAMEWORK_FILE_MAP.get(framework.lower()) + if not filename: + return {"error": f"Unknown framework: {framework}"} + + framework_path = FRAMEWORKS_DIR / filename + if not framework_path.exists(): + return {"error": f"Framework file not found: {filename}"} + + with open(framework_path) as fh: + framework_data = json.load(fh) + + controls = framework_data.get("controls", {}) + + # Get rule IDs that have at least one finding + conn = self._get_conn() + with conn.cursor() as cur: + cur.execute("SELECT DISTINCT rule_id FROM findings") + failed_rule_ids = {row[0] for row in cur.fetchall()} + + results = [] + for rule_id, control in controls.items(): + status = "FAIL" if rule_id in failed_rule_ids else "PASS" + results.append({ + "rule_id": rule_id, + "control_id": control["control_id"], + "control_name": control["control_name"], + "status": status, + }) + + total = len(results) + passed = sum(1 for r in results if r["status"] == "PASS") + failed = total - passed + score_pct = round((passed / total) * 100) if total else 0 + + return { + "framework": framework_data.get("framework"), + "version": framework_data.get("version"), + "total_controls": total, + "passed": passed, + "failed": failed, + "score_percent": score_pct, + "controls": results, + } diff --git a/api/routes/__init__.py b/api/routes/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/api/routes/compliance.py b/api/routes/compliance.py new file mode 100644 index 00000000..e3b68a23 --- /dev/null +++ b/api/routes/compliance.py @@ -0,0 +1,39 @@ +"""Compliance routes: framework-specific posture breakdown.""" + +import os +from flask import Blueprint, jsonify + +from api.models.finding import DatabaseManager + +compliance_bp = Blueprint("compliance", __name__) + +SUPPORTED_FRAMEWORKS = ("cis", "nist", "iso27001") + + +def _get_db() -> DatabaseManager: + db = DatabaseManager(os.environ["DATABASE_URL"]) + db.connect() + return db + + +@compliance_bp.get("/api/compliance/") +def get_compliance(framework: str): + """Return pass/fail compliance breakdown for a framework. + + Supported frameworks: cis, nist, iso27001 + + Returns control-level pass/fail status mapped to current open findings. + """ + if framework.lower() not in SUPPORTED_FRAMEWORKS: + return jsonify({ + "error": f"Unknown framework '{framework}'", + "supported": list(SUPPORTED_FRAMEWORKS), + }), 400 + + db = _get_db() + result = db.get_compliance_score(framework.lower()) + + if "error" in result: + return jsonify(result), 500 + + return jsonify(result) diff --git a/api/routes/findings.py b/api/routes/findings.py new file mode 100644 index 00000000..fb8d7550 --- /dev/null +++ b/api/routes/findings.py @@ -0,0 +1,44 @@ +"""Findings routes: list and retrieve individual findings.""" + +import os +from flask import Blueprint, jsonify, request + +from api.models.finding import DatabaseManager + +findings_bp = Blueprint("findings", __name__) + + +def _get_db() -> DatabaseManager: + db = DatabaseManager(os.environ["DATABASE_URL"]) + db.connect() + return db + + +@findings_bp.get("/api/findings") +def list_findings(): + """Return findings, optionally filtered by severity, category, or rule_id. + + Query parameters: + severity — HIGH | MEDIUM | LOW | INFO + category — Storage | Network | Identity | Database | Compute | KeyVault + rule_id — e.g. AZ-STOR-001 + scan_id — UUID of a specific scan + """ + filters = { + k: v + for k, v in request.args.items() + if k in ("severity", "category", "rule_id", "scan_id") + } + db = _get_db() + findings = db.get_findings(filters) + return jsonify({"count": len(findings), "findings": findings}) + + +@findings_bp.get("/api/findings/") +def get_finding(finding_id: int): + """Return a single finding by its integer ID.""" + db = _get_db() + finding = db.get_finding_by_id(finding_id) + if not finding: + return jsonify({"error": "Finding not found"}), 404 + return jsonify(finding) diff --git a/api/routes/scans.py b/api/routes/scans.py new file mode 100644 index 00000000..85612a44 --- /dev/null +++ b/api/routes/scans.py @@ -0,0 +1,60 @@ +"""Scan routes: list historical scans and trigger new ones.""" + +import logging +import os +from flask import Blueprint, jsonify, request + +from api.models.finding import DatabaseManager + +scans_bp = Blueprint("scans", __name__) +logger = logging.getLogger(__name__) + + +def _get_db() -> DatabaseManager: + db = DatabaseManager(os.environ["DATABASE_URL"]) + db.connect() + return db + + +@scans_bp.get("/api/scans") +def list_scans(): + """Return all historical scan results ordered by most recent first.""" + db = _get_db() + scans = db.get_scans() + return jsonify({"count": len(scans), "scans": scans}) + + +@scans_bp.post("/api/scans/trigger") +def trigger_scan(): + """Trigger a synchronous scan against the configured subscription. + + Accepts an optional JSON body with ``subscription_id``. Falls back to the + ``AZURE_SUBSCRIPTION_ID`` environment variable if not provided. + + Note: For production use, replace this with an async task queue (e.g. + Celery or Azure Functions) to avoid request timeouts on large subscriptions. + """ + from scanner.engine import ScanEngine # deferred to avoid import at startup + + body = request.get_json(silent=True) or {} + subscription_id = body.get("subscription_id") or os.environ.get( + "AZURE_SUBSCRIPTION_ID" + ) + + if not subscription_id: + return jsonify({"error": "subscription_id is required"}), 400 + + logger.info("Scan triggered for subscription %s", subscription_id) + + try: + engine = ScanEngine(subscription_id) + result = engine.run_scan() + except Exception as exc: + logger.error("Scan failed: %s", exc) + return jsonify({"error": "Scan failed", "detail": str(exc)}), 500 + + db = _get_db() + db.create_tables() + db.save_scan(result) + + return jsonify(result), 201 diff --git a/api/routes/score.py b/api/routes/score.py new file mode 100644 index 00000000..b7317ee2 --- /dev/null +++ b/api/routes/score.py @@ -0,0 +1,27 @@ +"""Score route: overall security posture score.""" + +import os +from flask import Blueprint, jsonify + +from api.models.finding import DatabaseManager + +score_bp = Blueprint("score", __name__) + + +def _get_db() -> DatabaseManager: + db = DatabaseManager(os.environ["DATABASE_URL"]) + db.connect() + return db + + +@score_bp.get("/api/score") +def get_score(): + """Return the overall security posture score (0–100). + + Score calculation: + Starts at 100. Deducts 10 per HIGH finding, 5 per MEDIUM, 2 per LOW. + Floors at 0. + """ + db = _get_db() + score = db.get_score() + return jsonify({"score": score, "max_score": 100}) diff --git a/compliance/frameworks/cis_azure_benchmark.json b/compliance/frameworks/cis_azure_benchmark.json new file mode 100644 index 00000000..c575a6fb --- /dev/null +++ b/compliance/frameworks/cis_azure_benchmark.json @@ -0,0 +1,57 @@ +{ + "framework": "CIS Microsoft Azure Foundations Benchmark", + "version": "2.0.0", + "published": "2023-02", + "controls": { + "AZ-STOR-001": { + "control_id": "3.5", + "control_name": "Ensure that 'Public access level' is set to Private for blob containers", + "description": "Disabling public access level for blob containers prevents anonymous unauthenticated access to Azure Blob storage. This setting eliminates the risk of inadvertent or unauthorized public data exposure." + }, + "AZ-STOR-002": { + "control_id": "3.1", + "control_name": "Ensure that 'Secure transfer required' is set to 'Enabled'", + "description": "Enabling 'Secure transfer required' on a storage account ensures that all requests made to the storage account use HTTPS. Any requests using HTTP are rejected, protecting data in transit from eavesdropping and man-in-the-middle attacks." + }, + "AZ-NET-001": { + "control_id": "6.2", + "control_name": "Ensure that SSH access from the Internet is evaluated and restricted", + "description": "Network security groups should not allow unrestricted SSH access from the internet. Restricting inbound SSH access reduces attack surface and prevents unauthorized access attempts, brute-force attacks, and exploitation of SSH service vulnerabilities." + }, + "AZ-NET-002": { + "control_id": "6.3", + "control_name": "Ensure that RDP access from the Internet is evaluated and restricted", + "description": "Network security groups should not permit unrestricted inbound RDP from the internet. Open RDP ports are a leading cause of ransomware infections and credential-based attacks. Access should be restricted to specific trusted IP ranges or removed in favour of Azure Bastion." + }, + "AZ-IDN-001": { + "control_id": "1.23", + "control_name": "Ensure That No Custom Subscription Owner Roles Are Created", + "description": "Service principals or custom roles should not be assigned the Owner role at subscription scope. The Owner role grants full control including the ability to modify access controls. Assignment should follow the principle of least privilege." + }, + "AZ-IDN-002": { + "control_id": "1.2.4", + "control_name": "Ensure that 'Multi-Factor Authentication Status' is 'Enabled' for all Privileged Users", + "description": "Multi-Factor Authentication requires an individual to present a minimum of two separate forms of authentication before access is granted. MFA should be enforced for all users with administrative privileges via Conditional Access policies." + }, + "AZ-DB-001": { + "control_id": "4.3.1", + "control_name": "Ensure 'Allow access to Azure services' for PostgreSQL Database Server is disabled", + "description": "Disabling public network access on PostgreSQL Database Server prevents public access and reduces the attack surface. Access should be restricted to private networks using VNet service endpoints or private endpoints." + }, + "AZ-DB-002": { + "control_id": "4.1.3", + "control_name": "Ensure that 'Auditing' Retention is 'greater than 90 days' for SQL servers", + "description": "SQL Server audit logs must be enabled and retained for a minimum of 90 days. Enabling auditing provides a record of database events that can be used to detect threats, investigate incidents, and demonstrate compliance." + }, + "AZ-CMP-001": { + "control_id": "7.2", + "control_name": "Ensure that 'OS disk' are encrypted", + "description": "Virtual machines that are reachable from the internet should have Network Security Groups attached to their network interfaces to control and restrict inbound and outbound traffic, reducing the attack surface." + }, + "AZ-KV-001": { + "control_id": "8.5", + "control_name": "Ensure the Key Vault is Recoverable", + "description": "Azure Key Vault soft delete should be enabled on all Key Vaults. The soft delete feature allows recovery of deleted vaults and vault objects (keys, secrets, certificates) for a configurable retention period (7–90 days), protecting against accidental or malicious deletion." + } + } +} diff --git a/compliance/frameworks/iso27001.json b/compliance/frameworks/iso27001.json new file mode 100644 index 00000000..42837904 --- /dev/null +++ b/compliance/frameworks/iso27001.json @@ -0,0 +1,57 @@ +{ + "framework": "ISO/IEC 27001:2013", + "version": "2013", + "published": "2013-10", + "controls": { + "AZ-STOR-001": { + "control_id": "A.9.4.1", + "control_name": "Information access restriction", + "description": "Access to information and application system functions shall be restricted in accordance with the access control policy. Enabling public blob access on storage accounts removes all access restrictions and allows any internet user to read stored data without authentication, directly violating this control." + }, + "AZ-STOR-002": { + "control_id": "A.10.1.1", + "control_name": "Policy on the use of cryptographic controls", + "description": "A policy on the use of cryptographic controls for protection of information shall be developed and implemented. Storage accounts transmitting data over HTTP do not apply encryption in transit, violating the organisation's cryptographic control policy requirement to protect data confidentiality." + }, + "AZ-NET-001": { + "control_id": "A.13.1.1", + "control_name": "Network controls", + "description": "Networks shall be managed and controlled to protect information in systems and applications. NSGs permitting unrestricted SSH from the internet represent a failure of network access control, exposing systems to direct internet-based attack with no network-layer filtering." + }, + "AZ-NET-002": { + "control_id": "A.13.1.1", + "control_name": "Network controls", + "description": "Networks shall be managed and controlled to protect information in systems and applications. NSGs permitting unrestricted RDP from the internet represent a critical network control failure, as RDP is the most commonly exploited protocol for ransomware initial access." + }, + "AZ-IDN-001": { + "control_id": "A.9.2.3", + "control_name": "Management of privileged access rights", + "description": "The allocation and use of privileged access rights shall be restricted and controlled. Assigning the Owner role to service principals at subscription scope grants excessive privileged access rights beyond operational requirements, violating the principle of least privilege and privileged access management controls." + }, + "AZ-IDN-002": { + "control_id": "A.9.4.2", + "control_name": "Secure log-on procedures", + "description": "Where required by the access control policy, access to systems and applications shall be controlled by a secure log-on procedure. Multi-factor authentication is a required component of secure log-on for privileged accounts. Absence of MFA enforcement via Conditional Access violates this control." + }, + "AZ-DB-001": { + "control_id": "A.13.1.1", + "control_name": "Network controls", + "description": "Networks shall be managed and controlled to protect information in systems and applications. Database servers with public network access enabled lack the network-level isolation required to protect sensitive data from direct internet exposure and attack." + }, + "AZ-DB-002": { + "control_id": "A.12.4.1", + "control_name": "Event logging", + "description": "Event logs recording user activities, exceptions, faults and information security events shall be produced, kept and regularly reviewed. Disabling SQL Server auditing means that database access events, failed logins, and schema changes are not logged, making incident detection and forensic investigation impossible." + }, + "AZ-CMP-001": { + "control_id": "A.13.1.1", + "control_name": "Network controls", + "description": "Networks shall be managed and controlled to protect information in systems and applications. Virtual machines with public IPs and no Network Security Group have no network-layer access controls, exposing all ports and services to the internet without any filtering." + }, + "AZ-KV-001": { + "control_id": "A.17.2.1", + "control_name": "Availability of information processing facilities", + "description": "Information processing facilities shall be implemented with sufficient redundancy to meet availability requirements. Disabling soft delete on Key Vault removes the ability to recover deleted secrets, keys, and certificates, creating a single point of failure for critical cryptographic material and violating availability and recovery requirements." + } + } +} diff --git a/compliance/frameworks/nist_csf.json b/compliance/frameworks/nist_csf.json new file mode 100644 index 00000000..869bc5aa --- /dev/null +++ b/compliance/frameworks/nist_csf.json @@ -0,0 +1,57 @@ +{ + "framework": "NIST Cybersecurity Framework", + "version": "1.1", + "published": "2018-04", + "controls": { + "AZ-STOR-001": { + "control_id": "PR.AC-3", + "control_name": "Remote access is managed", + "description": "Remote access to data assets is controlled. Unauthenticated public blob access on storage accounts violates access management controls by allowing anonymous access to potentially sensitive data without any form of authentication or authorisation." + }, + "AZ-STOR-002": { + "control_id": "PR.DS-2", + "control_name": "Data-in-transit is protected", + "description": "Data in transit is protected to prevent interception and tampering. Storage accounts that allow HTTP traffic transmit data in plaintext, violating the requirement to protect data in transit through encryption (TLS)." + }, + "AZ-NET-001": { + "control_id": "PR.AC-3", + "control_name": "Remote access is managed", + "description": "Remote access to systems must be controlled. Allowing unrestricted SSH access from the internet bypasses access management controls and exposes systems to unauthorised remote access, brute-force attacks, and exploitation." + }, + "AZ-NET-002": { + "control_id": "PR.AC-3", + "control_name": "Remote access is managed", + "description": "Remote access to systems must be managed. Allowing unrestricted RDP access from the internet bypasses access management controls and is a primary vector for ransomware delivery and credential-based attacks on Windows systems." + }, + "AZ-IDN-001": { + "control_id": "PR.AC-4", + "control_name": "Access permissions and authorisations are managed, incorporating the principles of least privilege and separation of duties", + "description": "Access to cloud resources should follow the principle of least privilege. Assigning the Owner role to service principals at subscription scope grants excessive permissions that violate least-privilege and separation-of-duties requirements." + }, + "AZ-IDN-002": { + "control_id": "PR.AC-1", + "control_name": "Identities and credentials are issued, managed, verified, revoked, and audited for authorised devices, users and processes", + "description": "Credentials must be managed to ensure only authorised parties can authenticate. Without MFA enforcement, a single compromised password grants full access to administrator accounts, undermining identity management controls." + }, + "AZ-DB-001": { + "control_id": "PR.AC-3", + "control_name": "Remote access is managed", + "description": "Database servers should not be reachable from the public internet without restriction. Public network access to PostgreSQL servers removes the network-based access control layer, exposing the database to direct internet-based attacks." + }, + "AZ-DB-002": { + "control_id": "DE.CM-7", + "control_name": "Monitoring for unauthorised personnel, connections, devices, and software is performed", + "description": "Audit logging on SQL servers enables detection of unauthorised access attempts, privilege escalation, and suspicious database activity. Without auditing enabled, security events go undetected and incident investigation is severely limited." + }, + "AZ-CMP-001": { + "control_id": "PR.AC-3", + "control_name": "Remote access is managed", + "description": "Virtual machines accessible from the internet must have compensating network controls. A VM with a public IP and no NSG has all ports exposed to the internet with no filtering, violating remote access management requirements." + }, + "AZ-KV-001": { + "control_id": "PR.IP-4", + "control_name": "Backups of information are conducted, maintained, and tested", + "description": "Key material in Azure Key Vault must be recoverable after accidental or malicious deletion. Soft delete provides a recoverable state for secrets, keys, and certificates, supporting backup and recovery requirements for critical cryptographic material." + } + } +} diff --git a/docs/adding-a-rule.md b/docs/adding-a-rule.md new file mode 100644 index 00000000..2d60b2b5 --- /dev/null +++ b/docs/adding-a-rule.md @@ -0,0 +1,216 @@ +# Adding a New Scan Rule + +This is the fastest way to contribute to OpenShield. You can write, test, and submit a new rule in under 30 minutes. + +--- + +## The Rule Template + +Create a new file in `scanner/rules/`. The filename should match your rule ID in lowercase with underscores: + +``` +scanner/rules/az_stor_001.py ← for rule AZ-STOR-001 +``` + +Every rule file must have this exact structure: + +```python +"""AZ-XXXX-000: One-line description of what this rule detects.""" + +from typing import Any, Dict, List + +# ── Required module-level constants ───────────────────────────────────────── + +RULE_ID = "AZ-XXXX-000" # Unique ID. Check existing rules to avoid clashes. +RULE_NAME = "Human-readable name" # Shown in the dashboard and reports. +SEVERITY = "HIGH" # HIGH | MEDIUM | LOW | INFO +CATEGORY = "Storage" # Storage | Network | Identity | Database | Compute | KeyVault +FRAMEWORKS = { + "CIS": "3.5", # CIS Azure Benchmark control ID + "NIST": "PR.AC-3", # NIST CSF subcategory + "ISO27001": "A.9.4.1", # ISO 27001 Annex A control +} +DESCRIPTION = ( + "Explain WHY this is a security risk. One or two sentences. " + "What can an attacker do if this misconfiguration exists?" +) +REMEDIATION = ( + "Explain HOW to fix it. What setting to change, or what command to run." +) +PLAYBOOK = "playbooks/cli/fix_az_xxxx_000.sh" # path to the matching fix script + + +# ── Required scan function ─────────────────────────────────────────────────── + +def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: + """Return a list of findings. Return [] if no issues are found. + + Args: + azure_client: An AzureClient instance with all SDK clients pre-configured. + subscription_id: The Azure subscription ID being scanned. + + Returns: + A list of finding dicts. Each dict must contain the keys below. + """ + findings: List[Dict[str, Any]] = [] + + for resource in azure_client.get_storage_accounts(): # ← replace with the right method + if : + findings.append({ + "rule_id": RULE_ID, + "rule_name": RULE_NAME, + "severity": SEVERITY, + "category": CATEGORY, + "resource_id": resource.id, + "resource_name": resource.name, + "resource_type": "Microsoft.Storage/storageAccounts", # ← update + "description": DESCRIPTION, + "remediation": REMEDIATION, + "playbook": PLAYBOOK, + "frameworks": FRAMEWORKS, + }) + + return findings +``` + +--- + +## Field-by-Field Explanation + +| Field | What to write | +|---|---| +| `RULE_ID` | `AZ-[CATEGORY]-[NUMBER]`. Prefix map: STOR, NET, IDN, DB, CMP, KV. Look at existing rules for the next number. | +| `SEVERITY` | `HIGH` = direct exploitation risk, `MEDIUM` = indirect or partial risk, `LOW` = best practice, `INFO` = informational only | +| `CATEGORY` | Matches the resource type being scanned | +| `FRAMEWORKS` | Use real control IDs from each framework. Refer to `compliance/frameworks/` JSON files for examples. | +| `DESCRIPTION` | Focus on WHY it matters — what is the real-world attack scenario? | +| `REMEDIATION` | Be specific. Name the Azure Portal setting or the exact CLI flag. | +| `PLAYBOOK` | Path to the matching bash script in `playbooks/cli/`. You must create this file too. | +| `resource_type` | The full Azure resource provider type string, e.g. `Microsoft.Network/networkSecurityGroups` | + +--- + +## AzureClient Methods Available + +| Method | Returns | +|---|---| +| `azure_client.get_storage_accounts()` | List of StorageAccount objects | +| `azure_client.get_network_security_groups()` | List of NetworkSecurityGroup objects | +| `azure_client.get_virtual_machines()` | List of VirtualMachine objects | +| `azure_client.get_postgresql_servers()` | List of Server objects (PostgreSQL single-server) | +| `azure_client.get_sql_servers()` | List of Server objects (Azure SQL) | +| `azure_client.get_sql_server_auditing_policy(rg, name)` | ServerBlobAuditingPolicy or None | +| `azure_client.get_key_vaults()` | List of Vault objects (with full properties) | +| `azure_client.get_service_principals()` | List of RoleAssignment objects for service principals | +| `azure_client.get_network_interface(rg, name)` | NetworkInterface or None | +| `azure_client.get_conditional_access_policies()` | List of CA policy dicts from MS Graph | +| `azure_client.parse_resource_id(id)` | Dict with `resource_group` and `name` | + +All methods return an empty list on failure — your scan function never needs to handle SDK exceptions. + +--- + +## Write the Remediation Playbook + +Create a matching bash script in `playbooks/cli/`: + +```bash +#!/bin/bash +# OpenShield Remediation Playbook +# Rule: AZ-XXXX-000 — Your Rule Name +# Usage: ./fix_az_xxxx_000.sh +# Severity: HIGH + +set -e + +RESOURCE_GROUP=$1 +RESOURCE_NAME=$2 + +if [ -z "$RESOURCE_GROUP" ] || [ -z "$RESOURCE_NAME" ]; then + echo "Usage: $0 " + exit 1 +fi + +# The actual az CLI command to fix the issue +az update \ + --resource-group "$RESOURCE_GROUP" \ + --name "$RESOURCE_NAME" \ + -- + +echo "✅ Remediation complete for $RESOURCE_NAME" +``` + +--- + +## Test Your Rule Locally + +```bash +# 1. Set credentials +cp .env.example .env +# Fill in your Azure credentials in .env + +# 2. Load env and run your rule in isolation +python -c " +from dotenv import load_dotenv; load_dotenv() +import os +from scanner.azure_client import AzureClient +from scanner.rules import az_xxxx_000 as rule # replace with your module name + +client = AzureClient(os.environ['AZURE_SUBSCRIPTION_ID']) +findings = rule.scan(client, os.environ['AZURE_SUBSCRIPTION_ID']) +print(f'Found {len(findings)} issue(s):') +for f in findings: + print(f' [{f[\"severity\"]}] {f[\"resource_name\"]} — {f[\"rule_name\"]}') +" + +# 3. Or run the full scan engine (loads all rules) +python -c " +from dotenv import load_dotenv; load_dotenv() +import json, os +from scanner.engine import ScanEngine +engine = ScanEngine(os.environ['AZURE_SUBSCRIPTION_ID']) +result = engine.run_scan() +print(json.dumps(result, indent=2)) +" +``` + +--- + +## Update the Compliance Framework Files + +If your rule maps to controls not yet in the compliance JSON files, add entries to the relevant file(s) in `compliance/frameworks/`: + +```json +{ + "controls": { + "AZ-XXXX-000": { + "control_id": "3.7", + "control_name": "CIS control name here", + "description": "Why this control is relevant to your finding." + } + } +} +``` + +--- + +## Submit a Pull Request + +```bash +git checkout -b rule/az-xxxx-000-short-description +git add scanner/rules/az_xxxx_000.py playbooks/cli/fix_az_xxxx_000.sh +git commit -m "feat: add rule AZ-XXXX-000 — short description" +git push origin rule/az-xxxx-000-short-description +``` + +Then open a PR. Use the PR template — it will ask you for the rule ID, severity, and which frameworks you mapped. A maintainer will review within 48 hours. + +--- + +## Common Mistakes to Avoid + +- **Rule ID clash**: always check `scanner/rules/` for existing IDs before numbering your rule. +- **Missing playbook**: every rule must have a matching `playbooks/cli/fix_*.sh` file. +- **Hardcoded subscription ID**: use the `subscription_id` parameter passed to `scan()`, never hardcode. +- **Exceptions crashing the scan**: the engine catches unhandled exceptions per rule, but write defensively — use `getattr(obj, "field", default)` for optional SDK attributes. +- **Empty `frameworks` dict**: always populate all three keys (CIS, NIST, ISO27001) even if you map to `"N/A"`. diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 00000000..5217407d --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,187 @@ +# OpenShield Architecture + +## Overview + +OpenShield is a modular, open source Cloud Security Posture Management (CSPM) platform for Azure. It continuously scans your Azure subscription against a library of security rules, maps every finding to compliance frameworks (CIS, NIST CSF, ISO 27001), and exposes results via a REST API consumed by a React dashboard. + +--- + +## High-Level Architecture + +``` +┌──────────────────────────────────────────────────────────────────┐ +│ React Dashboard │ +│ (Azure Static Web Apps — Free tier) │ +└────────────────────────────┬─────────────────────────────────────┘ + │ HTTPS / JWT +┌────────────────────────────▼─────────────────────────────────────┐ +│ Flask REST API (api/) │ +│ │ +│ GET /api/findings GET /api/score │ +│ GET /api/findings/ GET /api/compliance/ │ +│ GET /api/scans POST /api/scans/trigger │ +└───────────┬──────────────────────────────────┬───────────────────┘ + │ │ +┌───────────▼──────────────┐ ┌───────────────▼───────────────────┐ +│ Scanner Engine │ │ Compliance Mapper │ +│ (scanner/) │ │ (compliance/frameworks/) │ +│ │ │ │ +│ ScanEngine │ │ cis_azure_benchmark.json │ +│ └── load_rules() │ │ nist_csf.json │ +│ └── run_scan() │ │ iso27001.json │ +└───────────┬───────────────┘ └────────────────────────────────────┘ + │ +┌───────────▼──────────────────────────────────────────────────────┐ +│ Rule Modules (scanner/rules/) │ +│ │ +│ az_stor_001.py az_net_001.py az_idn_001.py az_db_001.py │ +│ az_stor_002.py az_net_002.py az_idn_002.py az_db_002.py │ +│ az_cmp_001.py az_kv_001.py │ +└───────────┬───────────────────────────────────────────────────────┘ + │ calls +┌───────────▼──────────────────────────────────────────────────────┐ +│ AzureClient (scanner/azure_client.py) │ +│ │ +│ DefaultAzureCredential │ +│ StorageManagementClient NetworkManagementClient │ +│ ComputeManagementClient PostgreSQLManagementClient │ +│ SqlManagementClient KeyVaultManagementClient │ +│ AuthorizationManagementClient MS Graph REST API │ +└───────────┬───────────────────────────────────────────────────────┘ + │ Azure SDK calls +┌───────────▼──────────────────────────────────────────────────────┐ +│ Azure Subscription (target) │ +└──────────────────────────────────────────────────────────────────┘ + │ +┌───────────▼──────────────────────────────────────────────────────┐ +│ PostgreSQL Database │ +│ (findings, scans, rules tables) │ +└──────────────────────────────────────────────────────────────────┘ +``` + +--- + +## How the Scanner Works + +### 1. Initialisation + +```python +engine = ScanEngine(subscription_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx") +``` + +`ScanEngine.__init__` creates an `AzureClient` using `DefaultAzureCredential`, which automatically resolves credentials from (in order): environment variables, managed identity, Azure CLI, or VS Code login. + +### 2. Rule Loading + +```python +engine.load_rules() +``` + +`load_rules()` iterates over every `*.py` file in `scanner/rules/` that does not start with `_`. It uses Python's `importlib.util` to load each file as a module and checks that the module exposes a `scan()` function. This means: + +- **Adding a rule requires no code change to the engine** — drop a file into `scanner/rules/` and it is automatically discovered on next startup. +- Rules that fail to load (syntax errors, missing imports) are logged and skipped. The remaining rules still run. + +### 3. Scan Execution + +```python +result = engine.run_scan() +``` + +`run_scan()` iterates through all loaded rule modules, calling `module.scan(azure_client, subscription_id)` for each. Individual rule failures are caught and logged without stopping the scan. The engine collects all findings and returns a structured result dict. + +### 4. Finding Schema + +Every finding returned by a rule must conform to this schema: + +```python +{ + "rule_id": str, # e.g. "AZ-STOR-001" + "rule_name": str, + "severity": str, # HIGH | MEDIUM | LOW | INFO + "category": str, # Storage | Network | Identity | Database | Compute | KeyVault + "resource_id": str, # full Azure resource ID + "resource_name": str, + "resource_type": str, # e.g. "Microsoft.Storage/storageAccounts" + "description": str, + "remediation": str, + "playbook": str, # path to the CLI remediation script + "frameworks": dict, # {"CIS": "3.5", "NIST": "PR.AC-3", "ISO27001": "A.9.4.1"} + "detected_at": str, # ISO 8601, added by engine + "scan_id": str, # UUID, added by engine +} +``` + +--- + +## How Findings Flow to the API + +``` +run_scan() + → findings[] in memory + → db.save_scan(result) # persists to PostgreSQL + → return scan result JSON + +GET /api/findings + → db.get_findings(filters) # reads from PostgreSQL + → returns JSON array + +GET /api/score + → db.get_score() # severity-weighted 0-100 + → returns {"score": 82} + +GET /api/compliance/cis + → db.get_compliance_score("cis") # joins DB findings with CIS JSON + → returns per-control pass/fail breakdown +``` + +--- + +## How Rules Are Loaded Dynamically + +The engine uses Python's `importlib` to load rule files at runtime. No registry or central list is needed: + +```python +for rule_path in sorted(RULES_DIR.glob("*.py")): + if rule_path.name.startswith("_"): + continue + spec = importlib.util.spec_from_file_location(rule_path.stem, rule_path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + if callable(getattr(module, "scan", None)): + self.rules.append(module) +``` + +Each rule module is a plain Python file — no base class, no registration decorator. The only contract is the `scan(azure_client, subscription_id)` function signature. + +--- + +## How Sentinel Integration Works + +> **Note:** Sentinel push is handled by a separate team. This section documents the integration point. + +After `run_scan()` returns, findings can be forwarded to Microsoft Sentinel via the Azure Monitor Ingestion API. The `sentinel/` directory contains the KQL detection rules and the ingestion client configuration. + +The flow: +1. `POST /api/scans/trigger` → scan completes → findings in DB +2. A Sentinel push worker (separate process or Azure Function) polls the DB for new findings +3. New findings are batched and sent to a Log Analytics Workspace via `azure-monitor-ingestion` +4. KQL detection rules in Sentinel fire alerts on HIGH-severity findings + +The required environment variable is `SENTINEL_WORKSPACE_ID` (see `.env.example`). + +--- + +## Configuration + +All runtime configuration is provided via environment variables (see `.env.example`): + +| Variable | Description | +|---|---| +| `AZURE_SUBSCRIPTION_ID` | Target subscription to scan | +| `AZURE_CLIENT_ID` | Service principal client ID | +| `AZURE_CLIENT_SECRET` | Service principal client secret | +| `AZURE_TENANT_ID` | Azure AD tenant ID | +| `DATABASE_URL` | PostgreSQL connection string | +| `JWT_SECRET` | Secret used to sign/verify API JWTs | +| `SENTINEL_WORKSPACE_ID` | Log Analytics workspace ID for Sentinel push | diff --git a/docs/azure-setup.md b/docs/azure-setup.md new file mode 100644 index 00000000..d2f82317 --- /dev/null +++ b/docs/azure-setup.md @@ -0,0 +1,205 @@ +# Azure Setup Guide + +This guide gets you from zero to a running OpenShield scan in under 20 minutes using a free Azure account. + +--- + +## Step 1 — Create a Free Azure Account + +1. Go to [azure.microsoft.com/free](https://azure.microsoft.com/free) and click **Start free**. +2. Sign in with a Microsoft account (or create one). +3. Complete the sign-up — you will receive $200 in free credits and access to free-tier services. +4. After sign-up, navigate to the [Azure Portal](https://portal.azure.com). + +--- + +## Step 2 — Get Your Subscription ID + +1. In the Azure Portal, search for **Subscriptions** in the top search bar. +2. Click on your subscription name. +3. Copy the **Subscription ID** (a UUID like `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`). + +You will need this value for `AZURE_SUBSCRIPTION_ID` in your `.env` file. + +--- + +## Step 3 — Create a Service Principal with Reader Role + +OpenShield only needs read access to scan your subscription. Use the Azure CLI: + +```bash +# Install Azure CLI if you haven't already +# https://learn.microsoft.com/en-us/cli/azure/install-azure-cli + +# Login +az login + +# Create the service principal with Reader role +az ad sp create-for-rbac \ + --name "openshield-scanner" \ + --role Reader \ + --scopes /subscriptions/ \ + --output json +``` + +This command outputs JSON like: + +```json +{ + "appId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", + "displayName": "openshield-scanner", + "password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", + "tenant": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" +} +``` + +Map these values: +- `appId` → `AZURE_CLIENT_ID` +- `password` → `AZURE_CLIENT_SECRET` +- `tenant` → `AZURE_TENANT_ID` + +> **Important:** The `password` is only shown once. Copy it immediately. + +--- + +## Step 4 — Grant Additional Read Permissions (Optional) + +For the Conditional Access MFA rule (AZ-IDN-002), the service principal needs the +`Policy.Read.All` Microsoft Graph API permission: + +```bash +# Get the service principal object ID +SP_OBJECT_ID=$(az ad sp show --id --query id --output tsv) + +# Grant Policy.Read.All application permission +# This requires a Global Administrator to consent +az rest \ + --method POST \ + --uri "https://graph.microsoft.com/v1.0/servicePrincipals/$SP_OBJECT_ID/appRoleAssignments" \ + --body '{ + "principalId": "'$SP_OBJECT_ID'", + "resourceId": "", + "appRoleId": "246dd0d5-5bd0-4def-940b-0421030a5b68" + }' +``` + +If you skip this step, AZ-IDN-002 will produce a finding by default (it cannot verify MFA status without Graph access). + +--- + +## Step 5 — Configure Your .env File + +Copy the example and fill in your values: + +```bash +cp .env.example .env +``` + +Edit `.env`: + +``` +AZURE_SUBSCRIPTION_ID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx +AZURE_CLIENT_ID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx +AZURE_CLIENT_SECRET=your-client-secret-from-step-3 +AZURE_TENANT_ID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx +DATABASE_URL=postgresql://openshield:openshield@localhost:5432/openshield +JWT_SECRET=your-random-secret-at-least-32-chars +SENTINEL_WORKSPACE_ID= +``` + +--- + +## Step 6 — Start a Local PostgreSQL Database + +```bash +# Option A: Docker (easiest) +docker run --name openshield-db \ + -e POSTGRES_USER=openshield \ + -e POSTGRES_PASSWORD=openshield \ + -e POSTGRES_DB=openshield \ + -p 5432:5432 \ + -d postgres:15 + +# Option B: Homebrew (macOS) +brew install postgresql@15 +brew services start postgresql@15 +createdb openshield +``` + +The `DatabaseManager.create_tables()` call in the scan trigger will create the schema automatically on first run. + +--- + +## Step 7 — Run Your First Scan + +```bash +# From the openshield/ directory +cd openshield + +# Install dependencies +pip install -r requirements.txt + +# Run the scanner directly +python -c " +from dotenv import load_dotenv; load_dotenv() +import json, os +from scanner.engine import ScanEngine +engine = ScanEngine(os.environ['AZURE_SUBSCRIPTION_ID']) +result = engine.run_scan() +print(json.dumps(result, indent=2)) +" +``` + +Or trigger via the API: + +```bash +# Start the API server +FLASK_APP=api/app.py flask run + +# Trigger a scan +curl -X POST http://localhost:5000/api/scans/trigger \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{"subscription_id": "your-subscription-id"}' +``` + +--- + +## Step 8 — Activate the Microsoft Sentinel 90-Day Trial (Optional) + +Microsoft Sentinel includes a 90-day free trial for new Log Analytics workspaces. + +1. In the Azure Portal, search for **Microsoft Sentinel**. +2. Click **Create Microsoft Sentinel**. +3. Click **Create a new workspace** and fill in: + - Workspace name: `openshield-logs` + - Region: choose the same region as your resources +4. Click **Add Microsoft Sentinel** — the 90-day trial activates automatically. +5. Copy the **Workspace ID** from the workspace Overview page. +6. Add it to your `.env`: `SENTINEL_WORKSPACE_ID=` + +> **Cost after trial:** ~$2.76/GB ingested. For a small subscription with few findings, this is negligible. + +--- + +## Step 9 — Create a Log Analytics Workspace (for SQL Auditing) + +The AZ-DB-002 remediation playbook writes SQL audit logs to a storage account. To route them to Log Analytics instead: + +1. Go to your SQL server in the portal. +2. Under **Security**, click **Auditing**. +3. Set **Auditing** to **ON**. +4. Check **Log Analytics** and select your `openshield-logs` workspace. +5. Click **Save**. + +--- + +## Troubleshooting + +| Problem | Fix | +|---|---| +| `DefaultAzureCredential` fails | Run `az login` in the terminal, or verify env vars are set | +| `AZURE_CLIENT_SECRET` rejected | The secret may have expired — rotate it with `az ad sp credential reset` | +| `psycopg2.OperationalError` | Check your PostgreSQL container is running and `DATABASE_URL` is correct | +| Empty findings | Verify the service principal has `Reader` role on the subscription | +| AZ-IDN-002 always fires | The service principal needs `Policy.Read.All` Graph permission — see Step 4 | diff --git a/openshield b/openshield deleted file mode 160000 index 647f74b6..00000000 --- a/openshield +++ /dev/null @@ -1 +0,0 @@ -Subproject commit 647f74b69888891b39d3af8aa77aacfcfae83770 diff --git a/playbooks/cli/fix_az_cmp_001.sh b/playbooks/cli/fix_az_cmp_001.sh new file mode 100755 index 00000000..5d997ded --- /dev/null +++ b/playbooks/cli/fix_az_cmp_001.sh @@ -0,0 +1,33 @@ +#!/bin/bash +# OpenShield Remediation Playbook +# Rule: AZ-CMP-001 — VM with Public IP and No Associated NSG on Network Interface +# Usage: ./fix_az_cmp_001.sh +# Severity: HIGH +# +# This script associates an existing NSG with the vulnerable NIC. +# If the NSG does not yet exist, create it first: +# az network nsg create --resource-group --name + +set -e + +RESOURCE_GROUP=$1 +NIC_NAME=$2 +NSG_NAME=$3 + +if [ -z "$RESOURCE_GROUP" ] || [ -z "$NIC_NAME" ] || [ -z "$NSG_NAME" ]; then + echo "Usage: $0 " + echo "" + echo "To create a new NSG first:" + echo " az network nsg create --resource-group --name " + exit 1 +fi + +echo "Associating NSG '$NSG_NAME' with NIC '$NIC_NAME'..." + +az network nic update \ + --resource-group "$RESOURCE_GROUP" \ + --name "$NIC_NAME" \ + --network-security-group "$NSG_NAME" + +echo "✅ Remediation complete for $NIC_NAME — NSG '$NSG_NAME' is now associated." +echo "⚠️ Review the NSG rules to ensure only necessary inbound traffic is permitted." diff --git a/playbooks/cli/fix_az_db_001.sh b/playbooks/cli/fix_az_db_001.sh new file mode 100755 index 00000000..93e90681 --- /dev/null +++ b/playbooks/cli/fix_az_db_001.sh @@ -0,0 +1,27 @@ +#!/bin/bash +# OpenShield Remediation Playbook +# Rule: AZ-DB-001 — PostgreSQL Server Allows Public Network Access +# Usage: ./fix_az_db_001.sh +# Severity: HIGH + +set -e + +RESOURCE_GROUP=$1 +RESOURCE_NAME=$2 + +if [ -z "$RESOURCE_GROUP" ] || [ -z "$RESOURCE_NAME" ]; then + echo "Usage: $0 " + exit 1 +fi + +echo "Disabling public network access on PostgreSQL server: $RESOURCE_NAME" + +az postgres server update \ + --resource-group "$RESOURCE_GROUP" \ + --name "$RESOURCE_NAME" \ + --public-network-access Disabled + +echo "" +echo "✅ Remediation complete for $RESOURCE_NAME — public network access is now disabled." +echo "⚠️ Ensure a private endpoint or VNet service endpoint is configured before" +echo " disabling public access, or applications will lose connectivity." diff --git a/playbooks/cli/fix_az_db_002.sh b/playbooks/cli/fix_az_db_002.sh new file mode 100755 index 00000000..ad05c6eb --- /dev/null +++ b/playbooks/cli/fix_az_db_002.sh @@ -0,0 +1,36 @@ +#!/bin/bash +# OpenShield Remediation Playbook +# Rule: AZ-DB-002 — Azure SQL Server Has No Auditing Configured +# Usage: ./fix_az_db_002.sh +# Severity: MEDIUM + +set -e + +RESOURCE_GROUP=$1 +RESOURCE_NAME=$2 +STORAGE_ACCOUNT=$3 + +if [ -z "$RESOURCE_GROUP" ] || [ -z "$RESOURCE_NAME" ] || [ -z "$STORAGE_ACCOUNT" ]; then + echo "Usage: $0 " + exit 1 +fi + +echo "Enabling SQL server auditing on: $RESOURCE_NAME" +echo "Audit logs will be written to storage account: $STORAGE_ACCOUNT" + +# Get the storage account endpoint +STORAGE_ENDPOINT=$(az storage account show \ + --name "$STORAGE_ACCOUNT" \ + --resource-group "$RESOURCE_GROUP" \ + --query primaryEndpoints.blob \ + --output tsv) + +az sql server audit-policy update \ + --resource-group "$RESOURCE_GROUP" \ + --name "$RESOURCE_NAME" \ + --state Enabled \ + --blob-storage-target-state Enabled \ + --storage-account "$STORAGE_ACCOUNT" \ + --retention-days 90 + +echo "✅ Remediation complete for $RESOURCE_NAME — SQL auditing enabled with 90-day retention." diff --git a/playbooks/cli/fix_az_idn_001.sh b/playbooks/cli/fix_az_idn_001.sh new file mode 100755 index 00000000..a076bbb0 --- /dev/null +++ b/playbooks/cli/fix_az_idn_001.sh @@ -0,0 +1,45 @@ +#!/bin/bash +# OpenShield Remediation Playbook +# Rule: AZ-IDN-001 — Service Principal Assigned Owner Role at Subscription Scope +# Usage: ./fix_az_idn_001.sh +# Severity: HIGH +# +# This script removes the Owner role from the service principal at subscription scope. +# You will need to assign a least-privilege replacement role manually afterwards. + +set -e + +SUBSCRIPTION_ID=$1 +PRINCIPAL_ID=$2 + +if [ -z "$SUBSCRIPTION_ID" ] || [ -z "$PRINCIPAL_ID" ]; then + echo "Usage: $0 " + exit 1 +fi + +SCOPE="/subscriptions/$SUBSCRIPTION_ID" +OWNER_ROLE="Owner" + +echo "Finding Owner role assignment for principal $PRINCIPAL_ID at subscription scope..." + +ASSIGNMENT_ID=$(az role assignment list \ + --scope "$SCOPE" \ + --assignee "$PRINCIPAL_ID" \ + --role "$OWNER_ROLE" \ + --query "[0].id" \ + --output tsv) + +if [ -z "$ASSIGNMENT_ID" ]; then + echo "No Owner role assignment found for principal $PRINCIPAL_ID — already remediated." + exit 0 +fi + +echo "Deleting role assignment: $ASSIGNMENT_ID" + +az role assignment delete \ + --ids "$ASSIGNMENT_ID" + +echo "" +echo "✅ Remediation complete — Owner role removed from $PRINCIPAL_ID." +echo "⚠️ ACTION REQUIRED: Assign a least-privilege replacement role to the service principal." +echo " Example: az role assignment create --assignee $PRINCIPAL_ID --role Contributor --scope $SCOPE" diff --git a/playbooks/cli/fix_az_idn_002.sh b/playbooks/cli/fix_az_idn_002.sh new file mode 100755 index 00000000..c66a51d1 --- /dev/null +++ b/playbooks/cli/fix_az_idn_002.sh @@ -0,0 +1,69 @@ +#!/bin/bash +# OpenShield Remediation Playbook +# Rule: AZ-IDN-002 — No MFA Enforced on Admin Accounts via Conditional Access +# Usage: ./fix_az_idn_002.sh +# Severity: HIGH +# +# This script creates a Conditional Access policy via Microsoft Graph API +# that requires MFA for all users assigned administrator directory roles. +# Prerequisites: +# - az login with a Global Administrator or Conditional Access Administrator account +# - Microsoft Graph PowerShell or Graph API access + +set -e + +echo "Creating Conditional Access policy to enforce MFA for administrators..." +echo "" +echo "This operation requires Global Administrator or Conditional Access Administrator privileges." +echo "" + +# Prompt for confirmation +read -p "Proceed with creating the MFA enforcement policy? [y/N] " CONFIRM +if [[ "$CONFIRM" != "y" && "$CONFIRM" != "Y" ]]; then + echo "Aborted." + exit 0 +fi + +# Acquire a Graph API token +TOKEN=$(az account get-access-token \ + --resource https://graph.microsoft.com \ + --query accessToken \ + --output tsv) + +POLICY_BODY='{ + "displayName": "OpenShield: Require MFA for Administrators", + "state": "enabled", + "conditions": { + "users": { + "includeRoles": [ + "62e90394-69f5-4237-9190-012177145e10", + "e8611ab8-c189-46e8-94e1-60213ab1f814", + "194ae4cb-b126-40b2-bd5b-6091b380977d", + "9b895d92-2cd3-44c7-9d02-a6ac2d5ea5c3" + ] + }, + "applications": { + "includeApplications": ["All"] + } + }, + "grantControls": { + "operator": "OR", + "builtInControls": ["mfa"] + } +}' + +RESPONSE=$(curl -s -X POST \ + "https://graph.microsoft.com/v1.0/identity/conditionalAccess/policies" \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d "$POLICY_BODY") + +POLICY_ID=$(echo "$RESPONSE" | python3 -c "import sys,json; print(json.load(sys.stdin).get('id',''))" 2>/dev/null) + +if [ -n "$POLICY_ID" ]; then + echo "✅ Remediation complete — Conditional Access policy created: $POLICY_ID" +else + echo "❌ Policy creation failed. Response:" + echo "$RESPONSE" + exit 1 +fi diff --git a/playbooks/cli/fix_az_kv_001.sh b/playbooks/cli/fix_az_kv_001.sh new file mode 100755 index 00000000..ee900d70 --- /dev/null +++ b/playbooks/cli/fix_az_kv_001.sh @@ -0,0 +1,39 @@ +#!/bin/bash +# OpenShield Remediation Playbook +# Rule: AZ-KV-001 — Key Vault with Soft Delete Disabled +# Usage: ./fix_az_kv_001.sh +# Severity: MEDIUM +# +# Note: Enabling soft delete is a one-way operation — it cannot be reversed. +# Once enabled, deleted objects enter a recoverable state for the retention period. + +set -e + +RESOURCE_GROUP=$1 +RESOURCE_NAME=$2 + +if [ -z "$RESOURCE_GROUP" ] || [ -z "$RESOURCE_NAME" ]; then + echo "Usage: $0 " + exit 1 +fi + +echo "Enabling soft delete on Key Vault: $RESOURCE_NAME" +echo "⚠️ This operation is irreversible. Soft delete, once enabled, cannot be disabled." +echo "" + +read -p "Proceed? [y/N] " CONFIRM +if [[ "$CONFIRM" != "y" && "$CONFIRM" != "Y" ]]; then + echo "Aborted." + exit 0 +fi + +az keyvault update \ + --resource-group "$RESOURCE_GROUP" \ + --name "$RESOURCE_NAME" \ + --enable-soft-delete true \ + --retention-days 90 + +echo "" +echo "✅ Remediation complete for $RESOURCE_NAME — soft delete is now enabled (90-day retention)." +echo "Consider also enabling purge protection:" +echo " az keyvault update --name $RESOURCE_NAME --resource-group $RESOURCE_GROUP --enable-purge-protection true" diff --git a/playbooks/cli/fix_az_net_001.sh b/playbooks/cli/fix_az_net_001.sh new file mode 100755 index 00000000..fd3fbe8e --- /dev/null +++ b/playbooks/cli/fix_az_net_001.sh @@ -0,0 +1,49 @@ +#!/bin/bash +# OpenShield Remediation Playbook +# Rule: AZ-NET-001 — NSG Allows Unrestricted Inbound SSH from Any Source +# Usage: ./fix_az_net_001.sh [rule-name] +# Severity: HIGH +# +# Pass the optional rule-name if the offending rule is known (shown in finding metadata). +# Without it, the script removes any Allow-Inbound-TCP-22-from-Any rule it finds. + +set -e + +RESOURCE_GROUP=$1 +RESOURCE_NAME=$2 +RULE_NAME=${3:-""} + +if [ -z "$RESOURCE_GROUP" ] || [ -z "$RESOURCE_NAME" ]; then + echo "Usage: $0 [rule-name]" + exit 1 +fi + +if [ -n "$RULE_NAME" ]; then + echo "Deleting NSG rule '$RULE_NAME' from $RESOURCE_NAME" + az network nsg rule delete \ + --resource-group "$RESOURCE_GROUP" \ + --nsg-name "$RESOURCE_NAME" \ + --name "$RULE_NAME" +else + echo "Searching for inbound SSH rules in $RESOURCE_NAME..." + RULES=$(az network nsg rule list \ + --resource-group "$RESOURCE_GROUP" \ + --nsg-name "$RESOURCE_NAME" \ + --query "[?direction=='Inbound' && access=='Allow' && destinationPortRange=='22' && (sourceAddressPrefix=='*' || sourceAddressPrefix=='0.0.0.0/0' || sourceAddressPrefix=='Internet')].name" \ + --output tsv) + + if [ -z "$RULES" ]; then + echo "No matching open SSH rule found — manual review recommended." + exit 0 + fi + + for RULE in $RULES; do + echo "Deleting rule: $RULE" + az network nsg rule delete \ + --resource-group "$RESOURCE_GROUP" \ + --nsg-name "$RESOURCE_NAME" \ + --name "$RULE" + done +fi + +echo "✅ Remediation complete for $RESOURCE_NAME — unrestricted SSH access removed." diff --git a/playbooks/cli/fix_az_net_002.sh b/playbooks/cli/fix_az_net_002.sh new file mode 100755 index 00000000..4cb78338 --- /dev/null +++ b/playbooks/cli/fix_az_net_002.sh @@ -0,0 +1,48 @@ +#!/bin/bash +# OpenShield Remediation Playbook +# Rule: AZ-NET-002 — NSG Allows Unrestricted Inbound RDP from Any Source +# Usage: ./fix_az_net_002.sh [rule-name] +# Severity: HIGH +# +# Pass the optional rule-name if the offending rule is known (shown in finding metadata). + +set -e + +RESOURCE_GROUP=$1 +RESOURCE_NAME=$2 +RULE_NAME=${3:-""} + +if [ -z "$RESOURCE_GROUP" ] || [ -z "$RESOURCE_NAME" ]; then + echo "Usage: $0 [rule-name]" + exit 1 +fi + +if [ -n "$RULE_NAME" ]; then + echo "Deleting NSG rule '$RULE_NAME' from $RESOURCE_NAME" + az network nsg rule delete \ + --resource-group "$RESOURCE_GROUP" \ + --nsg-name "$RESOURCE_NAME" \ + --name "$RULE_NAME" +else + echo "Searching for inbound RDP rules in $RESOURCE_NAME..." + RULES=$(az network nsg rule list \ + --resource-group "$RESOURCE_GROUP" \ + --nsg-name "$RESOURCE_NAME" \ + --query "[?direction=='Inbound' && access=='Allow' && destinationPortRange=='3389' && (sourceAddressPrefix=='*' || sourceAddressPrefix=='0.0.0.0/0' || sourceAddressPrefix=='Internet')].name" \ + --output tsv) + + if [ -z "$RULES" ]; then + echo "No matching open RDP rule found — manual review recommended." + exit 0 + fi + + for RULE in $RULES; do + echo "Deleting rule: $RULE" + az network nsg rule delete \ + --resource-group "$RESOURCE_GROUP" \ + --nsg-name "$RESOURCE_NAME" \ + --name "$RULE" + done +fi + +echo "✅ Remediation complete for $RESOURCE_NAME — unrestricted RDP access removed." diff --git a/playbooks/cli/fix_az_stor_001.sh b/playbooks/cli/fix_az_stor_001.sh new file mode 100755 index 00000000..2839a5c0 --- /dev/null +++ b/playbooks/cli/fix_az_stor_001.sh @@ -0,0 +1,24 @@ +#!/bin/bash +# OpenShield Remediation Playbook +# Rule: AZ-STOR-001 — Public Blob Access Enabled on Storage Account +# Usage: ./fix_az_stor_001.sh +# Severity: HIGH + +set -e + +RESOURCE_GROUP=$1 +RESOURCE_NAME=$2 + +if [ -z "$RESOURCE_GROUP" ] || [ -z "$RESOURCE_NAME" ]; then + echo "Usage: $0 " + exit 1 +fi + +echo "Disabling public blob access on storage account: $RESOURCE_NAME" + +az storage account update \ + --name "$RESOURCE_NAME" \ + --resource-group "$RESOURCE_GROUP" \ + --allow-blob-public-access false + +echo "✅ Remediation complete for $RESOURCE_NAME — public blob access is now disabled." diff --git a/playbooks/cli/fix_az_stor_002.sh b/playbooks/cli/fix_az_stor_002.sh new file mode 100755 index 00000000..1d96907c --- /dev/null +++ b/playbooks/cli/fix_az_stor_002.sh @@ -0,0 +1,24 @@ +#!/bin/bash +# OpenShield Remediation Playbook +# Rule: AZ-STOR-002 — Storage Account Allows HTTP Traffic (Not HTTPS-Only) +# Usage: ./fix_az_stor_002.sh +# Severity: HIGH + +set -e + +RESOURCE_GROUP=$1 +RESOURCE_NAME=$2 + +if [ -z "$RESOURCE_GROUP" ] || [ -z "$RESOURCE_NAME" ]; then + echo "Usage: $0 " + exit 1 +fi + +echo "Enabling HTTPS-only (secure transfer required) on: $RESOURCE_NAME" + +az storage account update \ + --name "$RESOURCE_NAME" \ + --resource-group "$RESOURCE_GROUP" \ + --https-only true + +echo "✅ Remediation complete for $RESOURCE_NAME — HTTPS-only traffic is now enforced." diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 00000000..ee813479 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,16 @@ +flask==3.0.0 +flask-cors==4.0.0 +azure-identity==1.15.0 +azure-mgmt-storage==21.0.0 +azure-mgmt-network==25.0.0 +azure-mgmt-compute==30.0.0 +azure-mgmt-resource==23.0.0 +azure-mgmt-sql==3.0.1 +azure-mgmt-keyvault==10.3.0 +azure-mgmt-rdbms==10.1.0 +azure-mgmt-authorization==4.0.0 +azure-monitor-ingestion==1.0.3 +psycopg2-binary==2.9.9 +python-dotenv==1.0.0 +pyjwt==2.8.0 +requests==2.31.0 diff --git a/scanner/__init__.py b/scanner/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/scanner/azure_client.py b/scanner/azure_client.py index e69de29b..e7381b5d 100644 --- a/scanner/azure_client.py +++ b/scanner/azure_client.py @@ -0,0 +1,188 @@ +"""Azure SDK wrapper providing typed accessors for all CSPM scan operations.""" + +import logging +from typing import Any, Dict, List, Optional + +from azure.identity import DefaultAzureCredential +from azure.mgmt.authorization import AuthorizationManagementClient +from azure.mgmt.compute import ComputeManagementClient +from azure.mgmt.keyvault import KeyVaultManagementClient +from azure.mgmt.network import NetworkManagementClient +from azure.mgmt.rdbms.postgresql import PostgreSQLManagementClient +from azure.mgmt.sql import SqlManagementClient +from azure.mgmt.storage import StorageManagementClient + +logger = logging.getLogger(__name__) + +# Azure built-in role definition GUIDs (subscription-scoped) +OWNER_ROLE_ID = "8e3af657-a8ff-443c-a75c-2fe8c4bcb635" + + +class AzureClient: + """Wraps Azure SDK management clients for all CSPM scan operations. + + Instantiate once per scan and share across all rule modules. Every method + logs on failure and returns an empty list so individual rule failures never + crash the scan engine. + """ + + def __init__( + self, subscription_id: str, credential: Optional[Any] = None + ) -> None: + self.subscription_id = subscription_id + self.credential = credential or DefaultAzureCredential() + + # ------------------------------------------------------------------ # + # Static helpers # + # ------------------------------------------------------------------ # + + @staticmethod + def parse_resource_id(resource_id: str) -> Dict[str, str]: + """Return resource_group and name parsed from an Azure resource ID.""" + parts = resource_id.split("/") + result: Dict[str, str] = {"name": parts[-1] if parts else ""} + for idx, segment in enumerate(parts): + if segment.lower() == "resourcegroups" and idx + 1 < len(parts): + result["resource_group"] = parts[idx + 1] + return result + + # ------------------------------------------------------------------ # + # Storage # + # ------------------------------------------------------------------ # + + def get_storage_accounts(self) -> List[Any]: + """List all storage accounts in the subscription.""" + try: + client = StorageManagementClient(self.credential, self.subscription_id) + return list(client.storage_accounts.list()) + except Exception as exc: + logger.error("get_storage_accounts failed: %s", exc) + return [] + + # ------------------------------------------------------------------ # + # Network # + # ------------------------------------------------------------------ # + + def get_network_security_groups(self) -> List[Any]: + """List all NSGs across all resource groups in the subscription.""" + try: + client = NetworkManagementClient(self.credential, self.subscription_id) + return list(client.network_security_groups.list_all()) + except Exception as exc: + logger.error("get_network_security_groups failed: %s", exc) + return [] + + def get_network_interface( + self, resource_group: str, nic_name: str + ) -> Optional[Any]: + """Fetch a single NIC by resource group and name.""" + try: + client = NetworkManagementClient(self.credential, self.subscription_id) + return client.network_interfaces.get(resource_group, nic_name) + except Exception as exc: + logger.error("get_network_interface(%s) failed: %s", nic_name, exc) + return None + + # ------------------------------------------------------------------ # + # Compute # + # ------------------------------------------------------------------ # + + def get_virtual_machines(self) -> List[Any]: + """List all VMs across all resource groups in the subscription.""" + try: + client = ComputeManagementClient(self.credential, self.subscription_id) + return list(client.virtual_machines.list_all()) + except Exception as exc: + logger.error("get_virtual_machines failed: %s", exc) + return [] + + # ------------------------------------------------------------------ # + # Databases # + # ------------------------------------------------------------------ # + + def get_postgresql_servers(self) -> List[Any]: + """List all PostgreSQL single-server instances in the subscription.""" + try: + client = PostgreSQLManagementClient(self.credential, self.subscription_id) + return list(client.servers.list()) + except Exception as exc: + logger.error("get_postgresql_servers failed: %s", exc) + return [] + + def get_sql_servers(self) -> List[Any]: + """List all Azure SQL servers in the subscription.""" + try: + client = SqlManagementClient(self.credential, self.subscription_id) + return list(client.servers.list()) + except Exception as exc: + logger.error("get_sql_servers failed: %s", exc) + return [] + + def get_sql_server_auditing_policy( + self, resource_group: str, server_name: str + ) -> Optional[Any]: + """Fetch the blob auditing policy for an Azure SQL server.""" + try: + client = SqlManagementClient(self.credential, self.subscription_id) + return client.server_blob_auditing_policies.get(resource_group, server_name) + except Exception as exc: + logger.error( + "get_sql_server_auditing_policy(%s) failed: %s", server_name, exc + ) + return None + + # ------------------------------------------------------------------ # + # Key Vault # + # ------------------------------------------------------------------ # + + def get_key_vaults(self) -> List[Any]: + """List all Key Vaults in the subscription with full properties.""" + try: + client = KeyVaultManagementClient(self.credential, self.subscription_id) + return list(client.vaults.list_by_subscription()) + except Exception as exc: + logger.error("get_key_vaults failed: %s", exc) + return [] + + # ------------------------------------------------------------------ # + # Identity / Authorization # + # ------------------------------------------------------------------ # + + def get_service_principals(self) -> List[Any]: + """Return role assignments whose principal type is ServicePrincipal.""" + try: + client = AuthorizationManagementClient( + self.credential, self.subscription_id + ) + scope = f"/subscriptions/{self.subscription_id}" + assignments = list(client.role_assignments.list_for_scope(scope)) + return [ + a + for a in assignments + if getattr(a, "principal_type", "") == "ServicePrincipal" + ] + except Exception as exc: + logger.error("get_service_principals failed: %s", exc) + return [] + + def get_conditional_access_policies(self) -> List[Any]: + """Fetch Conditional Access policies from the Microsoft Graph API. + + Requires the credential to have 'Policy.Read.All' Graph permission. + Returns empty list if the permission is not granted or the call fails. + """ + import requests # imported here to keep azure-only paths dependency-free + + try: + token = self.credential.get_token("https://graph.microsoft.com/.default") + headers = {"Authorization": f"Bearer {token.token}"} + response = requests.get( + "https://graph.microsoft.com/v1.0/identity/conditionalAccess/policies", + headers=headers, + timeout=30, + ) + response.raise_for_status() + return response.json().get("value", []) + except Exception as exc: + logger.error("get_conditional_access_policies failed: %s", exc) + return [] diff --git a/scanner/engine.py b/scanner/engine.py index e69de29b..46ce0e3e 100644 --- a/scanner/engine.py +++ b/scanner/engine.py @@ -0,0 +1,110 @@ +"""Scan engine: loads rules dynamically and orchestrates a full subscription scan.""" + +import importlib.util +import logging +import uuid +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Dict, List + +from scanner.azure_client import AzureClient + +logger = logging.getLogger(__name__) + +RULES_DIR = Path(__file__).parent / "rules" + + +class ScanEngine: + """Orchestrates Azure CSPM scans against a target subscription. + + Rules are loaded dynamically at initialisation time from ``scanner/rules/``. + Each rule module must expose a ``scan(azure_client, subscription_id)`` + function and the module-level constants ``RULE_ID``, ``RULE_NAME``, + ``SEVERITY``, ``CATEGORY``, ``FRAMEWORKS``, ``DESCRIPTION``, + ``REMEDIATION``, and ``PLAYBOOK``. + """ + + def __init__(self, subscription_id: str) -> None: + self.subscription_id = subscription_id + self.client = AzureClient(subscription_id) + self.rules: List[Any] = [] + self.load_rules() + + # ------------------------------------------------------------------ # + # Rule loading # + # ------------------------------------------------------------------ # + + def load_rules(self) -> None: + """Dynamically import every *.py file in scanner/rules/ as a rule module.""" + for rule_path in sorted(RULES_DIR.glob("*.py")): + if rule_path.name.startswith("_"): + continue + try: + spec = importlib.util.spec_from_file_location( + rule_path.stem, rule_path + ) + module = importlib.util.module_from_spec(spec) # type: ignore[arg-type] + spec.loader.exec_module(module) # type: ignore[union-attr] + if callable(getattr(module, "scan", None)): + self.rules.append(module) + logger.info( + "Loaded rule: %s", getattr(module, "RULE_ID", rule_path.stem) + ) + else: + logger.warning( + "Rule file %s has no scan() function — skipped", rule_path.name + ) + except Exception as exc: + logger.error("Failed to load rule %s: %s", rule_path.name, exc) + + # ------------------------------------------------------------------ # + # Scan execution # + # ------------------------------------------------------------------ # + + def run_scan(self) -> Dict[str, Any]: + """Execute all loaded rules and return a normalised scan result. + + Returns: + dict with keys: scan_id, subscription_id, started_at, + completed_at, total_findings, findings. + """ + scan_id = str(uuid.uuid4()) + started_at = datetime.now(timezone.utc).isoformat() + findings: List[Dict[str, Any]] = [] + detected_at = datetime.now(timezone.utc).isoformat() + + logger.info( + "Scan %s starting against subscription %s — %d rules loaded", + scan_id, + self.subscription_id, + len(self.rules), + ) + + for rule in self.rules: + rule_id = getattr(rule, "RULE_ID", "UNKNOWN") + try: + rule_findings = rule.scan(self.client, self.subscription_id) + for finding in rule_findings: + finding.setdefault("detected_at", detected_at) + finding.setdefault("scan_id", scan_id) + findings.extend(rule_findings) + logger.info( + "Rule %s produced %d finding(s)", rule_id, len(rule_findings) + ) + except Exception as exc: + logger.error("Rule %s raised an exception: %s", rule_id, exc) + + completed_at = datetime.now(timezone.utc).isoformat() + + logger.info( + "Scan %s complete — %d total finding(s)", scan_id, len(findings) + ) + + return { + "scan_id": scan_id, + "subscription_id": self.subscription_id, + "started_at": started_at, + "completed_at": completed_at, + "total_findings": len(findings), + "findings": findings, + } diff --git a/scanner/rules/__init__.py b/scanner/rules/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/scanner/rules/az_cmp_001.py b/scanner/rules/az_cmp_001.py new file mode 100644 index 00000000..64785b31 --- /dev/null +++ b/scanner/rules/az_cmp_001.py @@ -0,0 +1,77 @@ +"""AZ-CMP-001: Virtual machine has a public IP with no associated NSG.""" + +import logging +from typing import Any, Dict, List + +RULE_ID = "AZ-CMP-001" +RULE_NAME = "VM with Public IP and No Associated NSG on Network Interface" +SEVERITY = "HIGH" +CATEGORY = "Compute" +FRAMEWORKS = {"CIS": "7.2", "NIST": "PR.AC-3", "ISO27001": "A.13.1.1"} +DESCRIPTION = ( + "A virtual machine has a public IP address assigned to its network interface " + "but no Network Security Group protecting that interface. Without an NSG, " + "all inbound ports are open to the internet by default, creating an unrestricted " + "attack surface." +) +REMEDIATION = ( + "Associate an NSG with the VM's network interface or its subnet that allows " + "only required inbound traffic. Remove the public IP if internet access is not needed " + "and use Azure Bastion or a VPN gateway for administrative access." +) +PLAYBOOK = "playbooks/cli/fix_az_cmp_001.sh" + +logger = logging.getLogger(__name__) + + +def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: + """Detect VMs whose NIC has a public IP but no NSG attached.""" + findings: List[Dict[str, Any]] = [] + + for vm in azure_client.get_virtual_machines(): + network_profile = getattr(vm, "network_profile", None) + if not network_profile: + continue + + for nic_ref in getattr(network_profile, "network_interfaces", []) or []: + nic_id = getattr(nic_ref, "id", "") + if not nic_id: + continue + + parsed = azure_client.parse_resource_id(nic_id) + resource_group = parsed.get("resource_group", "") + nic_name = parsed.get("name", "") + if not resource_group or not nic_name: + continue + + nic = azure_client.get_network_interface(resource_group, nic_name) + if not nic: + continue + + has_public_ip = any( + getattr(ip_cfg, "public_ip_address", None) + for ip_cfg in (getattr(nic, "ip_configurations", []) or []) + ) + has_nsg = bool(getattr(nic, "network_security_group", None)) + + if has_public_ip and not has_nsg: + findings.append({ + "rule_id": RULE_ID, + "rule_name": RULE_NAME, + "severity": SEVERITY, + "category": CATEGORY, + "resource_id": vm.id, + "resource_name": vm.name, + "resource_type": "Microsoft.Compute/virtualMachines", + "description": DESCRIPTION, + "remediation": REMEDIATION, + "playbook": PLAYBOOK, + "frameworks": FRAMEWORKS, + "metadata": { + "nic_id": nic_id, + "nic_name": nic_name, + }, + }) + break # one finding per VM is sufficient + + return findings diff --git a/scanner/rules/az_db_001.py b/scanner/rules/az_db_001.py new file mode 100644 index 00000000..be9e3676 --- /dev/null +++ b/scanner/rules/az_db_001.py @@ -0,0 +1,48 @@ +"""AZ-DB-001: PostgreSQL server allows public network access.""" + +from typing import Any, Dict, List + +RULE_ID = "AZ-DB-001" +RULE_NAME = "PostgreSQL Server Allows Public Network Access" +SEVERITY = "HIGH" +CATEGORY = "Database" +FRAMEWORKS = {"CIS": "4.3.1", "NIST": "PR.AC-3", "ISO27001": "A.13.1.1"} +DESCRIPTION = ( + "The Azure Database for PostgreSQL server is configured to allow public network access. " + "This means the server endpoint is reachable from the public internet, increasing the " + "attack surface. Database servers should only be accessible from trusted private networks." +) +REMEDIATION = ( + "Disable public network access on the PostgreSQL server and configure a private endpoint " + "or VNet service endpoint to restrict connectivity to trusted networks only." +) +PLAYBOOK = "playbooks/cli/fix_az_db_001.sh" + + +def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: + """Detect PostgreSQL servers with public_network_access set to Enabled.""" + findings: List[Dict[str, Any]] = [] + + for server in azure_client.get_postgresql_servers(): + public_access = getattr(server, "public_network_access", "Enabled") + if str(public_access).lower() in ("enabled", "true", "1"): + parsed = azure_client.parse_resource_id(server.id) + findings.append({ + "rule_id": RULE_ID, + "rule_name": RULE_NAME, + "severity": SEVERITY, + "category": CATEGORY, + "resource_id": server.id, + "resource_name": server.name, + "resource_type": "Microsoft.DBforPostgreSQL/servers", + "description": DESCRIPTION, + "remediation": REMEDIATION, + "playbook": PLAYBOOK, + "frameworks": FRAMEWORKS, + "metadata": { + "resource_group": parsed.get("resource_group", ""), + "location": getattr(server, "location", ""), + }, + }) + + return findings diff --git a/scanner/rules/az_db_002.py b/scanner/rules/az_db_002.py new file mode 100644 index 00000000..7edba32b --- /dev/null +++ b/scanner/rules/az_db_002.py @@ -0,0 +1,60 @@ +"""AZ-DB-002: Azure SQL server has no auditing configured.""" + +from typing import Any, Dict, List + +RULE_ID = "AZ-DB-002" +RULE_NAME = "Azure SQL Server Has No Auditing Configured" +SEVERITY = "MEDIUM" +CATEGORY = "Database" +FRAMEWORKS = {"CIS": "4.1.3", "NIST": "DE.CM-7", "ISO27001": "A.12.4.1"} +DESCRIPTION = ( + "Azure SQL Server auditing is disabled. Without auditing, database access, " + "schema changes, and failed login attempts are not logged, making forensic " + "investigation and compliance reporting impossible." +) +REMEDIATION = ( + "Enable SQL Server auditing and configure a storage account, Log Analytics " + "workspace, or Event Hub as the audit log destination. " + "Retain logs for at least 90 days to satisfy most compliance frameworks." +) +PLAYBOOK = "playbooks/cli/fix_az_db_002.sh" + + +def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: + """Detect SQL servers where server-level blob auditing is disabled.""" + findings: List[Dict[str, Any]] = [] + + for server in azure_client.get_sql_servers(): + parsed = azure_client.parse_resource_id(server.id) + resource_group = parsed.get("resource_group", "") + if not resource_group: + continue + + policy = azure_client.get_sql_server_auditing_policy(resource_group, server.name) + if policy is None: + # Could not retrieve policy — treat as unaudited + is_disabled = True + else: + state = str(getattr(policy, "state", "Disabled")) + is_disabled = state.lower() != "enabled" + + if is_disabled: + findings.append({ + "rule_id": RULE_ID, + "rule_name": RULE_NAME, + "severity": SEVERITY, + "category": CATEGORY, + "resource_id": server.id, + "resource_name": server.name, + "resource_type": "Microsoft.Sql/servers", + "description": DESCRIPTION, + "remediation": REMEDIATION, + "playbook": PLAYBOOK, + "frameworks": FRAMEWORKS, + "metadata": { + "resource_group": resource_group, + "auditing_state": getattr(policy, "state", "Unknown") if policy else "Unknown", + }, + }) + + return findings diff --git a/scanner/rules/az_idn_001.py b/scanner/rules/az_idn_001.py new file mode 100644 index 00000000..ac64ccfa --- /dev/null +++ b/scanner/rules/az_idn_001.py @@ -0,0 +1,57 @@ +"""AZ-IDN-001: Service principal assigned Owner role at subscription scope.""" + +from typing import Any, Dict, List + +RULE_ID = "AZ-IDN-001" +RULE_NAME = "Service Principal Assigned Owner Role at Subscription Scope" +SEVERITY = "HIGH" +CATEGORY = "Identity" +FRAMEWORKS = {"CIS": "1.23", "NIST": "PR.AC-4", "ISO27001": "A.9.2.3"} +DESCRIPTION = ( + "A service principal holds the Owner role at subscription scope, granting it " + "full control over all resources and the ability to assign roles to other principals. " + "This violates the principle of least privilege and represents a critical blast-radius " + "risk if the service principal credentials are compromised." +) +REMEDIATION = ( + "Replace the Owner role assignment with a narrower built-in role (e.g., Contributor, " + "or a custom role) that covers only the required permissions. " + "Audit and rotate the service principal's client secret or certificate." +) +PLAYBOOK = "playbooks/cli/fix_az_idn_001.sh" + +# Azure built-in Owner role definition GUID +OWNER_ROLE_GUID = "8e3af657-a8ff-443c-a75c-2fe8c4bcb635" + + +def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: + """Detect service principals holding the Owner role at subscription scope.""" + findings: List[Dict[str, Any]] = [] + + for assignment in azure_client.get_service_principals(): + role_def_id = getattr(assignment, "role_definition_id", "") or "" + if not role_def_id.endswith(OWNER_ROLE_GUID): + continue + + principal_id = getattr(assignment, "principal_id", "unknown") + resource_id = getattr(assignment, "id", "") + + findings.append({ + "rule_id": RULE_ID, + "rule_name": RULE_NAME, + "severity": SEVERITY, + "category": CATEGORY, + "resource_id": resource_id, + "resource_name": principal_id, + "resource_type": "Microsoft.Authorization/roleAssignments", + "description": DESCRIPTION, + "remediation": REMEDIATION, + "playbook": PLAYBOOK, + "frameworks": FRAMEWORKS, + "metadata": { + "principal_id": principal_id, + "scope": getattr(assignment, "scope", ""), + }, + }) + + return findings diff --git a/scanner/rules/az_idn_002.py b/scanner/rules/az_idn_002.py new file mode 100644 index 00000000..9a8417ae --- /dev/null +++ b/scanner/rules/az_idn_002.py @@ -0,0 +1,84 @@ +"""AZ-IDN-002: No MFA enforced on admin accounts via Conditional Access.""" + +from typing import Any, Dict, List + +RULE_ID = "AZ-IDN-002" +RULE_NAME = "No MFA Enforced on Admin Accounts via Conditional Access" +SEVERITY = "HIGH" +CATEGORY = "Identity" +FRAMEWORKS = {"CIS": "1.2.4", "NIST": "PR.AC-1", "ISO27001": "A.9.4.2"} +DESCRIPTION = ( + "No Conditional Access policy is enabled that requires multi-factor authentication " + "for administrator accounts. Without MFA enforcement, a single compromised password " + "is sufficient for an attacker to gain privileged access to the Azure tenant." +) +REMEDIATION = ( + "Create a Conditional Access policy that targets administrator directory roles " + "(Global Administrator, Privileged Role Administrator, etc.) and requires " + "MFA as a grant control. Ensure the policy state is set to 'enabled'." +) +PLAYBOOK = "playbooks/cli/fix_az_idn_002.sh" + +# Privileged Azure AD directory role IDs (subset most relevant for MFA enforcement) +ADMIN_ROLE_IDS = { + "62e90394-69f5-4237-9190-012177145e10", # Global Administrator + "e8611ab8-c189-46e8-94e1-60213ab1f814", # Privileged Role Administrator + "194ae4cb-b126-40b2-bd5b-6091b380977d", # Security Administrator + "9b895d92-2cd3-44c7-9d02-a6ac2d5ea5c3", # Application Administrator +} + + +def _policy_enforces_mfa_for_admins(policy: Dict[str, Any]) -> bool: + """Return True if the CA policy is enabled, requires MFA, and targets admins.""" + if policy.get("state") != "enabled": + return False + + grant = policy.get("grantControls") or {} + controls = grant.get("builtInControls", []) + if "mfa" not in controls: + return False + + conditions = policy.get("conditions") or {} + users = conditions.get("users") or {} + + # Covers all users → definitely covers admins + if "All" in (users.get("includeUsers") or []): + return True + + # Covers specific admin roles + included_roles = set(users.get("includeRoles") or []) + return bool(included_roles & ADMIN_ROLE_IDS) + + +def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: + """Detect tenants where no CA policy enforces MFA for administrators. + + Requires the credential to have the 'Policy.Read.All' Microsoft Graph + permission. If the Graph call fails (e.g. insufficient permissions), a + finding is still raised because the posture cannot be verified. + """ + policies = azure_client.get_conditional_access_policies() + + if policies and any(_policy_enforces_mfa_for_admins(p) for p in policies): + return [] + + reason = ( + "No Conditional Access policies found — Graph API may be inaccessible." + if not policies + else "Existing Conditional Access policies do not enforce MFA for admin roles." + ) + + return [{ + "rule_id": RULE_ID, + "rule_name": RULE_NAME, + "severity": SEVERITY, + "category": CATEGORY, + "resource_id": f"/tenants/{subscription_id}/conditionalAccess", + "resource_name": "Conditional Access Policies", + "resource_type": "Microsoft.AzureActiveDirectory/conditionalAccessPolicies", + "description": DESCRIPTION, + "remediation": REMEDIATION, + "playbook": PLAYBOOK, + "frameworks": FRAMEWORKS, + "metadata": {"reason": reason, "policies_found": len(policies)}, + }] diff --git a/scanner/rules/az_kv_001.py b/scanner/rules/az_kv_001.py new file mode 100644 index 00000000..f9cada65 --- /dev/null +++ b/scanner/rules/az_kv_001.py @@ -0,0 +1,55 @@ +"""AZ-KV-001: Key Vault with soft delete disabled.""" + +from typing import Any, Dict, List + +RULE_ID = "AZ-KV-001" +RULE_NAME = "Key Vault with Soft Delete Disabled" +SEVERITY = "MEDIUM" +CATEGORY = "KeyVault" +FRAMEWORKS = {"CIS": "8.5", "NIST": "PR.IP-4", "ISO27001": "A.17.2.1"} +DESCRIPTION = ( + "Azure Key Vault soft delete is disabled. Without soft delete, secrets, keys, " + "and certificates can be permanently destroyed immediately upon deletion — " + "whether by accident, a disgruntled insider, or an attacker who has gained access. " + "Soft delete provides a recoverable state for 7–90 days after deletion." +) +REMEDIATION = ( + "Enable soft delete on the Key Vault. Note: once enabled, soft delete cannot be disabled. " + "Also consider enabling purge protection to prevent permanent deletion during the retention period." +) +PLAYBOOK = "playbooks/cli/fix_az_kv_001.sh" + + +def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: + """Detect Key Vaults where enable_soft_delete is False or None.""" + findings: List[Dict[str, Any]] = [] + + for vault in azure_client.get_key_vaults(): + props = getattr(vault, "properties", None) + if props is None: + continue + + # Soft delete defaults to True in Azure API version 2021-04-01+ + # but older vaults or explicitly disabled vaults may have it False. + soft_delete = getattr(props, "enable_soft_delete", True) + if soft_delete is False: + parsed = azure_client.parse_resource_id(vault.id) + findings.append({ + "rule_id": RULE_ID, + "rule_name": RULE_NAME, + "severity": SEVERITY, + "category": CATEGORY, + "resource_id": vault.id, + "resource_name": vault.name, + "resource_type": "Microsoft.KeyVault/vaults", + "description": DESCRIPTION, + "remediation": REMEDIATION, + "playbook": PLAYBOOK, + "frameworks": FRAMEWORKS, + "metadata": { + "resource_group": parsed.get("resource_group", ""), + "purge_protection": getattr(props, "enable_purge_protection", False), + }, + }) + + return findings diff --git a/scanner/rules/az_net_001.py b/scanner/rules/az_net_001.py new file mode 100644 index 00000000..6976a963 --- /dev/null +++ b/scanner/rules/az_net_001.py @@ -0,0 +1,68 @@ +"""AZ-NET-001: NSG allows unrestricted inbound SSH (port 22) from 0.0.0.0/0.""" + +from typing import Any, Dict, List + +RULE_ID = "AZ-NET-001" +RULE_NAME = "NSG Allows Unrestricted Inbound SSH from Any Source" +SEVERITY = "HIGH" +CATEGORY = "Network" +FRAMEWORKS = {"CIS": "6.2", "NIST": "PR.AC-3", "ISO27001": "A.13.1.1"} +DESCRIPTION = ( + "The Network Security Group has an Allow rule for inbound TCP port 22 (SSH) " + "from any source address (0.0.0.0/0, *, or Internet). Exposing SSH to the " + "internet dramatically increases the attack surface and risk of brute-force attacks." +) +REMEDIATION = ( + "Remove or restrict the inbound SSH rule to known trusted IP ranges only. " + "Consider using Azure Bastion for privileged access instead of direct SSH exposure." +) +PLAYBOOK = "playbooks/cli/fix_az_net_001.sh" + +_OPEN_SOURCES = {"*", "0.0.0.0/0", "Internet", "Any"} + + +def _rule_allows_port_from_any(rule: Any, port: str) -> bool: + """Return True if a security rule allows inbound traffic on the given port from any source.""" + if str(getattr(rule, "direction", "")).lower() != "inbound": + return False + if str(getattr(rule, "access", "")).lower() != "allow": + return False + + source = getattr(rule, "source_address_prefix", "") or "" + source_prefixes = getattr(rule, "source_address_prefixes", []) or [] + source_open = source in _OPEN_SOURCES or any( + s in _OPEN_SOURCES for s in source_prefixes + ) + + if not source_open: + return False + + dest_range = str(getattr(rule, "destination_port_range", "") or "") + dest_ranges = [str(r) for r in (getattr(rule, "destination_port_ranges", []) or [])] + return dest_range in (port, "*") or port in dest_ranges or "*" in dest_ranges + + +def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: + """Detect NSGs with Allow-inbound-SSH-from-any rules.""" + findings: List[Dict[str, Any]] = [] + + for nsg in azure_client.get_network_security_groups(): + for rule in getattr(nsg, "security_rules", []) or []: + if _rule_allows_port_from_any(rule, "22"): + findings.append({ + "rule_id": RULE_ID, + "rule_name": RULE_NAME, + "severity": SEVERITY, + "category": CATEGORY, + "resource_id": nsg.id, + "resource_name": nsg.name, + "resource_type": "Microsoft.Network/networkSecurityGroups", + "description": DESCRIPTION, + "remediation": REMEDIATION, + "playbook": PLAYBOOK, + "frameworks": FRAMEWORKS, + "metadata": {"offending_rule": rule.name}, + }) + break # one finding per NSG is enough + + return findings diff --git a/scanner/rules/az_net_002.py b/scanner/rules/az_net_002.py new file mode 100644 index 00000000..906529ac --- /dev/null +++ b/scanner/rules/az_net_002.py @@ -0,0 +1,69 @@ +"""AZ-NET-002: NSG allows unrestricted inbound RDP (port 3389) from 0.0.0.0/0.""" + +from typing import Any, Dict, List + +RULE_ID = "AZ-NET-002" +RULE_NAME = "NSG Allows Unrestricted Inbound RDP from Any Source" +SEVERITY = "HIGH" +CATEGORY = "Network" +FRAMEWORKS = {"CIS": "6.3", "NIST": "PR.AC-3", "ISO27001": "A.13.1.1"} +DESCRIPTION = ( + "The Network Security Group has an Allow rule for inbound TCP port 3389 (RDP) " + "from any source address (0.0.0.0/0, *, or Internet). Exposing RDP to the " + "internet is one of the most common initial access vectors for ransomware and " + "credential-stuffing attacks." +) +REMEDIATION = ( + "Remove or restrict the inbound RDP rule to known trusted IP ranges only. " + "Consider using Azure Bastion for privileged Windows access instead of direct RDP exposure." +) +PLAYBOOK = "playbooks/cli/fix_az_net_002.sh" + +_OPEN_SOURCES = {"*", "0.0.0.0/0", "Internet", "Any"} + + +def _rule_allows_port_from_any(rule: Any, port: str) -> bool: + """Return True if a security rule allows inbound traffic on the given port from any source.""" + if str(getattr(rule, "direction", "")).lower() != "inbound": + return False + if str(getattr(rule, "access", "")).lower() != "allow": + return False + + source = getattr(rule, "source_address_prefix", "") or "" + source_prefixes = getattr(rule, "source_address_prefixes", []) or [] + source_open = source in _OPEN_SOURCES or any( + s in _OPEN_SOURCES for s in source_prefixes + ) + + if not source_open: + return False + + dest_range = str(getattr(rule, "destination_port_range", "") or "") + dest_ranges = [str(r) for r in (getattr(rule, "destination_port_ranges", []) or [])] + return dest_range in (port, "*") or port in dest_ranges or "*" in dest_ranges + + +def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: + """Detect NSGs with Allow-inbound-RDP-from-any rules.""" + findings: List[Dict[str, Any]] = [] + + for nsg in azure_client.get_network_security_groups(): + for rule in getattr(nsg, "security_rules", []) or []: + if _rule_allows_port_from_any(rule, "3389"): + findings.append({ + "rule_id": RULE_ID, + "rule_name": RULE_NAME, + "severity": SEVERITY, + "category": CATEGORY, + "resource_id": nsg.id, + "resource_name": nsg.name, + "resource_type": "Microsoft.Network/networkSecurityGroups", + "description": DESCRIPTION, + "remediation": REMEDIATION, + "playbook": PLAYBOOK, + "frameworks": FRAMEWORKS, + "metadata": {"offending_rule": rule.name}, + }) + break # one finding per NSG is enough + + return findings diff --git a/scanner/rules/az_stor_001.py b/scanner/rules/az_stor_001.py new file mode 100644 index 00000000..801ecbab --- /dev/null +++ b/scanner/rules/az_stor_001.py @@ -0,0 +1,42 @@ +"""AZ-STOR-001: Storage account with public blob access enabled.""" + +from typing import Any, Dict, List + +RULE_ID = "AZ-STOR-001" +RULE_NAME = "Public Blob Access Enabled on Storage Account" +SEVERITY = "HIGH" +CATEGORY = "Storage" +FRAMEWORKS = {"CIS": "3.5", "NIST": "PR.AC-3", "ISO27001": "A.9.4.1"} +DESCRIPTION = ( + "Storage accounts with public blob access enabled allow unauthenticated " + "read access to blob data over the internet. This setting can expose " + "sensitive files, backups, or configuration data to any external actor." +) +REMEDIATION = ( + "Disable public blob access on the storage account. " + "Navigate to Storage Account > Configuration > Blob public access and set it to Disabled." +) +PLAYBOOK = "playbooks/cli/fix_az_stor_001.sh" + + +def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: + """Detect storage accounts with allow_blob_public_access set to True.""" + findings: List[Dict[str, Any]] = [] + + for account in azure_client.get_storage_accounts(): + if getattr(account, "allow_blob_public_access", False): + findings.append({ + "rule_id": RULE_ID, + "rule_name": RULE_NAME, + "severity": SEVERITY, + "category": CATEGORY, + "resource_id": account.id, + "resource_name": account.name, + "resource_type": "Microsoft.Storage/storageAccounts", + "description": DESCRIPTION, + "remediation": REMEDIATION, + "playbook": PLAYBOOK, + "frameworks": FRAMEWORKS, + }) + + return findings diff --git a/scanner/rules/az_stor_002.py b/scanner/rules/az_stor_002.py new file mode 100644 index 00000000..fd3b1d2c --- /dev/null +++ b/scanner/rules/az_stor_002.py @@ -0,0 +1,42 @@ +"""AZ-STOR-002: Storage account not configured for HTTPS-only traffic.""" + +from typing import Any, Dict, List + +RULE_ID = "AZ-STOR-002" +RULE_NAME = "Storage Account Allows HTTP Traffic (Not HTTPS-Only)" +SEVERITY = "HIGH" +CATEGORY = "Storage" +FRAMEWORKS = {"CIS": "3.1", "NIST": "PR.DS-2", "ISO27001": "A.10.1.1"} +DESCRIPTION = ( + "Storage accounts that do not enforce HTTPS-only traffic allow data to be " + "transmitted in plaintext over HTTP. This exposes credentials and data to " + "man-in-the-middle attacks and interception." +) +REMEDIATION = ( + "Enable the 'Secure transfer required' setting on the storage account. " + "Navigate to Storage Account > Configuration > Secure transfer required and enable it." +) +PLAYBOOK = "playbooks/cli/fix_az_stor_002.sh" + + +def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: + """Detect storage accounts where enable_https_traffic_only is False.""" + findings: List[Dict[str, Any]] = [] + + for account in azure_client.get_storage_accounts(): + if not getattr(account, "enable_https_traffic_only", True): + findings.append({ + "rule_id": RULE_ID, + "rule_name": RULE_NAME, + "severity": SEVERITY, + "category": CATEGORY, + "resource_id": account.id, + "resource_name": account.name, + "resource_type": "Microsoft.Storage/storageAccounts", + "description": DESCRIPTION, + "remediation": REMEDIATION, + "playbook": PLAYBOOK, + "frameworks": FRAMEWORKS, + }) + + return findings From 053be0388c2a8ff659c600e514bb5a39d6953642 Mon Sep 17 00:00:00 2001 From: Vishnu Ajith <86302373+Vishnu2707@users.noreply.github.com> Date: Sat, 25 Apr 2026 15:58:00 +0100 Subject: [PATCH 05/15] docs: replace ASCII architecture with interactive Mermaid diagram --- README.md | 58 ++++++++++++++++++++++--------------------------------- 1 file changed, 23 insertions(+), 35 deletions(-) diff --git a/README.md b/README.md index 19783bc7..e2a19005 100644 --- a/README.md +++ b/README.md @@ -32,43 +32,31 @@ Startups, SMEs, universities, and student teams are left with **zero visibility* --- -## Architecture - -``` -┌─────────────────────────────────────────────────────┐ -│ React Dashboard │ -│ (Azure Static Web Apps — Free) │ -└──────────────────────┬──────────────────────────────┘ - │ -┌──────────────────────▼──────────────────────────────┐ -│ Flask REST API │ -│ (Azure App Service F1 — Free) │ -└────┬──────────────┬──────────────────┬──────────────┘ - │ │ │ -┌────▼────┐ ┌──────▼──────┐ ┌───────▼───────┐ -│ Scanner │ │ Compliance │ │ Remediation │ -│ Engine │ │ Mapper │ │ Playbooks │ -│(Python) │ │ (Python) │ │ (ARM/TF/CLI) │ -└────┬────┘ └──────┬──────┘ └───────────────┘ - │ │ -┌────▼──────────────▼──────────────────────────────────┐ -│ PostgreSQL Database │ -│ (findings, rules, history, playbooks) │ -└──────────────────────────────────────────────────────┘ - │ -┌──────────────────────▼──────────────────────────────┐ -│ Azure Monitor + Sentinel │ -│ (real-time alerting, SIEM integration) │ -└──────────────────────────────────────────────────────┘ - │ -┌──────────────────────▼──────────────────────────────┐ -│ Azure Subscription │ -│ (target environment being scanned via SDK) │ -└─────────────────────────────────────────────────────┘ +## 🏗️ Architecture + +```mermaid +flowchart TD + A["🌐 React Dashboard\nAzure Static Web Apps — Free"] + B["⚙️ Flask REST API\nAzure App Service F1 — Free"] + C["🔍 Scanner Engine\nPython + Azure SDK"] + D["📋 Compliance Mapper\nCIS · NIST · ISO 27001"] + E["🔧 Remediation Playbooks\nARM · Terraform · CLI"] + F["🗄️ PostgreSQL Database\nFindings · Rules · History · Scans"] + G["🛡️ Azure Monitor + Sentinel\nReal-time Alerting · SIEM · KQL Rules"] + H["☁️ Azure Subscription\nTarget environment scanned via SDK"] + + A -->|REST calls| B + B --> C + B --> D + B --> E + C --> F + D --> F + E --> F + F --> G + C -->|Azure SDK| H + G -->|Alerts| A ``` ---- - ## Tech Stack | Layer | Technology | Cost | From b31ecb7dd99ab98d88591bd1f3a2b04b6e7af2c5 Mon Sep 17 00:00:00 2001 From: Tanvir Farhad Date: Sat, 2 May 2026 12:30:45 +0100 Subject: [PATCH 06/15] =?UTF-8?q?feat:=20Sentinel=20integration=20?= =?UTF-8?q?=E2=80=94=20ingest.py,=204=20KQL=20rules,=20setup=20guide=20(#1?= =?UTF-8?q?2)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: add sentinel/ingest.py — Log Analytics ingestion via HMAC-SHA256 * feat: add sentinel/__init__.py * feat: add KQL rule — HIGH severity finding detected * feat: add KQL rule — misconfiguration wave detection * feat: add KQL rule — new resource type critical detection * Delete sentinel/rules directory * Create rules * Delete sentinel/rules * Add KQL rule for high severity findings * Add Misconfiguration Wave detection rule * Add KQL rule for persistent misconfiguration detection * Add KQL rule for new critical resource types This rule identifies new resource types with critical findings that have occurred in the last 24 hours, excluding known types from the last 30 days. * Add script to generate test findings in JSON format This script generates test findings related to security compliance and saves them in a JSON file. * Add Sentinel integration test plan and results Added a comprehensive test plan for Sentinel integration, detailing test objectives, results, and acceptance criteria for various KQL rules and data ingestion. * docs: add sentinel integration setup guide Added a comprehensive setup guide for integrating Sentinel with Azure, covering prerequisites, workspace creation, activation, environment variable setup, ingestion, log verification, KQL rules deployment, and incident verification. --- docs/sentinel-setup.md | 67 +++++++++ sentinel/TEST_PLAN.md | 135 ++++++++++++++++++ sentinel/__init__.py | 1 + sentinel/ingest.py | 79 ++++++++++ sentinel/rules/high_severity_finding.kql | 15 ++ sentinel/rules/misconfiguration_wave.kql | 18 +++ sentinel/rules/new_resource_type_critical.kql | 18 +++ .../rules/persistent_misconfiguration.kql | 17 +++ sentinel/tests/generate_test_findings.py | 22 +++ 9 files changed, 372 insertions(+) create mode 100644 docs/sentinel-setup.md create mode 100644 sentinel/TEST_PLAN.md create mode 100644 sentinel/__init__.py create mode 100644 sentinel/ingest.py create mode 100644 sentinel/rules/high_severity_finding.kql create mode 100644 sentinel/rules/misconfiguration_wave.kql create mode 100644 sentinel/rules/new_resource_type_critical.kql create mode 100644 sentinel/rules/persistent_misconfiguration.kql create mode 100644 sentinel/tests/generate_test_findings.py diff --git a/docs/sentinel-setup.md b/docs/sentinel-setup.md new file mode 100644 index 00000000..fcf7e29f --- /dev/null +++ b/docs/sentinel-setup.md @@ -0,0 +1,67 @@ +# Sentinel Integration Setup Guide + +## Prerequisites +- Azure account (free trial at azure.microsoft.com/free) +- Python 3.9+ +- Azure CLI installed + +## Part 1 - Create Log Analytics Workspace + +az group create --name openshield-rg --location uksouth + +az monitor log-analytics workspace create --resource-group openshield-rg --workspace-name openshield-laws --location uksouth --retention-time 30 + +Get Workspace ID: +az monitor log-analytics workspace show --resource-group openshield-rg --workspace-name openshield-laws --query customerId --output tsv + +Get Shared Key: +az monitor log-analytics workspace get-shared-keys --resource-group openshield-rg --workspace-name openshield-laws --query primarySharedKey --output tsv + +## Part 2 - Activate Sentinel + +az extension add --name sentinel + +az sentinel onboarding-state create --resource-group openshield-rg --workspace-name openshield-laws --name default + +## Part 3 - Set Environment Variables + +export SENTINEL_WORKSPACE_ID="your-workspace-id" +export SENTINEL_SHARED_KEY="your-shared-key" +export SENTINEL_LOG_TYPE="OpenShieldFindings" +export AZURE_SUBSCRIPTION_ID="your-subscription-id" +export AZURE_TENANT_ID="your-tenant-id" +export AZURE_CLIENT_ID="your-client-id" +export AZURE_CLIENT_SECRET="your-client-secret" + +## Part 4 - Run Ingestion + +Install dependencies: +pip install requests + +Generate test findings: +python3 sentinel/tests/generate_test_findings.py + +Push findings to Sentinel: +python3 sentinel/ingest.py scanner/output/test_findings.json scan-001 + +## Part 5 - Verify in Sentinel Logs + +Run this query in Log Analytics: +OpenShieldFindings_CL | take 10 + +If you see rows the ingestion is working correctly. + +## Part 6 - Deploy KQL Rules in Sentinel Analytics + +Go to Microsoft Sentinel or Microsoft Defender XDR and navigate to Analytics. Create a Scheduled query rule for each file in sentinel/rules/ + +high_severity_finding.kql - Severity High - Run every 1 hour +misconfiguration_wave.kql - Severity High - Run every 2 hours +persistent_misconfiguration.kql - Severity Medium - Run every 24 hours +new_resource_type_critical.kql - Severity Critical - Run every 1 hour + +Set alert threshold to greater than 0 for all rules. + +## Part 7 - Verify Incidents + +Go to Incidents in Sentinel or Microsoft Defender XDR. Within a few hours of deploying the rules you should see OpenShield incidents appearing automatically. diff --git a/sentinel/TEST_PLAN.md b/sentinel/TEST_PLAN.md new file mode 100644 index 00000000..c32a26c4 --- /dev/null +++ b/sentinel/TEST_PLAN.md @@ -0,0 +1,135 @@ +# Sentinel Integration - Test Plan and Results + +Branch: feat/sentinel-integration +Issue: #4 +Tested by: TFT444 +Date: 28 April 2026 +Workspace: soc-siem-log (soc-lab-rg, UK South) + +--- + +## Test Environment + +- Azure Subscription: Azure subscription 1 +- Log Analytics Workspace: soc-siem-log +- Resource Group: soc-lab-rg +- Location: UK South +- Sentinel Status: Active - connected to Microsoft Defender XDR +- Custom Log Table: OpenShieldFindings_CL +- Service Principal: openshield-scanner-sp (Reader role only) + +--- + +## Test 1 - Data Ingestion + +Objective: Confirm findings from scanner reach Log Analytics + +Result: PASS + +12 findings confirmed in OpenShieldFindings_CL table. Table created automatically on first ingestion. All fields correctly mapped including Severity_s, RuleName_s, ResourceName_s, CisControl_s. + +--- + +## Test 2 - KQL Rule 1: HIGH Severity Finding Detected + +Objective: Rule fires on any HIGH or CRITICAL finding + +Result: PASS + +7 distinct findings returned: + +- Unencrypted managed disk - Critical - vm-disk-001 +- NSG allows SSH from internet - High - nsg-open-ssh +- Key Vault purge protection disabled - High - kv-nopurge +- SQL Server TDE disabled - High - sql-no-tde +- App Service HTTP not disabled - High - webapp-http +- Container registry admin enabled - High - acr-admin +- Overprivileged service principal - High - sp-contributor + +--- + +## Test 3 - KQL Rule 2: Misconfiguration Wave + +Objective: Rule fires when 5 or more HIGH findings appear in a single scan + +Result: PASS + +- Scan ID: scan-openshield-001 +- Total HIGH/CRITICAL findings: 12 +- Unique rules triggered: 10 +- Wave Score: 120 + +Wave score of 120 confirmed. Rule correctly identifies bulk misconfiguration event. + +--- + +## Test 4 - KQL Rule 3: Persistent Misconfiguration + +Objective: Rule fires when same resource flagged across 3 or more consecutive scans + +Setup: Ingested findings under 4 scan IDs - scan-001, scan-002, scan-003, scan-004 + +Result: PASS + +10 resources returned with ScanCount = 4. Escalation logic confirmed. sp-contributor reached ScanCount = 6 and P1 flag activated automatically. + +--- + +## Test 5 - KQL Rule 4: New Resource Type with Critical Finding + +Objective: Rule fires when unknown resource type appears with CRITICAL finding + +Result: EXPECTED - No results in same-day test environment + +All test data ingested within same session so all resource types exist in both windows. Rule functions correctly in production environments running 30 or more days. Microsoft.ContainerInstance/containerGroups would trigger this rule in a live environment. + +--- + +## Test 6 - Sentinel Analytics Rules Deployment + +Objective: All 4 rules deployed as Scheduled Analytics Rules in Microsoft Defender XDR + +Result: PASS + +- OpenShield - HIGH Severity Finding Detected - High - Enabled - 1 hour - Initial Access +- OpenShield - Misconfiguration Wave - High - Enabled - 2 hours - Impact +- OpenShield - Persistent Misconfiguration - Medium - Enabled - 24 hours - Persistence +- OpenShield - New Resource Type Critical - High - Enabled - 1 hour - Discovery + +All 4 rules confirmed active in Microsoft Defender XDR Analytics dashboard. + +--- + +## Acceptance Criteria + +- Finding from scanner appears as alert in Sentinel: PASS +- KQL rules fire correctly on test data: PASS +- Setup guide works end to end on free Azure trial: PASS + +--- + +## How to Reproduce + +Set environment variables: + +export SENTINEL_WORKSPACE_ID="your-workspace-id" +export SENTINEL_SHARED_KEY="your-shared-key" +export SENTINEL_LOG_TYPE="OpenShieldFindings" + +Install dependencies: + +pip install requests + +Generate test findings: + +python3 sentinel/tests/generate_test_findings.py + +Ingest into Sentinel: + +python3 sentinel/ingest.py scanner/output/test_findings.json scan-001 + +Verify in Sentinel Logs: + +OpenShieldFindings_CL | take 20 + +Copy queries from sentinel/rules into Sentinel Logs and set time range to Last 7 days to verify all rules fire correctly. diff --git a/sentinel/__init__.py b/sentinel/__init__.py new file mode 100644 index 00000000..c15dc57e --- /dev/null +++ b/sentinel/__init__.py @@ -0,0 +1 @@ +"""OpenShield Sentinel integration package.""" diff --git a/sentinel/ingest.py b/sentinel/ingest.py new file mode 100644 index 00000000..02cabfb1 --- /dev/null +++ b/sentinel/ingest.py @@ -0,0 +1,79 @@ +import base64, datetime, hashlib, hmac, json, os, sys, time +import requests + +WORKSPACE_ID = os.environ.get("SENTINEL_WORKSPACE_ID", "") +SHARED_KEY = os.environ.get("SENTINEL_SHARED_KEY", "") +LOG_TYPE = os.environ.get("SENTINEL_LOG_TYPE", "OpenShieldFindings") + +def build_signature(date, content_length): + x_headers = f"x-ms-date:{date}" + string_to_hash = f"POST\n{content_length}\napplication/json\n{x_headers}\n/api/logs" + decoded_key = base64.b64decode(SHARED_KEY) + encoded_hash = base64.b64encode( + hmac.new(decoded_key, string_to_hash.encode("utf-8"), digestmod=hashlib.sha256).digest() + ).decode("utf-8") + return f"SharedKey {WORKSPACE_ID}:{encoded_hash}" + +def normalise(raw, scan_id): + sev_map = {"CRITICAL":4,"HIGH":3,"MEDIUM":2,"LOW":1,"INFO":0} + sev = str(raw.get("severity","MEDIUM")).upper() + return { + "ScanId": scan_id, + "FindingId": raw.get("id",""), + "TimeGenerated": raw.get("detected_at", datetime.datetime.utcnow().isoformat()+"Z"), + "ResourceId": raw.get("resource_id",""), + "ResourceType": raw.get("resource_type",""), + "ResourceName": raw.get("resource_name",""), + "SubscriptionId": raw.get("subscription_id",""), + "ResourceGroup": raw.get("resource_group",""), + "Region": raw.get("region",""), + "RuleId": raw.get("rule_id",""), + "RuleName": raw.get("rule_name",""), + "Severity": sev.capitalize(), + "SeverityScore": sev_map.get(sev,0), + "Description": raw.get("description",""), + "Remediation": raw.get("remediation",""), + "CisControl": raw.get("compliance",{}).get("cis",""), + "NistControl": raw.get("compliance",{}).get("nist",""), + "Source": "OpenShield", + "ToolVersion": raw.get("tool_version","0.1.0"), + } + +def send(records): + body = json.dumps(records).encode("utf-8") + rfc_date = datetime.datetime.utcnow().strftime("%a, %d %b %Y %H:%M:%S GMT") + sig = build_signature(rfc_date, len(body)) + url = f"https://{WORKSPACE_ID}.ods.opinsights.azure.com/api/logs?api-version=2016-04-01" + headers = { + "Content-Type": "application/json", + "Authorization": sig, + "Log-Type": LOG_TYPE, + "x-ms-date": rfc_date, + "time-generated-field": "TimeGenerated", + } + for attempt in range(1, 4): + try: + r = requests.post(url, data=body, headers=headers, timeout=30) + if r.status_code == 200: + print(f"[OK] Ingested {len(records)} findings → {LOG_TYPE}_CL") + return True + print(f"[WARN] Attempt {attempt} — HTTP {r.status_code}: {r.text}") + except Exception as e: + print(f"[WARN] Attempt {attempt} — {e}") + time.sleep(2 ** attempt) + print("[ERROR] Failed after 3 attempts") + return False + +def main(): + path = sys.argv[1] if len(sys.argv) > 1 else "scanner/output/test_findings.json" + scan_id = sys.argv[2] if len(sys.argv) > 2 else datetime.datetime.utcnow().strftime("scan-%Y%m%d-%H%M") + print(f"[INFO] Scan ID: {scan_id}") + with open(path) as f: + data = json.load(f) + findings = data if isinstance(data, list) else data.get("findings", []) + print(f"[INFO] Loaded {len(findings)} findings") + records = [normalise(f, scan_id) for f in findings] + send(records) + +if __name__ == "__main__": + main() diff --git a/sentinel/rules/high_severity_finding.kql b/sentinel/rules/high_severity_finding.kql new file mode 100644 index 00000000..8154c791 --- /dev/null +++ b/sentinel/rules/high_severity_finding.kql @@ -0,0 +1,15 @@ +// Rule: HIGH Severity Finding Detected +// Tactic: Initial Access / Impact +// Severity: High +// Run: Every 1 hour + +OpenShieldFindings_CL +| where TimeGenerated >= ago(1h) +| where Severity_s in ("High", "Critical") +| summarize + FindingCount = count(), + FirstSeen = min(TimeGenerated), + LastSeen = max(TimeGenerated) + by RuleName_s, Severity_s, ResourceName_s, ResourceGroup +| where FindingCount >= 1 +| order by Severity_s asc, LastSeen desc diff --git a/sentinel/rules/misconfiguration_wave.kql b/sentinel/rules/misconfiguration_wave.kql new file mode 100644 index 00000000..903a1813 --- /dev/null +++ b/sentinel/rules/misconfiguration_wave.kql @@ -0,0 +1,18 @@ +// Rule: Misconfiguration Wave +// Tactic: Impact / Defence Evasion +// Severity: High +// Run: Every 2 hours +// Threshold: 5+ HIGH findings in one scan + +let THRESHOLD = 5; +OpenShieldFindings_CL +| where TimeGenerated >= ago(2h) +| where Severity_s in ("High", "Critical") +| summarize + TotalHigh = count(), + UniqueRules = dcount(RuleId_s), + TopRules = make_set(RuleName_s, 5) + by ScanId_s, SubscriptionId +| where TotalHigh >= THRESHOLD +| extend WaveScore = TotalHigh * UniqueRules +| order by WaveScore desc diff --git a/sentinel/rules/new_resource_type_critical.kql b/sentinel/rules/new_resource_type_critical.kql new file mode 100644 index 00000000..89767e16 --- /dev/null +++ b/sentinel/rules/new_resource_type_critical.kql @@ -0,0 +1,18 @@ +// Rule: New Resource Type with Critical Finding +// Tactic: Discovery / Lateral Movement +// Severity: Critical +// Run: Every 1 hour + +let KnownTypes = OpenShieldFindings_CL + | where TimeGenerated between (ago(30d) .. ago(24h)) + | summarize by ResourceType; +OpenShieldFindings_CL +| where TimeGenerated >= ago(24h) +| where Severity_s == "Critical" +| join kind=leftanti (KnownTypes) + on $left.ResourceType == $right.ResourceType +| summarize + FindingCount = count(), + AffectedResources = make_set(ResourceName_s), + TriggeringRules = make_set(RuleName_s) + by ResourceType, SubscriptionId diff --git a/sentinel/rules/persistent_misconfiguration.kql b/sentinel/rules/persistent_misconfiguration.kql new file mode 100644 index 00000000..280c5258 --- /dev/null +++ b/sentinel/rules/persistent_misconfiguration.kql @@ -0,0 +1,17 @@ +// Rule: Persistent Misconfiguration +// Tactic: Persistence +// Severity: Medium +// Run: Every 24 hours + +OpenShieldFindings_CL +| where TimeGenerated >= ago(7d) +| where Severity_s in ("Critical", "High", "Medium") +| summarize + ScanCount = dcount(ScanId_s), + FirstSeen = min(TimeGenerated), + LastSeen = max(TimeGenerated), + DaysOpen = datetime_diff("day", now(), min(TimeGenerated)) + by ResourceId, RuleId_s, RuleName_s, Severity_s, ResourceName_s +| where ScanCount >= 3 +| extend Escalation = iff(ScanCount >= 6, "P1 - Escalate to CISO", "P3 - Assign to team") +| order by ScanCount desc diff --git a/sentinel/tests/generate_test_findings.py b/sentinel/tests/generate_test_findings.py new file mode 100644 index 00000000..69c811c3 --- /dev/null +++ b/sentinel/tests/generate_test_findings.py @@ -0,0 +1,22 @@ +import json, uuid, datetime + +def ts(hours_ago=0): + dt = datetime.datetime.utcnow() - datetime.timedelta(hours=hours_ago) + return dt.strftime("%Y-%m-%dT%H:%M:%SZ") + +findings = [ + {"id": str(uuid.uuid4()), "rule_id": "OS-001", "rule_name": "Public blob storage container", "severity": "HIGH", "resource_id": "/subscriptions/SUB/resourceGroups/rg/providers/Microsoft.Storage/storageAccounts/testblob001", "resource_type": "Microsoft.Storage/storageAccounts", "resource_name": "testblob001", "resource_group": "openshield-rg", "subscription_id": "YOUR_SUBSCRIPTION_ID", "region": "uksouth", "description": "Storage container allows anonymous public read access.", "remediation": "Set publicAccess to None on the container.", "detected_at": ts(0), "tool_version": "0.1.0", "compliance": {"cis": "CIS 3.6", "nist": "SC-7", "iso27001": ""}}, + {"id": str(uuid.uuid4()), "rule_id": "OS-002", "rule_name": "Unencrypted managed disk", "severity": "CRITICAL", "resource_id": "/subscriptions/SUB/resourceGroups/rg/providers/Microsoft.Compute/disks/vm-disk-001", "resource_type": "Microsoft.Compute/disks", "resource_name": "vm-disk-001", "resource_group": "openshield-rg", "subscription_id": "YOUR_SUBSCRIPTION_ID", "region": "uksouth", "description": "Managed disk is not encrypted.", "remediation": "Enable disk encryption.", "detected_at": ts(0), "tool_version": "0.1.0", "compliance": {"cis": "CIS 7.2", "nist": "SC-28", "iso27001": ""}}, + {"id": str(uuid.uuid4()), "rule_id": "OS-003", "rule_name": "NSG allows RDP from internet", "severity": "HIGH", "resource_id": "/subscriptions/SUB/resourceGroups/rg/providers/Microsoft.Network/networkSecurityGroups/nsg-rdp", "resource_type": "Microsoft.Network/networkSecurityGroups", "resource_name": "nsg-open-rdp", "resource_group": "openshield-rg", "subscription_id": "YOUR_SUBSCRIPTION_ID", "region": "uksouth", "description": "NSG allows RDP from 0.0.0.0/0.", "remediation": "Restrict RDP to corporate IP.", "detected_at": ts(0), "tool_version": "0.1.0", "compliance": {"cis": "CIS 6.1", "nist": "SC-7", "iso27001": ""}}, + {"id": str(uuid.uuid4()), "rule_id": "OS-004", "rule_name": "NSG allows SSH from internet", "severity": "HIGH", "resource_id": "/subscriptions/SUB/resourceGroups/rg/providers/Microsoft.Network/networkSecurityGroups/nsg-ssh", "resource_type": "Microsoft.Network/networkSecurityGroups", "resource_name": "nsg-open-ssh", "resource_group": "openshield-rg", "subscription_id": "YOUR_SUBSCRIPTION_ID", "region": "uksouth", "description": "NSG allows SSH from 0.0.0.0/0.", "remediation": "Restrict SSH to corporate IP.", "detected_at": ts(0), "tool_version": "0.1.0", "compliance": {"cis": "CIS 6.2", "nist": "SC-7", "iso27001": ""}}, + {"id": str(uuid.uuid4()), "rule_id": "OS-005", "rule_name": "Key Vault purge protection disabled", "severity": "HIGH", "resource_id": "/subscriptions/SUB/resourceGroups/rg/providers/Microsoft.KeyVault/vaults/kv-nopurge", "resource_type": "Microsoft.KeyVault/vaults", "resource_name": "kv-nopurge", "resource_group": "openshield-rg", "subscription_id": "YOUR_SUBSCRIPTION_ID", "region": "uksouth", "description": "Key Vault purge protection is disabled.", "remediation": "Enable purge protection.", "detected_at": ts(0), "tool_version": "0.1.0", "compliance": {"cis": "CIS 8.4", "nist": "SC-28", "iso27001": ""}}, + {"id": str(uuid.uuid4()), "rule_id": "OS-006", "rule_name": "SQL Server TDE disabled", "severity": "HIGH", "resource_id": "/subscriptions/SUB/resourceGroups/rg/providers/Microsoft.Sql/servers/sql-no-tde", "resource_type": "Microsoft.Sql/servers", "resource_name": "sql-no-tde", "resource_group": "openshield-rg", "subscription_id": "YOUR_SUBSCRIPTION_ID", "region": "uksouth", "description": "SQL TDE disabled.", "remediation": "Enable TDE.", "detected_at": ts(0), "tool_version": "0.1.0", "compliance": {"cis": "CIS 4.1", "nist": "SC-28", "iso27001": ""}}, + {"id": str(uuid.uuid4()), "rule_id": "OS-007", "rule_name": "App Service HTTP not disabled", "severity": "HIGH", "resource_id": "/subscriptions/SUB/resourceGroups/rg/providers/Microsoft.Web/sites/webapp-http", "resource_type": "Microsoft.Web/sites", "resource_name": "webapp-http", "resource_group": "openshield-rg", "subscription_id": "YOUR_SUBSCRIPTION_ID", "region": "uksouth", "description": "App Service allows HTTP.", "remediation": "Enable HTTPS only.", "detected_at": ts(0), "tool_version": "0.1.0", "compliance": {"cis": "CIS 9.2", "nist": "SC-8", "iso27001": ""}}, + {"id": str(uuid.uuid4()), "rule_id": "OS-008", "rule_name": "Container registry admin enabled", "severity": "HIGH", "resource_id": "/subscriptions/SUB/resourceGroups/rg/providers/Microsoft.ContainerRegistry/registries/acr-admin", "resource_type": "Microsoft.ContainerRegistry/registries", "resource_name": "acr-admin", "resource_group": "openshield-rg", "subscription_id": "YOUR_SUBSCRIPTION_ID", "region": "uksouth", "description": "ACR admin account enabled.", "remediation": "Disable admin account.", "detected_at": ts(0), "tool_version": "0.1.0", "compliance": {"cis": "CIS 5.6", "nist": "AC-6", "iso27001": ""}}, + {"id": str(uuid.uuid4()), "rule_id": "OS-009", "rule_name": "Overprivileged service principal", "severity": "HIGH", "resource_id": "/subscriptions/SUB/resourceGroups/rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/sp-contributor", "resource_type": "Microsoft.ManagedIdentity/userAssignedIdentities", "resource_name": "sp-contributor", "resource_group": "openshield-rg", "subscription_id": "YOUR_SUBSCRIPTION_ID", "region": "uksouth", "description": "SP has Contributor at subscription scope.", "remediation": "Scope to minimum resource group.", "detected_at": ts(0), "tool_version": "0.1.0", "compliance": {"cis": "CIS 1.23", "nist": "AC-6", "iso27001": ""}}, + {"id": str(uuid.uuid4()), "rule_id": "OS-010", "rule_name": "Container instance privileged execution", "severity": "CRITICAL", "resource_id": "/subscriptions/SUB/resourceGroups/rg/providers/Microsoft.ContainerInstance/containerGroups/aci-suspicious", "resource_type": "Microsoft.ContainerInstance/containerGroups", "resource_name": "aci-suspicious", "resource_group": "openshield-rg", "subscription_id": "YOUR_SUBSCRIPTION_ID", "region": "uksouth", "description": "Container runs with privileged context.", "remediation": "Remove privileged flag.", "detected_at": ts(0), "tool_version": "0.1.0", "compliance": {"cis": "CIS 5.2", "nist": "CM-7", "iso27001": ""}} +] + +with open("scanner/output/test_findings.json", "w") as f: + json.dump({"findings": findings}, f, indent=2) +print(f"Generated {len(findings)} test findings") From d545744a1cd4a104fe8448e135674d7883bf0657 Mon Sep 17 00:00:00 2001 From: Vishnu Ajith <27vishnu07@gmail.com> Date: Mon, 4 May 2026 22:18:36 +0100 Subject: [PATCH 07/15] fix: add AZ-STOR-003 compliance mappings, correct NIST control to PR.DS-3 --- compliance/frameworks/cis_azure_benchmark.json | 5 +++++ compliance/frameworks/iso27001.json | 5 +++++ compliance/frameworks/nist_csf.json | 5 +++++ 3 files changed, 15 insertions(+) diff --git a/compliance/frameworks/cis_azure_benchmark.json b/compliance/frameworks/cis_azure_benchmark.json index c575a6fb..d5c456dc 100644 --- a/compliance/frameworks/cis_azure_benchmark.json +++ b/compliance/frameworks/cis_azure_benchmark.json @@ -52,6 +52,11 @@ "control_id": "8.5", "control_name": "Ensure the Key Vault is Recoverable", "description": "Azure Key Vault soft delete should be enabled on all Key Vaults. The soft delete feature allows recovery of deleted vaults and vault objects (keys, secrets, certificates) for a configurable retention period (7–90 days), protecting against accidental or malicious deletion." + }, + "AZ-STOR-003": { + "control_id": "3.7", + "control_name": "Ensure that storage accounts have lifecycle management policies configured", + "description": "Storage accounts without lifecycle management policies retain data indefinitely. This increases storage costs, expands the attack surface through accumulation of stale data, and may violate data retention compliance requirements. Lifecycle policies automate the transition and deletion of blobs based on age and access patterns." } } } diff --git a/compliance/frameworks/iso27001.json b/compliance/frameworks/iso27001.json index 42837904..1021811b 100644 --- a/compliance/frameworks/iso27001.json +++ b/compliance/frameworks/iso27001.json @@ -52,6 +52,11 @@ "control_id": "A.17.2.1", "control_name": "Availability of information processing facilities", "description": "Information processing facilities shall be implemented with sufficient redundancy to meet availability requirements. Disabling soft delete on Key Vault removes the ability to recover deleted secrets, keys, and certificates, creating a single point of failure for critical cryptographic material and violating availability and recovery requirements." + }, + "AZ-STOR-003": { + "control_id": "A.8.3.1", + "control_name": "Management of removable media", + "description": "Information stored on Azure storage accounts should be subject to formal lifecycle management controls governing retention and disposal. Storage accounts without lifecycle policies retain data indefinitely with no automated disposal mechanism, violating information handling and disposal requirements under this control." } } } diff --git a/compliance/frameworks/nist_csf.json b/compliance/frameworks/nist_csf.json index 869bc5aa..187978c1 100644 --- a/compliance/frameworks/nist_csf.json +++ b/compliance/frameworks/nist_csf.json @@ -52,6 +52,11 @@ "control_id": "PR.IP-4", "control_name": "Backups of information are conducted, maintained, and tested", "description": "Key material in Azure Key Vault must be recoverable after accidental or malicious deletion. Soft delete provides a recoverable state for secrets, keys, and certificates, supporting backup and recovery requirements for critical cryptographic material." + }, + "AZ-STOR-003": { + "control_id": "PR.DS-3", + "control_name": "Assets are formally managed throughout removal, transfers, and disposition", + "description": "Data stored in Azure storage accounts should be subject to formal lifecycle management policies that govern retention, transition, and deletion. Without these policies, stale data accumulates indefinitely and is never formally dispositioned, violating data management and minimisation requirements." } } } From 6c0c58ebf41e3a99795db06f10c4890b148f7d9d Mon Sep 17 00:00:00 2001 From: Tanvir Farhad Date: Mon, 4 May 2026 22:25:57 +0100 Subject: [PATCH 08/15] docs: add real-world breach scenarios for all 10 starter rules (#15) --- docs/adding-a-rule.md | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/docs/adding-a-rule.md b/docs/adding-a-rule.md index 2d60b2b5..0ef1d791 100644 --- a/docs/adding-a-rule.md +++ b/docs/adding-a-rule.md @@ -214,3 +214,41 @@ Then open a PR. Use the PR template — it will ask you for the rule ID, severit - **Hardcoded subscription ID**: use the `subscription_id` parameter passed to `scan()`, never hardcode. - **Exceptions crashing the scan**: the engine catches unhandled exceptions per rule, but write defensively — use `getattr(obj, "field", default)` for optional SDK attributes. - **Empty `frameworks` dict**: always populate all three keys (CIS, NIST, ISO27001) even if you map to `"N/A"`. + + + +## Real-world impact of each rule + +**AZ-STOR-001 — Public blob access enabled** +This is how 38 million records leaked in the 2021 Power Apps breach — blob containers set to public, no authentication needed, just know the URL and download everything. Attackers don't even need to "hack" anything. Automated tools scan Azure for public blobs constantly. If yours is exposed it will be found, usually within hours. + +**AZ-STOR-002 — Storage account allows unencrypted HTTP** +Any data moving over plain HTTP can be read by anyone on the same network path. This sounds theoretical until you realise most corporate VPNs, shared offices and cloud interconnects are exactly that kind of shared environment. One internal tool uploading customer data over HTTP to Azure storage is all it takes. The fix is one toggle — HTTPS only — but it gets missed constantly. + +**AZ-NET-001 — NSG allows SSH from internet** + +SSH brute force attacks are constant — attackers run automated scripts trying millions of username and password combinations against any open port 22 they find. In 2023 a university research cluster was compromised through an exposed SSH port, with attackers using it to mine cryptocurrency for three months before detection. Restricting SSH to known IP ranges or using Azure Bastion eliminates this risk entirely. + + +**AZ-NET-002 — NSG allows RDP from internet** + +RDP on port 3389 open to 0.0.0.0/0 is one of the most scanned ports on the internet — automated bots find it within minutes of a VM being provisioned. The 2021 Colonial Pipeline attack started with an exposed RDP port and a compromised password. Once an attacker gets in via RDP they have full GUI access to the machine and can move laterally across the entire network. + + +**AZ-IDN-001 — Overprivileged service principal** +Contributor at subscription scope means the service principal can touch everything — every VM, every database, every storage account across the whole subscription. The moment that client secret leaks — through a git commit, a build log, a misconfigured app — the attacker has the keys to the kingdom. This exact pattern showed up in the SolarWinds breach. Least privilege is not optional. + +**AZ-IDN-002 — MFA not enforced on privileged accounts** +Credential stuffing is not sophisticated. Attackers just take leaked password lists from other breaches and try them on Azure AD. Without MFA a matching password is all they need. Microsoft says MFA stops 99.9% of these attacks. A Global Admin account without MFA is genuinely one of the highest risk findings you can have — one leaked password from any other service and your entire tenant is gone. + +**AZ-DB-001 — SQL Server TDE disabled** +The database itself might be behind a firewall, but what about the backups? Backup files get moved around — to blob storage, to tapes, to DR sites. Without TDE the data is sitting in plain text in all of those places. A healthcare company learned this the hard way in 2019 when stolen backup files exposed 2.3 million patient records. The attacker never touched the live database. + +**AZ-DB-002 — SQL Server firewall allows all IPs** +Opening the SQL Server firewall to all IPs is the same as putting your database on the public internet. Shodan and similar tools index these constantly. In 2020 a startup had their production database dumped within days of launching because the firewall rule was still set to 0.0.0.0 from a development config that nobody cleaned up. Lock it to your app service IPs only — nothing else needs direct database access. + +**AZ-CMP-001 — Unencrypted managed disk** +An attacker who gets into your subscription — even temporarily — can snapshot a disk in seconds. They create the snapshot, export it, mount it on their own VM and read everything on it at their leisure. The original VM keeps running, no one notices. A SaaS company found out about this 6 weeks after it happened when their data showed up for sale. The disks were unencrypted so the snapshot was immediately readable. + +**AZ-KV-001 — Key Vault soft delete disabled** +Key Vault is where everything important lives — database passwords, API keys, TLS certificates, encryption keys. Without soft delete an attacker or a disgruntled employee can delete every single secret permanently in about 30 seconds. No recovery, no rollback. A real incident in 2021 saw an employee delete an entire production Key Vault on their last day. The company was down for 6 days rebuilding access from scratch. Soft delete costs nothing to enable. From e4382cd07813f0b130934d4d7a1b7d254f3c5b65 Mon Sep 17 00:00:00 2001 From: PARTH J ROHIT Date: Mon, 4 May 2026 23:18:31 +0100 Subject: [PATCH 09/15] feat: add AZ-KV-002 key vault public access rule and remediation playbook (#14) --- .../frameworks/cis_azure_benchmark.json | 5 ++ compliance/frameworks/iso27001.json | 7 +- compliance/frameworks/nist_csf.json | 5 ++ playbooks/cli/fix_az_kv_002.sh | 21 ++++++ requirements.txt | 2 +- scanner/rules/az_kv_002.py | 71 +++++++++++++++++++ 6 files changed, 109 insertions(+), 2 deletions(-) create mode 100755 playbooks/cli/fix_az_kv_002.sh create mode 100644 scanner/rules/az_kv_002.py diff --git a/compliance/frameworks/cis_azure_benchmark.json b/compliance/frameworks/cis_azure_benchmark.json index d5c456dc..28a5e344 100644 --- a/compliance/frameworks/cis_azure_benchmark.json +++ b/compliance/frameworks/cis_azure_benchmark.json @@ -57,6 +57,11 @@ "control_id": "3.7", "control_name": "Ensure that storage accounts have lifecycle management policies configured", "description": "Storage accounts without lifecycle management policies retain data indefinitely. This increases storage costs, expands the attack surface through accumulation of stale data, and may violate data retention compliance requirements. Lifecycle policies automate the transition and deletion of blobs based on age and access patterns." + }, + "AZ-KV-002": { + "control_id": "8.3", + "control_name": "Ensure that public network access to Key Vault is disabled", + "description": "Azure Key Vault should not allow public network access unless absolutely necessary. Enabling public access increases the attack surface and exposes sensitive secrets, keys, and certificates to potential unauthorized access. Private endpoints should be used to restrict access to trusted networks." } } } diff --git a/compliance/frameworks/iso27001.json b/compliance/frameworks/iso27001.json index 1021811b..df8fc6fc 100644 --- a/compliance/frameworks/iso27001.json +++ b/compliance/frameworks/iso27001.json @@ -57,6 +57,11 @@ "control_id": "A.8.3.1", "control_name": "Management of removable media", "description": "Information stored on Azure storage accounts should be subject to formal lifecycle management controls governing retention and disposal. Storage accounts without lifecycle policies retain data indefinitely with no automated disposal mechanism, violating information handling and disposal requirements under this control." - } + }, + "AZ-KV-002": { + "control_id": "A.13.1.1", + "control_name": "Network controls", + "description": "Networks should be managed and controlled to protect information systems and applications. Allowing public network access to Azure Key Vault increases exposure of sensitive secrets, keys, and certificates to external networks. Access should be restricted to trusted networks using private endpoints or network controls." + } } } diff --git a/compliance/frameworks/nist_csf.json b/compliance/frameworks/nist_csf.json index 187978c1..fe1ca80f 100644 --- a/compliance/frameworks/nist_csf.json +++ b/compliance/frameworks/nist_csf.json @@ -57,6 +57,11 @@ "control_id": "PR.DS-3", "control_name": "Assets are formally managed throughout removal, transfers, and disposition", "description": "Data stored in Azure storage accounts should be subject to formal lifecycle management policies that govern retention, transition, and deletion. Without these policies, stale data accumulates indefinitely and is never formally dispositioned, violating data management and minimisation requirements." + }, + "AZ-KV-002": { + "control_id": "AC-17", + "control_name": "Remote Access", + "description": "Remote access to systems should be controlled, monitored, and restricted. Allowing public network access to Azure Key Vault increases exposure of sensitive secrets, keys, and certificates to external networks. Access should be limited to trusted networks using private endpoints or network restrictions." } } } diff --git a/playbooks/cli/fix_az_kv_002.sh b/playbooks/cli/fix_az_kv_002.sh new file mode 100755 index 00000000..4c69ef37 --- /dev/null +++ b/playbooks/cli/fix_az_kv_002.sh @@ -0,0 +1,21 @@ +#!/bin/bash + +set -euo pipefail + +VAULT_NAME="${1:-}" +RESOURCE_GROUP="${2:-}" + +if [[ -z "$VAULT_NAME" || -z "$RESOURCE_GROUP" ]]; then + echo "Usage: $0 " + exit 1 +fi + +echo "Disabling public network access for Key Vault: $VAULT_NAME (RG: $RESOURCE_GROUP)" + +az keyvault update \ + --name "$VAULT_NAME" \ + --resource-group "$RESOURCE_GROUP" \ + --public-network-access Disabled + +echo "Public network access disabled successfully." +echo "Next step: Configure a private endpoint for full protection." \ No newline at end of file diff --git a/requirements.txt b/requirements.txt index ee813479..10eff46b 100644 --- a/requirements.txt +++ b/requirements.txt @@ -13,4 +13,4 @@ azure-monitor-ingestion==1.0.3 psycopg2-binary==2.9.9 python-dotenv==1.0.0 pyjwt==2.8.0 -requests==2.31.0 +requests==2.31.0 \ No newline at end of file diff --git a/scanner/rules/az_kv_002.py b/scanner/rules/az_kv_002.py new file mode 100644 index 00000000..159d789a --- /dev/null +++ b/scanner/rules/az_kv_002.py @@ -0,0 +1,71 @@ +"""AZ-KV-002: Key Vault allows public network access without private endpoint.""" + +from typing import Any, Dict, List + +RULE_ID = "AZ-KV-002" +RULE_NAME = "Key Vault Allows Public Network Access Without Private Endpoint" +SEVERITY = "HIGH" +CATEGORY = "Key Vault" +FRAMEWORKS = { + "CIS": "8.3", + "NIST": "AC-17", + "ISO27001": "A.13.1.1" +} + +DESCRIPTION = ( + "The Azure Key Vault is accessible over the public internet without a private endpoint configured. " + "This increases the risk of unauthorized access to sensitive secrets, keys, and certificates." +) + +REMEDIATION = ( + "Disable public network access for the Key Vault and configure a private endpoint " + "to restrict access to trusted virtual networks." +) + +PLAYBOOK = "playbooks/cli/fix_az_kv_002.sh" + + +def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: + """Detect Key Vaults with public network access enabled and no private endpoint configured.""" + findings: List[Dict[str, Any]] = [] + + for vault in azure_client.get_key_vaults(): + props = getattr(vault, "properties", None) + if not props: + continue + + # Handle SDK inconsistencies (snake_case vs camelCase) + public_access = getattr(props, "public_network_access", None) + if public_access is None: + public_access = getattr(props, "publicNetworkAccess", None) + + private_endpoints = getattr(props, "private_endpoint_connections", None) + if private_endpoints is None: + private_endpoints = getattr(props, "privateEndpointConnections", None) + + # Normalize values safely + is_public = str(public_access).lower() in ("enabled", "true", "1") + has_private_endpoint = bool(private_endpoints) and len(private_endpoints) > 0 + + if is_public and not has_private_endpoint: + parsed = azure_client.parse_resource_id(vault.id) + + findings.append({ + "rule_id": RULE_ID, + "rule_name": RULE_NAME, + "severity": SEVERITY, + "category": CATEGORY, + "resource_id": vault.id, + "resource_name": vault.name, + "resource_type": "Microsoft.KeyVault/vaults", + "description": DESCRIPTION, + "remediation": REMEDIATION, + "playbook": PLAYBOOK, + "frameworks": FRAMEWORKS, + "metadata": { + "resource_group": parsed.get("resource_group", ""), + "location": getattr(vault, "location", ""), + }, + }) + + return findings \ No newline at end of file From e8fed83e2ea085f4b7f29c91773b65bf92384d8e Mon Sep 17 00:00:00 2001 From: Vishnu Ajith <27vishnu07@gmail.com> Date: Mon, 4 May 2026 23:36:09 +0100 Subject: [PATCH 10/15] docs: update README with rule count, roadmap progress and contributors --- README.md | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index e2a19005..93abb1f2 100644 --- a/README.md +++ b/README.md @@ -150,18 +150,32 @@ All contributors get credited in our [CONTRIBUTORS.md](CONTRIBUTORS.md). ## 📍 Roadmap - [x] Project scaffolding -- [ ] Core scanner engine (Azure SDK integration) -- [ ] First 10 misconfiguration rules -- [ ] Flask API + PostgreSQL schema +- [x] Core scanner engine (Azure SDK integration) +- [x] 11 scan rules +- [x] Flask API + PostgreSQL schema - [ ] React dashboard MVP - [ ] CIS Benchmark compliance mapping -- [ ] Sentinel alert integration +- [x] Sentinel alert integration +- [x] Real-world breach scenarios documented +- [x] First external contributor PR merged - [ ] Remediation playbook library - [ ] NIST CSF + ISO 27001 mappings - [ ] Multi-cloud support (AWS, GCP) --- +## Contributors + +Thanks to everyone who has contributed to OpenShield. + +| Contributor | GitHub | Contribution | +|---|---|---| +| Vishnu Ajith | @Vishnu2707 | Architecture, core scanner, Sentinel wiring | +| TFT444 | @TFT444 | Sentinel integration, 8 network rules, breach scenarios | +| Parth | @parthrohit22 | AZ-KV-002 Key Vault public access rule | + +--- + ## 📄 License MIT — free to use, modify, and distribute. From 35312d467de23008caa3c54c41305f478caa9fd9 Mon Sep 17 00:00:00 2001 From: Tanvir Farhad Date: Mon, 4 May 2026 23:53:11 +0100 Subject: [PATCH 11/15] feat: add network security rules AZ-NET-003 to AZ-NET-010 (#16) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Add az_net_003.py to check NSG rules for port 443 This script detects Network Security Groups (NSGs) with unrestricted inbound access on port 443 and provides remediation guidance. * Add AZ-NET-004 rule for empty NSG detection This script detects Network Security Groups (NSGs) that have no custom security rules configured, providing details for remediation. * Add AZ-NET-005 rule for DDoS protection check This script detects virtual networks in Azure that do not have DDoS protection enabled and provides remediation steps. * feat: add rule AZ-NET-006 — public IP unassociated with any resource This rule detects public IP addresses that are not associated with any resource, providing details for remediation. * feat: add rule AZ-NET-007 — Application Gateway without WAF enabled This rule detects Application Gateways that do not have WAF enabled, logging findings and providing remediation steps. * feat: add rule AZ-NET-008 — load balancer with no backend pool This rule detects load balancers in Azure that are not configured with a backend pool, indicating potential misconfiguration or unnecessary costs. * feat: add rule AZ-NET-009 — VPN gateway using outdated IKE version This script detects VPN gateways using the outdated IKEv1 protocol and provides remediation steps to migrate to IKEv2. * feat: add rule AZ-NET-010 — subnet with no NSG attached This script detects subnets in Azure that do not have a Network Security Group (NSG) attached, logging findings and providing remediation guidance. * feat: add playbook fix_az_net_003.sh This script updates the NSG rule to restrict inbound traffic on port 443 to a specified IP range. * feat: add playbook fix_az_net_004.sh This script adds a default deny-all inbound rule to a specified NSG. * feat: add playbook fix_az_net_005.sh This script enables DDoS protection on a specified virtual network in Azure. It checks for required parameters and provides usage instructions if they are missing. * feat: add playbook fix_az_net_006.sh This script deletes unassociated public IP addresses in Azure. * feat: add playbook fix_az_net_007.sh This script enables WAF on an Application Gateway, ensuring compliance with the AZ-NET-007 rule. * feat: add playbook fix_az_net_008.sh Script to remediate AZ-NET-008 by deleting empty load balancers. * feat:add script to update VPN connection to IKEv2 This script updates a VPN connection to use IKEv2, ensuring compliance with the AZ-NET-009 rule. * feat: add playbook fix_az_net_010.sh This script attaches a specified network security group to a given subnet in a virtual network, ensuring compliance with the AZ-NET-010 rule. * Clarify description and add note for public-facing services Updated the description to clarify the risk of exposing port 443 and added a note regarding public-facing services. * Change severity level from MEDIUM to HIGH * fix: AZ-NET-005 severity changed to LOW — DDoS Standard high cost on small subscriptions * Add note about NetworkManagementClient usage Added a note regarding the creation of NetworkManagementClient directly and suggested a follow-up for consistency. * Add note about NetworkManagementClient usage Added a note regarding the use of NetworkManagementClient and suggested a follow-up for consistency. * Add additional security controls to CIS Azure benchmark * Refine control descriptions in nist_csf.json Updated descriptions for various controls to enhance clarity and specificity regarding remote access management, data protection, and security measures. * fix: add AZ-NET-003 to AZ-NET-010 to ISO27001 compliance framework Updated descriptions for various controls to clarify compliance requirements and improve security guidance. --------- Co-authored-by: Vishnu Ajith <86302373+Vishnu2707@users.noreply.github.com> --- .../frameworks/cis_azure_benchmark.json | 40 ++++++++++ compliance/frameworks/iso27001.json | 65 ++++++++++++--- compliance/frameworks/nist_csf.json | 80 +++++++++++++------ playbooks/cli/fix_az_net_003.sh | 31 +++++++ playbooks/cli/fix_az_net_004.sh | 32 ++++++++ playbooks/cli/fix_az_net_005.sh | 30 +++++++ playbooks/cli/fix_az_net_006.sh | 24 ++++++ playbooks/cli/fix_az_net_007.sh | 32 ++++++++ playbooks/cli/fix_az_net_008.sh | 31 +++++++ playbooks/cli/fix_az_net_009.sh | 26 ++++++ playbooks/cli/fix_az_net_010.sh | 31 +++++++ scanner/rules/az_net_003.py | 63 +++++++++++++++ scanner/rules/az_net_004.py | 50 ++++++++++++ scanner/rules/az_net_005.py | 65 +++++++++++++++ scanner/rules/az_net_006.py | 67 ++++++++++++++++ scanner/rules/az_net_007.py | 68 ++++++++++++++++ scanner/rules/az_net_008.py | 62 ++++++++++++++ scanner/rules/az_net_009.py | 63 +++++++++++++++ scanner/rules/az_net_010.py | 68 ++++++++++++++++ 19 files changed, 894 insertions(+), 34 deletions(-) create mode 100644 playbooks/cli/fix_az_net_003.sh create mode 100644 playbooks/cli/fix_az_net_004.sh create mode 100644 playbooks/cli/fix_az_net_005.sh create mode 100644 playbooks/cli/fix_az_net_006.sh create mode 100644 playbooks/cli/fix_az_net_007.sh create mode 100644 playbooks/cli/fix_az_net_008.sh create mode 100644 playbooks/cli/fix_az_net_009.sh create mode 100644 playbooks/cli/fix_az_net_010.sh create mode 100644 scanner/rules/az_net_003.py create mode 100644 scanner/rules/az_net_004.py create mode 100644 scanner/rules/az_net_005.py create mode 100644 scanner/rules/az_net_006.py create mode 100644 scanner/rules/az_net_007.py create mode 100644 scanner/rules/az_net_008.py create mode 100644 scanner/rules/az_net_009.py create mode 100644 scanner/rules/az_net_010.py diff --git a/compliance/frameworks/cis_azure_benchmark.json b/compliance/frameworks/cis_azure_benchmark.json index 28a5e344..25552aa9 100644 --- a/compliance/frameworks/cis_azure_benchmark.json +++ b/compliance/frameworks/cis_azure_benchmark.json @@ -23,6 +23,46 @@ "control_name": "Ensure that RDP access from the Internet is evaluated and restricted", "description": "Network security groups should not permit unrestricted inbound RDP from the internet. Open RDP ports are a leading cause of ransomware infections and credential-based attacks. Access should be restricted to specific trusted IP ranges or removed in favour of Azure Bastion." }, + "AZ-NET-003": { + "control_id": "9.3", + "control_name": "Ensure that HTTPS access from the Internet is evaluated and restricted", + "description": "Network security groups should not allow unrestricted inbound access on port 443 from the internet. Public web services should be fronted by an Application Gateway with WAF rather than exposing port 443 directly via NSG rules." + }, + "AZ-NET-004": { + "control_id": "9.2", + "control_name": "Ensure that Network Security Groups have rules configured", + "description": "Network Security Groups with no custom rules configured provide no meaningful access control and rely entirely on Azure default rules. Explicit rules following least privilege should be defined for all NSGs." + }, + "AZ-NET-005": { + "control_id": "9.4", + "control_name": "Ensure that DDoS Protection Standard is enabled on all Virtual Networks", + "description": "Azure DDoS Protection Standard provides enhanced DDoS mitigation capabilities for Azure resources. Virtual networks hosting production workloads should have DDoS Protection Standard enabled." + }, + "AZ-NET-006": { + "control_id": "9.1", + "control_name": "Ensure that unassociated public IP addresses are removed", + "description": "Public IP addresses not associated with any resource represent unnecessary attack surface and cost. Unassociated public IPs should be deleted or documented and tagged for review." + }, + "AZ-NET-007": { + "control_id": "9.6", + "control_name": "Ensure that Web Application Firewall is enabled on Application Gateway", + "description": "Application Gateway should have Web Application Firewall enabled in Prevention mode. WAF protects web applications from common exploits including OWASP Top 10 vulnerabilities such as SQL injection and cross-site scripting." + }, + "AZ-NET-008": { + "control_id": "9.1", + "control_name": "Ensure that Load Balancers have backend pools configured", + "description": "Load balancers with no backend pool configured are either misconfigured or leftover resources. They represent unnecessary cost and poor resource hygiene and should be removed or configured correctly." + }, + "AZ-NET-009": { + "control_id": "9.5", + "control_name": "Ensure that VPN gateways use IKEv2", + "description": "VPN gateway connections should use IKEv2 rather than the outdated IKEv1 protocol. IKEv2 provides improved authentication, better performance and built-in NAT traversal support compared to IKEv1." + }, + "AZ-NET-010": { + "control_id": "9.2", + "control_name": "Ensure that all subnets have a Network Security Group attached", + "description": "All subnets except gateway subnets should have a Network Security Group attached. Without an NSG at subnet level, resources in the subnet have no network layer access control and are potentially reachable from other subnets or the internet." + }, "AZ-IDN-001": { "control_id": "1.23", "control_name": "Ensure That No Custom Subscription Owner Roles Are Created", diff --git a/compliance/frameworks/iso27001.json b/compliance/frameworks/iso27001.json index df8fc6fc..a3792b1a 100644 --- a/compliance/frameworks/iso27001.json +++ b/compliance/frameworks/iso27001.json @@ -6,49 +6,95 @@ "AZ-STOR-001": { "control_id": "A.9.4.1", "control_name": "Information access restriction", - "description": "Access to information and application system functions shall be restricted in accordance with the access control policy. Enabling public blob access on storage accounts removes all access restrictions and allows any internet user to read stored data without authentication, directly violating this control." + "description": "Public blob access allows unrestricted access to information stored in Azure Storage. Access to information and application system functions should be restricted in accordance with the access control policy." }, "AZ-STOR-002": { "control_id": "A.10.1.1", "control_name": "Policy on the use of cryptographic controls", - "description": "A policy on the use of cryptographic controls for protection of information shall be developed and implemented. Storage accounts transmitting data over HTTP do not apply encryption in transit, violating the organisation's cryptographic control policy requirement to protect data confidentiality." + "description": "Requiring secure transfer ensures cryptographic controls are applied to data in transit. A policy on the use of cryptographic controls for protection of information should be developed and implemented." }, "AZ-NET-001": { "control_id": "A.13.1.1", "control_name": "Network controls", - "description": "Networks shall be managed and controlled to protect information in systems and applications. NSGs permitting unrestricted SSH from the internet represent a failure of network access control, exposing systems to direct internet-based attack with no network-layer filtering." + "description": "Unrestricted SSH access from the internet violates network access controls. Networks should be managed and controlled to protect information in systems and applications." }, "AZ-NET-002": { "control_id": "A.13.1.1", "control_name": "Network controls", - "description": "Networks shall be managed and controlled to protect information in systems and applications. NSGs permitting unrestricted RDP from the internet represent a critical network control failure, as RDP is the most commonly exploited protocol for ransomware initial access." + "description": "Unrestricted RDP access from the internet violates network access controls. Networks should be managed and controlled to protect information in systems and applications." + }, + "AZ-NET-003": { + "control_id": "A.13.1.1", + "control_name": "Network controls", + "description": "Unrestricted inbound access on port 443 from the internet increases exposure. Networks should be managed and controlled with appropriate restrictions on inbound traffic to protect information systems." + }, + "AZ-NET-004": { + "control_id": "A.13.1.1", + "control_name": "Network controls", + "description": "NSGs with no rules provide no network controls. Networks should be managed and controlled with explicit rules that restrict traffic to what is required for the workload." + }, + "AZ-NET-005": { + "control_id": "A.13.1.1", + "control_name": "Network controls", + "description": "Virtual networks without DDoS protection are vulnerable to availability attacks. Network controls should include protection against denial of service attacks to maintain availability of information systems." + }, + "AZ-NET-006": { + "control_id": "A.13.1.1", + "control_name": "Network controls", + "description": "Unassociated public IP addresses represent unnecessary network exposure. Network resources that are no longer required should be removed to minimise the attack surface." + }, + "AZ-NET-007": { + "control_id": "A.13.1.1", + "control_name": "Network controls", + "description": "Application Gateways without WAF provide no protection against web application attacks. Network controls should include application layer filtering to protect against common web exploits." + }, + "AZ-NET-008": { + "control_id": "A.13.1.1", + "control_name": "Network controls", + "description": "Load balancers with no backend pool are unused resources. Unused network resources should be removed as part of regular network hygiene to maintain an accurate and minimal network topology." + }, + "AZ-NET-009": { + "control_id": "A.13.2.1", + "control_name": "Information transfer policies and procedures", + "description": "VPN connections using IKEv1 use an outdated protocol. Information transfer policies should require the use of current secure protocols to protect data in transit between networks." + }, + "AZ-NET-010": { + "control_id": "A.13.1.1", + "control_name": "Network controls", + "description": "Subnets without NSGs have no network layer access controls. All subnets should have NSGs attached with explicit rules to enforce network controls at the subnet boundary." }, "AZ-IDN-001": { "control_id": "A.9.2.3", "control_name": "Management of privileged access rights", - "description": "The allocation and use of privileged access rights shall be restricted and controlled. Assigning the Owner role to service principals at subscription scope grants excessive privileged access rights beyond operational requirements, violating the principle of least privilege and privileged access management controls." + "description": "Service principals with overly broad permissions violate privileged access management. The allocation and use of privileged access rights should be restricted and controlled." }, "AZ-IDN-002": { "control_id": "A.9.4.2", "control_name": "Secure log-on procedures", - "description": "Where required by the access control policy, access to systems and applications shall be controlled by a secure log-on procedure. Multi-factor authentication is a required component of secure log-on for privileged accounts. Absence of MFA enforcement via Conditional Access violates this control." + "description": "MFA enforces secure log-on for privileged accounts. Where required by the access control policy, access to systems and applications should be controlled by a secure log-on procedure including multi-factor authentication." }, "AZ-DB-001": { "control_id": "A.13.1.1", "control_name": "Network controls", - "description": "Networks shall be managed and controlled to protect information in systems and applications. Database servers with public network access enabled lack the network-level isolation required to protect sensitive data from direct internet exposure and attack." + "description": "Public network access to PostgreSQL servers should be disabled. Database servers should only be accessible via private network connections with appropriate network controls in place." }, "AZ-DB-002": { "control_id": "A.12.4.1", "control_name": "Event logging", - "description": "Event logs recording user activities, exceptions, faults and information security events shall be produced, kept and regularly reviewed. Disabling SQL Server auditing means that database access events, failed logins, and schema changes are not logged, making incident detection and forensic investigation impossible." + "description": "SQL Server auditing must be enabled to provide event logs. Event logs recording user activities, exceptions, faults and information security events should be produced, kept and regularly reviewed." }, "AZ-CMP-001": { "control_id": "A.13.1.1", "control_name": "Network controls", - "description": "Networks shall be managed and controlled to protect information in systems and applications. Virtual machines with public IPs and no Network Security Group have no network-layer access controls, exposing all ports and services to the internet without any filtering." + "description": "Virtual machines with public IPs and no NSG have unrestricted network access. Network controls should be applied to all compute resources accessible from the internet." }, "AZ-KV-001": { +<<<<<<< feat/network-rules-expansion + "control_id": "A.12.3.1", + "control_name": "Information backup", + "description": "Key Vault soft delete protects against loss of secrets, keys and certificates. Backup copies of information should be taken and tested regularly in accordance with an agreed backup policy." + } +======= "control_id": "A.17.2.1", "control_name": "Availability of information processing facilities", "description": "Information processing facilities shall be implemented with sufficient redundancy to meet availability requirements. Disabling soft delete on Key Vault removes the ability to recover deleted secrets, keys, and certificates, creating a single point of failure for critical cryptographic material and violating availability and recovery requirements." @@ -63,5 +109,6 @@ "control_name": "Network controls", "description": "Networks should be managed and controlled to protect information systems and applications. Allowing public network access to Azure Key Vault increases exposure of sensitive secrets, keys, and certificates to external networks. Access should be restricted to trusted networks using private endpoints or network controls." } +>>>>>>> dev } } diff --git a/compliance/frameworks/nist_csf.json b/compliance/frameworks/nist_csf.json index fe1ca80f..2c4ddf88 100644 --- a/compliance/frameworks/nist_csf.json +++ b/compliance/frameworks/nist_csf.json @@ -6,62 +6,92 @@ "AZ-STOR-001": { "control_id": "PR.AC-3", "control_name": "Remote access is managed", - "description": "Remote access to data assets is controlled. Unauthenticated public blob access on storage accounts violates access management controls by allowing anonymous access to potentially sensitive data without any form of authentication or authorisation." + "description": "Public blob access enables unauthenticated remote access to storage resources. Disabling public access ensures remote access to storage is managed and authenticated." }, "AZ-STOR-002": { "control_id": "PR.DS-2", "control_name": "Data-in-transit is protected", - "description": "Data in transit is protected to prevent interception and tampering. Storage accounts that allow HTTP traffic transmit data in plaintext, violating the requirement to protect data in transit through encryption (TLS)." + "description": "Requiring secure transfer ensures data in transit between clients and Azure Storage is encrypted using HTTPS, protecting against interception and tampering." }, "AZ-NET-001": { "control_id": "PR.AC-3", "control_name": "Remote access is managed", - "description": "Remote access to systems must be controlled. Allowing unrestricted SSH access from the internet bypasses access management controls and exposes systems to unauthorised remote access, brute-force attacks, and exploitation." + "description": "Unrestricted SSH access from the internet allows unmanaged remote access. NSG rules should restrict SSH to known IP ranges to ensure remote access is controlled and monitored." }, "AZ-NET-002": { "control_id": "PR.AC-3", "control_name": "Remote access is managed", - "description": "Remote access to systems must be managed. Allowing unrestricted RDP access from the internet bypasses access management controls and is a primary vector for ransomware delivery and credential-based attacks on Windows systems." + "description": "Unrestricted RDP access from the internet allows unmanaged remote access. NSG rules should restrict RDP to known IP ranges or use Azure Bastion to ensure remote access is controlled." + }, + "AZ-NET-003": { + "control_id": "SC-7", + "control_name": "Boundary Protection", + "description": "Unrestricted inbound access on port 443 from the internet increases the attack surface. Public-facing HTTPS services should be fronted by a WAF-enabled Application Gateway rather than exposed directly via NSG rules." + }, + "AZ-NET-004": { + "control_id": "SC-7", + "control_name": "Boundary Protection", + "description": "NSGs with no custom rules provide no meaningful boundary protection. Explicit least-privilege rules should be defined to control inbound and outbound traffic at the network perimeter." + }, + "AZ-NET-005": { + "control_id": "SC-5", + "control_name": "Denial of Service Protection", + "description": "Virtual networks without DDoS Protection Standard are vulnerable to volumetric denial of service attacks. DDoS Protection Standard provides enhanced mitigation for production workloads." + }, + "AZ-NET-006": { + "control_id": "CM-7", + "control_name": "Least Functionality", + "description": "Unassociated public IP addresses represent unnecessary functionality and attack surface. Resources that are no longer in use should be removed to maintain least functionality." + }, + "AZ-NET-007": { + "control_id": "SI-3", + "control_name": "Malicious Code Protection", + "description": "Application Gateways without WAF enabled provide no protection against web application attacks including OWASP Top 10 vulnerabilities. WAF in Prevention mode should be enabled on all public-facing Application Gateways." + }, + "AZ-NET-008": { + "control_id": "CM-7", + "control_name": "Least Functionality", + "description": "Load balancers with no backend pool configured serve no function and represent unnecessary resources. Unused resources should be removed to maintain least functionality and reduce cost." + }, + "AZ-NET-009": { + "control_id": "SC-8", + "control_name": "Transmission Confidentiality and Integrity", + "description": "VPN connections using IKEv1 use an outdated protocol with known vulnerabilities. IKEv2 should be used for all VPN gateway connections to ensure transmission confidentiality and integrity." + }, + "AZ-NET-010": { + "control_id": "SC-7", + "control_name": "Boundary Protection", + "description": "Subnets without NSGs attached have no network layer access control. All production subnets should have NSGs with explicit rules to enforce boundary protection at the subnet level." }, "AZ-IDN-001": { "control_id": "PR.AC-4", - "control_name": "Access permissions and authorisations are managed, incorporating the principles of least privilege and separation of duties", - "description": "Access to cloud resources should follow the principle of least privilege. Assigning the Owner role to service principals at subscription scope grants excessive permissions that violate least-privilege and separation-of-duties requirements." + "control_name": "Access permissions and authorizations are managed", + "description": "Service principals with overly broad permissions violate least privilege. Access permissions should be scoped to the minimum required for the workload to function." }, "AZ-IDN-002": { - "control_id": "PR.AC-1", - "control_name": "Identities and credentials are issued, managed, verified, revoked, and audited for authorised devices, users and processes", - "description": "Credentials must be managed to ensure only authorised parties can authenticate. Without MFA enforcement, a single compromised password grants full access to administrator accounts, undermining identity management controls." + "control_id": "PR.AC-7", + "control_name": "Users, devices, and other assets are authenticated", + "description": "MFA ensures privileged users are strongly authenticated before accessing Azure resources. Without MFA, a compromised password is sufficient for full administrative access." }, "AZ-DB-001": { "control_id": "PR.AC-3", "control_name": "Remote access is managed", - "description": "Database servers should not be reachable from the public internet without restriction. Public network access to PostgreSQL servers removes the network-based access control layer, exposing the database to direct internet-based attacks." + "description": "Public network access to PostgreSQL servers should be disabled. Database access should be restricted to private networks to ensure remote access is managed and controlled." }, "AZ-DB-002": { - "control_id": "DE.CM-7", - "control_name": "Monitoring for unauthorised personnel, connections, devices, and software is performed", - "description": "Audit logging on SQL servers enables detection of unauthorised access attempts, privilege escalation, and suspicious database activity. Without auditing enabled, security events go undetected and incident investigation is severely limited." + "control_id": "DE.AE-3", + "control_name": "Event data are aggregated and correlated", + "description": "SQL Server auditing must be enabled with sufficient retention to support threat detection and incident investigation. Audit logs provide the event data needed to detect and respond to anomalous database activity." }, "AZ-CMP-001": { "control_id": "PR.AC-3", "control_name": "Remote access is managed", - "description": "Virtual machines accessible from the internet must have compensating network controls. A VM with a public IP and no NSG has all ports exposed to the internet with no filtering, violating remote access management requirements." + "description": "Virtual machines with public IPs and no NSG have unrestricted network access. NSGs should be attached to control inbound and outbound traffic and manage remote access to compute resources." }, "AZ-KV-001": { "control_id": "PR.IP-4", "control_name": "Backups of information are conducted, maintained, and tested", - "description": "Key material in Azure Key Vault must be recoverable after accidental or malicious deletion. Soft delete provides a recoverable state for secrets, keys, and certificates, supporting backup and recovery requirements for critical cryptographic material." - }, - "AZ-STOR-003": { - "control_id": "PR.DS-3", - "control_name": "Assets are formally managed throughout removal, transfers, and disposition", - "description": "Data stored in Azure storage accounts should be subject to formal lifecycle management policies that govern retention, transition, and deletion. Without these policies, stale data accumulates indefinitely and is never formally dispositioned, violating data management and minimisation requirements." - }, - "AZ-KV-002": { - "control_id": "AC-17", - "control_name": "Remote Access", - "description": "Remote access to systems should be controlled, monitored, and restricted. Allowing public network access to Azure Key Vault increases exposure of sensitive secrets, keys, and certificates to external networks. Access should be limited to trusted networks using private endpoints or network restrictions." + "description": "Key Vault soft delete protects against accidental or malicious deletion of secrets, keys and certificates. Without soft delete, deleted vault objects cannot be recovered, causing potential data loss." } } } diff --git a/playbooks/cli/fix_az_net_003.sh b/playbooks/cli/fix_az_net_003.sh new file mode 100644 index 00000000..36e8b0ab --- /dev/null +++ b/playbooks/cli/fix_az_net_003.sh @@ -0,0 +1,31 @@ +#!/bin/bash +# OpenShield Remediation Playbook +# Rule: AZ-NET-003 — NSG allows unrestricted inbound on port 443 +# Usage: ./fix_az_net_003.sh +# Severity: MEDIUM + +set -e + +RESOURCE_GROUP=$1 +NSG_NAME=$2 +RULE_NAME=$3 +ALLOWED_IP=$4 + +if [ -z "$RESOURCE_GROUP" ] || [ -z "$NSG_NAME" ] || [ -z "$RULE_NAME" ] || [ -z "$ALLOWED_IP" ]; then + echo "Usage: $0 " + echo "" + echo "Example:" + echo " $0 my-rg my-nsg allow-https 203.0.113.0/24" + exit 1 +fi + +echo "Restricting port 443 inbound rule '$RULE_NAME' in NSG '$NSG_NAME'..." + +az network nsg rule update \ + --resource-group "$RESOURCE_GROUP" \ + --nsg-name "$NSG_NAME" \ + --name "$RULE_NAME" \ + --source-address-prefixes "$ALLOWED_IP" + +echo "✅ Remediation complete — port 443 now restricted to $ALLOWED_IP" +echo "⚠️ Verify your application still functions correctly after this change." diff --git a/playbooks/cli/fix_az_net_004.sh b/playbooks/cli/fix_az_net_004.sh new file mode 100644 index 00000000..cad9fbbb --- /dev/null +++ b/playbooks/cli/fix_az_net_004.sh @@ -0,0 +1,32 @@ +#!/bin/bash +# OpenShield Remediation Playbook +# Rule: AZ-NET-004 — NSG with no rules configured +# Usage: ./fix_az_net_004.sh +# Severity: MEDIUM + +set -e + +RESOURCE_GROUP=$1 +NSG_NAME=$2 + +if [ -z "$RESOURCE_GROUP" ] || [ -z "$NSG_NAME" ]; then + echo "Usage: $0 " + exit 1 +fi + +echo "Adding default deny-all inbound rule to NSG '$NSG_NAME'..." + +az network nsg rule create \ + --resource-group "$RESOURCE_GROUP" \ + --nsg-name "$NSG_NAME" \ + --name "DenyAllInbound" \ + --priority 4096 \ + --direction Inbound \ + --access Deny \ + --protocol "*" \ + --source-address-prefixes "*" \ + --destination-address-prefixes "*" \ + --destination-port-ranges "*" + +echo "✅ Default deny-all inbound rule added to $NSG_NAME" +echo "⚠️ Now add specific allow rules for your workload traffic." diff --git a/playbooks/cli/fix_az_net_005.sh b/playbooks/cli/fix_az_net_005.sh new file mode 100644 index 00000000..69905ed5 --- /dev/null +++ b/playbooks/cli/fix_az_net_005.sh @@ -0,0 +1,30 @@ +#!/bin/bash +# OpenShield Remediation Playbook +# Rule: AZ-NET-005 — Virtual network with no DDoS protection enabled +# Usage: ./fix_az_net_005.sh +# Severity: MEDIUM + +set -e + +RESOURCE_GROUP=$1 +VNET_NAME=$2 +DDOS_PLAN_NAME=$3 + +if [ -z "$RESOURCE_GROUP" ] || [ -z "$VNET_NAME" ] || [ -z "$DDOS_PLAN_NAME" ]; then + echo "Usage: $0 " + echo "" + echo "To create a new DDoS protection plan first:" + echo " az network ddos-protection create --resource-group --name " + exit 1 +fi + +echo "Enabling DDoS protection on VNet '$VNET_NAME'..." + +az network vnet update \ + --resource-group "$RESOURCE_GROUP" \ + --name "$VNET_NAME" \ + --ddos-protection true \ + --ddos-protection-plan "$DDOS_PLAN_NAME" + +echo "✅ DDoS Protection Standard enabled on $VNET_NAME" +echo "⚠️ DDoS Protection Standard incurs additional cost — review Azure pricing." diff --git a/playbooks/cli/fix_az_net_006.sh b/playbooks/cli/fix_az_net_006.sh new file mode 100644 index 00000000..6073122d --- /dev/null +++ b/playbooks/cli/fix_az_net_006.sh @@ -0,0 +1,24 @@ +#!/bin/bash +# OpenShield Remediation Playbook +# Rule: AZ-NET-006 — Public IP address unassociated with any resource +# Usage: ./fix_az_net_006.sh +# Severity: LOW + +set -e + +RESOURCE_GROUP=$1 +PUBLIC_IP_NAME=$2 + +if [ -z "$RESOURCE_GROUP" ] || [ -z "$PUBLIC_IP_NAME" ]; then + echo "Usage: $0 " + exit 1 +fi + +echo "Deleting unassociated public IP '$PUBLIC_IP_NAME'..." + +az network public-ip delete \ + --resource-group "$RESOURCE_GROUP" \ + --name "$PUBLIC_IP_NAME" + +echo "✅ Public IP '$PUBLIC_IP_NAME' deleted successfully." +echo "⚠️ If this IP was reserved for future use, reassign it to a resource instead of deleting." diff --git a/playbooks/cli/fix_az_net_007.sh b/playbooks/cli/fix_az_net_007.sh new file mode 100644 index 00000000..5e39efed --- /dev/null +++ b/playbooks/cli/fix_az_net_007.sh @@ -0,0 +1,32 @@ +#!/bin/bash +# OpenShield Remediation Playbook +# Rule: AZ-NET-007 — Application Gateway without WAF enabled +# Usage: ./fix_az_net_007.sh +# Severity: HIGH + +set -e + +RESOURCE_GROUP=$1 +AGW_NAME=$2 +WAF_POLICY=$3 + +if [ -z "$RESOURCE_GROUP" ] || [ -z "$AGW_NAME" ] || [ -z "$WAF_POLICY" ]; then + echo "Usage: $0 " + echo "" + echo "To create a WAF policy first:" + echo " az network application-gateway waf-policy create --resource-group --name " + exit 1 +fi + +echo "Enabling WAF on Application Gateway '$AGW_NAME'..." + +az network application-gateway waf-config set \ + --resource-group "$RESOURCE_GROUP" \ + --gateway-name "$AGW_NAME" \ + --enabled true \ + --firewall-mode Prevention \ + --rule-set-type OWASP \ + --rule-set-version 3.2 + +echo "✅ WAF enabled on $AGW_NAME in Prevention mode with OWASP 3.2 rule set." +echo "⚠️ Monitor WAF logs for false positives before relying on Prevention mode in production." diff --git a/playbooks/cli/fix_az_net_008.sh b/playbooks/cli/fix_az_net_008.sh new file mode 100644 index 00000000..014bf607 --- /dev/null +++ b/playbooks/cli/fix_az_net_008.sh @@ -0,0 +1,31 @@ +#!/bin/bash +# OpenShield Remediation Playbook +# Rule: AZ-NET-008 — Load balancer with no backend pool configured +# Usage: ./fix_az_net_008.sh +# Severity: LOW + +set -e + +RESOURCE_GROUP=$1 +LB_NAME=$2 + +if [ -z "$RESOURCE_GROUP" ] || [ -z "$LB_NAME" ]; then + echo "Usage: $0 " + echo "" + echo "Options:" + echo " 1. Delete the load balancer if no longer needed:" + echo " az network lb delete --resource-group --name " + echo "" + echo " 2. Add a backend pool if the load balancer is still required:" + echo " az network lb address-pool create --resource-group --lb-name --name " + exit 1 +fi + +echo "Deleting empty load balancer '$LB_NAME'..." + +az network lb delete \ + --resource-group "$RESOURCE_GROUP" \ + --name "$LB_NAME" + +echo "✅ Load balancer '$LB_NAME' deleted." +echo "⚠️ If this load balancer is still needed, create a backend pool instead of deleting." diff --git a/playbooks/cli/fix_az_net_009.sh b/playbooks/cli/fix_az_net_009.sh new file mode 100644 index 00000000..f6d9e752 --- /dev/null +++ b/playbooks/cli/fix_az_net_009.sh @@ -0,0 +1,26 @@ +#!/bin/bash +# OpenShield Remediation Playbook +# Rule: AZ-NET-009 — VPN gateway using outdated IKE version +# Usage: ./fix_az_net_009.sh +# Severity: HIGH + +set -e + +RESOURCE_GROUP=$1 +CONNECTION_NAME=$2 + +if [ -z "$RESOURCE_GROUP" ] || [ -z "$CONNECTION_NAME" ]; then + echo "Usage: $0 " + exit 1 +fi + +echo "Updating VPN connection '$CONNECTION_NAME' to use IKEv2..." + +az network vpn-connection update \ + --resource-group "$RESOURCE_GROUP" \ + --name "$CONNECTION_NAME" \ + --set connectionProtocol=IKEv2 + +echo "✅ VPN connection '$CONNECTION_NAME' updated to IKEv2." +echo "⚠️ Ensure the remote VPN peer also supports IKEv2 before applying this change." +echo "⚠️ The VPN connection will briefly disconnect during the update." diff --git a/playbooks/cli/fix_az_net_010.sh b/playbooks/cli/fix_az_net_010.sh new file mode 100644 index 00000000..b619e097 --- /dev/null +++ b/playbooks/cli/fix_az_net_010.sh @@ -0,0 +1,31 @@ +#!/bin/bash +# OpenShield Remediation Playbook +# Rule: AZ-NET-010 — Subnet with no network security group attached +# Usage: ./fix_az_net_010.sh +# Severity: HIGH + +set -e + +RESOURCE_GROUP=$1 +VNET_NAME=$2 +SUBNET_NAME=$3 +NSG_NAME=$4 + +if [ -z "$RESOURCE_GROUP" ] || [ -z "$VNET_NAME" ] || [ -z "$SUBNET_NAME" ] || [ -z "$NSG_NAME" ]; then + echo "Usage: $0 " + echo "" + echo "To create a new NSG first:" + echo " az network nsg create --resource-group --name " + exit 1 +fi + +echo "Attaching NSG '$NSG_NAME' to subnet '$SUBNET_NAME' in VNet '$VNET_NAME'..." + +az network vnet subnet update \ + --resource-group "$RESOURCE_GROUP" \ + --vnet-name "$VNET_NAME" \ + --name "$SUBNET_NAME" \ + --network-security-group "$NSG_NAME" + +echo "✅ NSG '$NSG_NAME' attached to subnet '$SUBNET_NAME'." +echo "⚠️ Review NSG rules to ensure only required traffic is permitted." diff --git a/scanner/rules/az_net_003.py b/scanner/rules/az_net_003.py new file mode 100644 index 00000000..54d2ca14 --- /dev/null +++ b/scanner/rules/az_net_003.py @@ -0,0 +1,63 @@ +"""AZ-NET-003: NSG allows unrestricted inbound on port 443.""" + +import logging +from typing import Any, Dict, List + +RULE_ID = "AZ-NET-003" +RULE_NAME = "NSG allows unrestricted inbound on port 443" +SEVERITY = "HIGH" +CATEGORY = "Network" +FRAMEWORKS = {"CIS": "9.3", "NIST": "SC-7", "ISO27001": "A.13.1.1"} +DESCRIPTION = ( + DESCRIPTION = ( + "A Network Security Group has an inbound rule allowing unrestricted access " + "on port 443 from any source (0.0.0.0/0). While HTTPS traffic is encrypted, " + "exposing port 443 to the entire internet unnecessarily increases the attack " + "surface and can expose web services to automated scanning and exploitation attempts. " + "Note: this finding is expected for intentionally public-facing web services. " + "Review manually before remediating — do not auto-remediate without confirming " + "the service is not meant to be publicly accessible." +) +) +REMEDIATION = ( + "Restrict the inbound rule on port 443 to known IP ranges or use an " + "Application Gateway with WAF to front any public-facing HTTPS services. " + "If the service must be public, ensure it is protected by DDoS Standard." +) +PLAYBOOK = "playbooks/cli/fix_az_net_003.sh" + +logger = logging.getLogger(__name__) + + +def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: + """Detect NSGs with unrestricted inbound access on port 443.""" + findings: List[Dict[str, Any]] = [] + + for nsg in azure_client.get_network_security_groups(): + for rule in getattr(nsg, "security_rules", []) or []: + if ( + getattr(rule, "direction", "") == "Inbound" + and getattr(rule, "access", "") == "Allow" + and getattr(rule, "source_address_prefix", "") in ("*", "0.0.0.0/0", "Internet", "Any") + and getattr(rule, "destination_port_range", "") in ("443", "*") + ): + findings.append({ + "rule_id": RULE_ID, + "rule_name": RULE_NAME, + "severity": SEVERITY, + "category": CATEGORY, + "resource_id": getattr(nsg, "id", ""), + "resource_name": getattr(nsg, "name", ""), + "resource_type": "Microsoft.Network/networkSecurityGroups", + "description": DESCRIPTION, + "remediation": REMEDIATION, + "playbook": PLAYBOOK, + "frameworks": FRAMEWORKS, + "metadata": { + "rule_name": getattr(rule, "name", ""), + "source_prefix": getattr(rule, "source_address_prefix", ""), + }, + }) + break + + return findings diff --git a/scanner/rules/az_net_004.py b/scanner/rules/az_net_004.py new file mode 100644 index 00000000..8ce41788 --- /dev/null +++ b/scanner/rules/az_net_004.py @@ -0,0 +1,50 @@ +"""AZ-NET-004: NSG with no rules configured (empty ruleset).""" + +import logging +from typing import Any, Dict, List + +RULE_ID = "AZ-NET-004" +RULE_NAME = "NSG with no rules configured" +SEVERITY = "MEDIUM" +CATEGORY = "Network" +FRAMEWORKS = {"CIS": "9.2", "NIST": "SC-7", "ISO27001": "A.13.1.1"} +DESCRIPTION = ( + "A Network Security Group exists but has no custom security rules configured. " + "An empty NSG relies entirely on Azure default rules which may not meet your " + "security requirements and provides no meaningful access control." +) +REMEDIATION = ( + "Add explicit inbound and outbound rules to the NSG that reflect the " + "principle of least privilege. Deny all traffic by default and only allow " + "what is required for the workload." +) +PLAYBOOK = "playbooks/cli/fix_az_net_004.sh" + +logger = logging.getLogger(__name__) + + +def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: + """Detect NSGs with no custom security rules.""" + findings: List[Dict[str, Any]] = [] + + for nsg in azure_client.get_network_security_groups(): + rules = getattr(nsg, "security_rules", []) or [] + if len(rules) == 0: + findings.append({ + "rule_id": RULE_ID, + "rule_name": RULE_NAME, + "severity": SEVERITY, + "category": CATEGORY, + "resource_id": getattr(nsg, "id", ""), + "resource_name": getattr(nsg, "name", ""), + "resource_type": "Microsoft.Network/networkSecurityGroups", + "description": DESCRIPTION, + "remediation": REMEDIATION, + "playbook": PLAYBOOK, + "frameworks": FRAMEWORKS, + "metadata": { + "rule_count": len(rules), + }, + }) + + return findings diff --git a/scanner/rules/az_net_005.py b/scanner/rules/az_net_005.py new file mode 100644 index 00000000..9f6702c8 --- /dev/null +++ b/scanner/rules/az_net_005.py @@ -0,0 +1,65 @@ +"""AZ-NET-005: Virtual network with no DDoS protection enabled.""" + +import logging +from typing import Any, Dict, List + +RULE_ID = "AZ-NET-005" +RULE_NAME = "Virtual network with no DDoS protection enabled" +SEVERITY = "LOW" +CATEGORY = "Network" +FRAMEWORKS = {"CIS": "9.4", "NIST": "SC-5", "ISO27001": "A.13.1.1"} +DESCRIPTION = ( + "The virtual network does not have Azure DDoS Protection Standard enabled. " + "Without DDoS protection, the network is vulnerable to volumetric attacks " + "that can overwhelm resources and cause service outages." +) +REMEDIATION = ( + "Enable Azure DDoS Protection Standard on the virtual network. " + "DDoS Protection Standard provides enhanced mitigation capabilities " + "and is recommended for all production virtual networks." +) +PLAYBOOK = "playbooks/cli/fix_az_net_005.sh" + +logger = logging.getLogger(__name__) + + +def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: + """Detect virtual networks without DDoS protection enabled.""" + findings: List[Dict[str, Any]] = [] + + try: + # NOTE: This rule creates a NetworkManagementClient directly rather than + # going through azure_client. A get_virtual_networks() method should be + # added to AzureClient in a follow-up PR for consistency. + from azure.mgmt.network import NetworkManagementClient + client = NetworkManagementClient( + azure_client.credential, azure_client.subscription_id + ) + vnets = list(client.virtual_networks.list_all()) + except Exception as exc: + logger.error("Failed to list virtual networks: %s", exc) + return findings + + for vnet in vnets: + ddos = getattr(vnet, "ddos_protection_plan", None) + enable_ddos = getattr(vnet, "enable_ddos_protection", False) + if not ddos and not enable_ddos: + findings.append({ + "rule_id": RULE_ID, + "rule_name": RULE_NAME, + "severity": SEVERITY, + "category": CATEGORY, + "resource_id": getattr(vnet, "id", ""), + "resource_name": getattr(vnet, "name", ""), + "resource_type": "Microsoft.Network/virtualNetworks", + "description": DESCRIPTION, + "remediation": REMEDIATION, + "playbook": PLAYBOOK, + "frameworks": FRAMEWORKS, + "metadata": { + "location": getattr(vnet, "location", ""), + "ddos_protection": enable_ddos, + }, + }) + + return findings diff --git a/scanner/rules/az_net_006.py b/scanner/rules/az_net_006.py new file mode 100644 index 00000000..180f49b9 --- /dev/null +++ b/scanner/rules/az_net_006.py @@ -0,0 +1,67 @@ +"""AZ-NET-006: Public IP address unassociated with any resource.""" + +import logging +from typing import Any, Dict, List + +RULE_ID = "AZ-NET-006" +RULE_NAME = "Public IP address unassociated with any resource" +SEVERITY = "LOW" +CATEGORY = "Network" +FRAMEWORKS = {"CIS": "9.1", "NIST": "CM-7", "ISO27001": "A.13.1.1"} +DESCRIPTION = ( + "A public IP address exists in the subscription but is not associated " + "with any resource such as a VM, load balancer or application gateway. " + "Unassociated public IPs represent unnecessary cost and attack surface " + "and may indicate leftover resources from decommissioned workloads." +) +REMEDIATION = ( + "Delete the unassociated public IP address if it is no longer needed. " + "If it is reserved for future use, document the reason and tag it " + "appropriately so it can be tracked and reviewed regularly." +) +PLAYBOOK = "playbooks/cli/fix_az_net_006.sh" + +logger = logging.getLogger(__name__) + + +def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: + """Detect public IP addresses not associated with any resource.""" + findings: List[Dict[str, Any]] = [] + + try: + # NOTE: This rule creates a NetworkManagementClient directly rather than + # going through azure_client. A get_public_ip_addresses() method should be + # added to AzureClient in a follow-up PR for consistency. + from azure.mgmt.network import NetworkManagementClient + client = NetworkManagementClient( + azure_client.credential, azure_client.subscription_id + ) + public_ips = list(client.public_ip_addresses.list_all()) + except Exception as exc: + logger.error("Failed to list public IP addresses: %s", exc) + return findings + + for pip in public_ips: + ip_config = getattr(pip, "ip_configuration", None) + nat_gateway = getattr(pip, "nat_gateway", None) + if not ip_config and not nat_gateway: + findings.append({ + "rule_id": RULE_ID, + "rule_name": RULE_NAME, + "severity": SEVERITY, + "category": CATEGORY, + "resource_id": getattr(pip, "id", ""), + "resource_name": getattr(pip, "name", ""), + "resource_type": "Microsoft.Network/publicIPAddresses", + "description": DESCRIPTION, + "remediation": REMEDIATION, + "playbook": PLAYBOOK, + "frameworks": FRAMEWORKS, + "metadata": { + "ip_address": getattr(pip, "ip_address", ""), + "location": getattr(pip, "location", ""), + "sku": getattr(getattr(pip, "sku", None), "name", ""), + }, + }) + + return findings diff --git a/scanner/rules/az_net_007.py b/scanner/rules/az_net_007.py new file mode 100644 index 00000000..08a3571c --- /dev/null +++ b/scanner/rules/az_net_007.py @@ -0,0 +1,68 @@ +"""AZ-NET-007: Application Gateway without WAF enabled.""" + +import logging +from typing import Any, Dict, List + +RULE_ID = "AZ-NET-007" +RULE_NAME = "Application Gateway without WAF enabled" +SEVERITY = "HIGH" +CATEGORY = "Network" +FRAMEWORKS = {"CIS": "9.6", "NIST": "SI-3", "ISO27001": "A.13.1.1"} +DESCRIPTION = ( + "An Application Gateway exists without Web Application Firewall enabled. " + "Without WAF, the application is unprotected against common web exploits " + "such as SQL injection, cross-site scripting and OWASP Top 10 attacks. " + "Any public-facing application behind an Application Gateway should have " + "WAF enabled in Prevention mode." +) +REMEDIATION = ( + "Upgrade the Application Gateway SKU to WAF_v2 and enable WAF in " + "Prevention mode. Configure the OWASP core rule set and review any " + "false positives before enabling Prevention mode in production." +) +PLAYBOOK = "playbooks/cli/fix_az_net_007.sh" + +logger = logging.getLogger(__name__) + + +def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: + """Detect Application Gateways without WAF enabled.""" + findings: List[Dict[str, Any]] = [] + + try: + from azure.mgmt.network import NetworkManagementClient + client = NetworkManagementClient( + azure_client.credential, azure_client.subscription_id + ) + app_gateways = list(client.application_gateways.list_all()) + except Exception as exc: + logger.error("Failed to list application gateways: %s", exc) + return findings + + for agw in app_gateways: + sku = getattr(agw, "sku", None) + sku_name = getattr(sku, "name", "") if sku else "" + waf_config = getattr(agw, "web_application_firewall_configuration", None) + waf_enabled = getattr(waf_config, "enabled", False) if waf_config else False + + if "WAF" not in sku_name or not waf_enabled: + findings.append({ + "rule_id": RULE_ID, + "rule_name": RULE_NAME, + "severity": SEVERITY, + "category": CATEGORY, + "resource_id": getattr(agw, "id", ""), + "resource_name": getattr(agw, "name", ""), + "resource_type": "Microsoft.Network/applicationGateways", + "description": DESCRIPTION, + "remediation": REMEDIATION, + "playbook": PLAYBOOK, + "frameworks": FRAMEWORKS, + "metadata": { + "sku": sku_name, + "waf_enabled": waf_enabled, + "location": getattr(agw, "location", ""), + }, + }) + + return findings diff --git a/scanner/rules/az_net_008.py b/scanner/rules/az_net_008.py new file mode 100644 index 00000000..d1ebe4ca --- /dev/null +++ b/scanner/rules/az_net_008.py @@ -0,0 +1,62 @@ +"""AZ-NET-008: Load balancer with no backend pool configured.""" + +import logging +from typing import Any, Dict, List + +RULE_ID = "AZ-NET-008" +RULE_NAME = "Load balancer with no backend pool configured" +SEVERITY = "LOW" +CATEGORY = "Network" +FRAMEWORKS = {"CIS": "9.1", "NIST": "CM-7", "ISO27001": "A.13.1.1"} +DESCRIPTION = ( + "A load balancer exists in the subscription but has no backend pool " + "configured. A load balancer with no backend pool is either misconfigured " + "or is a leftover resource from a decommissioned workload. It represents " + "unnecessary cost and indicates poor resource hygiene." +) +REMEDIATION = ( + "If the load balancer is no longer needed, delete it to reduce cost and " + "attack surface. If it is still required, configure a backend pool with " + "the appropriate virtual machines or scale set instances." +) +PLAYBOOK = "playbooks/cli/fix_az_net_008.sh" + +logger = logging.getLogger(__name__) + + +def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: + """Detect load balancers with no backend pool configured.""" + findings: List[Dict[str, Any]] = [] + + try: + from azure.mgmt.network import NetworkManagementClient + client = NetworkManagementClient( + azure_client.credential, azure_client.subscription_id + ) + load_balancers = list(client.load_balancers.list_all()) + except Exception as exc: + logger.error("Failed to list load balancers: %s", exc) + return findings + + for lb in load_balancers: + backend_pools = getattr(lb, "backend_address_pools", []) or [] + if len(backend_pools) == 0: + findings.append({ + "rule_id": RULE_ID, + "rule_name": RULE_NAME, + "severity": SEVERITY, + "category": CATEGORY, + "resource_id": getattr(lb, "id", ""), + "resource_name": getattr(lb, "name", ""), + "resource_type": "Microsoft.Network/loadBalancers", + "description": DESCRIPTION, + "remediation": REMEDIATION, + "playbook": PLAYBOOK, + "frameworks": FRAMEWORKS, + "metadata": { + "location": getattr(lb, "location", ""), + "backend_pool_count": len(backend_pools), + }, + }) + + return findings diff --git a/scanner/rules/az_net_009.py b/scanner/rules/az_net_009.py new file mode 100644 index 00000000..676e15c9 --- /dev/null +++ b/scanner/rules/az_net_009.py @@ -0,0 +1,63 @@ +"""AZ-NET-009: VPN gateway using outdated IKE version.""" + +import logging +from typing import Any, Dict, List + +RULE_ID = "AZ-NET-009" +RULE_NAME = "VPN gateway using outdated IKE version" +SEVERITY = "HIGH" +CATEGORY = "Network" +FRAMEWORKS = {"CIS": "9.5", "NIST": "SC-8", "ISO27001": "A.13.2.1"} +DESCRIPTION = ( + "A VPN gateway is configured to use IKEv1 which is an outdated and less " + "secure version of the Internet Key Exchange protocol. IKEv1 is vulnerable " + "to several known attacks and lacks features present in IKEv2 such as " + "improved authentication and built-in NAT traversal support." +) +REMEDIATION = ( + "Migrate the VPN gateway connection to use IKEv2. Update the VPN gateway " + "SKU if required and reconfigure all VPN connections to use IKEv2 only. " + "Coordinate with the remote VPN peer to ensure IKEv2 is supported on both ends." +) +PLAYBOOK = "playbooks/cli/fix_az_net_009.sh" + +logger = logging.getLogger(__name__) + + +def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: + """Detect VPN gateways using outdated IKEv1.""" + findings: List[Dict[str, Any]] = [] + + try: + from azure.mgmt.network import NetworkManagementClient + client = NetworkManagementClient( + azure_client.credential, azure_client.subscription_id + ) + connections = list(client.virtual_network_gateway_connections.list_all()) + except Exception as exc: + logger.error("Failed to list VPN gateway connections: %s", exc) + return findings + + for conn in connections: + ike_version = getattr(conn, "connection_protocol", "") or "" + if ike_version.upper() == "IKEV1": + findings.append({ + "rule_id": RULE_ID, + "rule_name": RULE_NAME, + "severity": SEVERITY, + "category": CATEGORY, + "resource_id": getattr(conn, "id", ""), + "resource_name": getattr(conn, "name", ""), + "resource_type": "Microsoft.Network/connections", + "description": DESCRIPTION, + "remediation": REMEDIATION, + "playbook": PLAYBOOK, + "frameworks": FRAMEWORKS, + "metadata": { + "ike_version": ike_version, + "location": getattr(conn, "location", ""), + "connection_type": getattr(conn, "connection_type", ""), + }, + }) + + return findings diff --git a/scanner/rules/az_net_010.py b/scanner/rules/az_net_010.py new file mode 100644 index 00000000..135e6789 --- /dev/null +++ b/scanner/rules/az_net_010.py @@ -0,0 +1,68 @@ +"""AZ-NET-010: Subnet with no network security group attached.""" + +import logging +from typing import Any, Dict, List + +RULE_ID = "AZ-NET-010" +RULE_NAME = "Subnet with no network security group attached" +SEVERITY = "HIGH" +CATEGORY = "Network" +FRAMEWORKS = {"CIS": "9.2", "NIST": "SC-7", "ISO27001": "A.13.1.1"} +DESCRIPTION = ( + "A subnet exists without a Network Security Group attached. Without an NSG " + "at the subnet level, all resources deployed into that subnet have no network " + "layer access control. Any VM or service in the subnet is reachable from " + "other subnets and potentially the internet with no filtering in place." +) +REMEDIATION = ( + "Create and attach an NSG to the subnet with rules that follow the principle " + "of least privilege. Define explicit allow rules for required traffic and " + "deny everything else. Apply NSGs at both the subnet and NIC level for " + "defence in depth." +) +PLAYBOOK = "playbooks/cli/fix_az_net_010.sh" + +logger = logging.getLogger(__name__) + + +def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: + """Detect subnets with no NSG attached.""" + findings: List[Dict[str, Any]] = [] + + try: + from azure.mgmt.network import NetworkManagementClient + client = NetworkManagementClient( + azure_client.credential, azure_client.subscription_id + ) + vnets = list(client.virtual_networks.list_all()) + except Exception as exc: + logger.error("Failed to list virtual networks: %s", exc) + return findings + + for vnet in vnets: + for subnet in getattr(vnet, "subnets", []) or []: + name = getattr(subnet, "name", "") + if name in ("GatewaySubnet", "AzureFirewallSubnet", "AzureBastionSubnet"): + continue + nsg = getattr(subnet, "network_security_group", None) + if not nsg: + findings.append({ + "rule_id": RULE_ID, + "rule_name": RULE_NAME, + "severity": SEVERITY, + "category": CATEGORY, + "resource_id": getattr(subnet, "id", ""), + "resource_name": name, + "resource_type": "Microsoft.Network/virtualNetworks/subnets", + "description": DESCRIPTION, + "remediation": REMEDIATION, + "playbook": PLAYBOOK, + "frameworks": FRAMEWORKS, + "metadata": { + "vnet_name": getattr(vnet, "name", ""), + "vnet_id": getattr(vnet, "id", ""), + "address_prefix": getattr(subnet, "address_prefix", ""), + }, + }) + + return findings From 2badbce0c1122a3df2d6c4c1290450227af7c4e8 Mon Sep 17 00:00:00 2001 From: Ritik Sah Date: Tue, 5 May 2026 19:42:34 +0100 Subject: [PATCH 12/15] Feat/az stor 003 (#21) * feat: add rule AZ-STOR-003 storage lifecycle policy check * feat: add rule AZ-STOR-003 storage lifecycle policy check --- compliance/frameworks/iso27001.json | 15 +- compliance/frameworks/nist_csf.json | 7 +- docs/az-stor-003-test-plan.md | 392 ++++++++++++++++++++++++++++ playbooks/cli/fix_az_stor_003.sh | 195 ++++++++++++++ scanner/azure_client.py | 71 ++++- scanner/rules/az_stor_003.py | 117 +++++++++ 6 files changed, 784 insertions(+), 13 deletions(-) create mode 100644 docs/az-stor-003-test-plan.md create mode 100755 playbooks/cli/fix_az_stor_003.sh create mode 100644 scanner/rules/az_stor_003.py diff --git a/compliance/frameworks/iso27001.json b/compliance/frameworks/iso27001.json index a3792b1a..79313752 100644 --- a/compliance/frameworks/iso27001.json +++ b/compliance/frameworks/iso27001.json @@ -89,12 +89,6 @@ "description": "Virtual machines with public IPs and no NSG have unrestricted network access. Network controls should be applied to all compute resources accessible from the internet." }, "AZ-KV-001": { -<<<<<<< feat/network-rules-expansion - "control_id": "A.12.3.1", - "control_name": "Information backup", - "description": "Key Vault soft delete protects against loss of secrets, keys and certificates. Backup copies of information should be taken and tested regularly in accordance with an agreed backup policy." - } -======= "control_id": "A.17.2.1", "control_name": "Availability of information processing facilities", "description": "Information processing facilities shall be implemented with sufficient redundancy to meet availability requirements. Disabling soft delete on Key Vault removes the ability to recover deleted secrets, keys, and certificates, creating a single point of failure for critical cryptographic material and violating availability and recovery requirements." @@ -105,10 +99,9 @@ "description": "Information stored on Azure storage accounts should be subject to formal lifecycle management controls governing retention and disposal. Storage accounts without lifecycle policies retain data indefinitely with no automated disposal mechanism, violating information handling and disposal requirements under this control." }, "AZ-KV-002": { - "control_id": "A.13.1.1", - "control_name": "Network controls", - "description": "Networks should be managed and controlled to protect information systems and applications. Allowing public network access to Azure Key Vault increases exposure of sensitive secrets, keys, and certificates to external networks. Access should be restricted to trusted networks using private endpoints or network controls." - } ->>>>>>> dev + "control_id": "A.13.1.1", + "control_name": "Network controls", + "description": "Networks should be managed and controlled to protect information systems and applications. Allowing public network access to Azure Key Vault increases exposure of sensitive secrets, keys, and certificates to external networks. Access should be restricted to trusted networks using private endpoints or network controls." + } } } diff --git a/compliance/frameworks/nist_csf.json b/compliance/frameworks/nist_csf.json index 2c4ddf88..7a9ebbaf 100644 --- a/compliance/frameworks/nist_csf.json +++ b/compliance/frameworks/nist_csf.json @@ -91,7 +91,12 @@ "AZ-KV-001": { "control_id": "PR.IP-4", "control_name": "Backups of information are conducted, maintained, and tested", - "description": "Key Vault soft delete protects against accidental or malicious deletion of secrets, keys and certificates. Without soft delete, deleted vault objects cannot be recovered, causing potential data loss." + "description": "Key material in Azure Key Vault must be recoverable after accidental or malicious deletion. Soft delete provides a recoverable state for secrets, keys, and certificates, supporting backup and recovery requirements for critical cryptographic material." + }, + "AZ-STOR-003": { + "control_id": "PR.DS-3", + "control_name": "Assets are formally managed throughout removal, transfers, and disposition", + "description": "NIST CSF PR.DS-3 requires that data assets are managed through their full lifecycle including secure disposal. Storage accounts without a lifecycle management policy have no automated mechanism for expiring or deleting aged data, meaning data subject to disposal requirements persists indefinitely and is never formally retired from the asset inventory." } } } diff --git a/docs/az-stor-003-test-plan.md b/docs/az-stor-003-test-plan.md new file mode 100644 index 00000000..65fb2729 --- /dev/null +++ b/docs/az-stor-003-test-plan.md @@ -0,0 +1,392 @@ +# Test Plan — AZ-STOR-003 +# Storage Account Has No Lifecycle Management Policy +# ============================================================ + +## 1. Overview + +This test plan covers verification of the AZ-STOR-003 scanner rule +and its remediation playbook. The goal is to confirm: + +- The rule correctly identifies non-compliant storage accounts +- The rule correctly ignores compliant storage accounts +- The playbook successfully creates a lifecycle policy +- The rule finds zero issues after the playbook runs + +--- + +## 2. Files Under Test + +| File | Purpose | +|---|---| +| scanner/rules/az_stor_003.py | Scanner rule | +| playbooks/cli/fix_az_stor_003.sh | Remediation script | +| scanner/azure_client.py | New method: get_storage_lifecycle_policy() | +| compliance/frameworks/cis_azure_benchmark.json | CIS mapping | +| compliance/frameworks/nist_csf.json | NIST mapping | +| compliance/frameworks/iso27001.json | ISO 27001 mapping | + +--- + +## 3. Test Environment Setup + +### 3.1 Prerequisites + +- Python 3.10+ +- Azure free trial account (portal.azure.com) +- Azure CLI installed and logged in (az login) +- OpenShield repo cloned and dependencies installed (pip install -r requirements.txt) +- .env file populated with AZURE_SUBSCRIPTION_ID, AZURE_CLIENT_ID, + AZURE_CLIENT_SECRET, AZURE_TENANT_ID +- StorageV2 accounts used for all tests + +### 3.2 Create Test Resources in Azure + +Run these commands once before testing. They create two storage accounts: +one without a lifecycle policy (should be flagged) and one with a policy +(should NOT be flagged). + + # Create resource group + az group create --name openshield-test-rg --location eastus + + # Storage account WITHOUT lifecycle policy (will be flagged) + az storage account create \ + --name oshieldtestnopolicy \ + --resource-group openshield-test-rg \ + --sku Standard_LRS \ + --location eastus + + # Storage account WITH lifecycle policy (will NOT be flagged) + az storage account create \ + --name oshieldtestpolicyon \ + --resource-group openshield-test-rg \ + --sku Standard_LRS \ + --location eastus + + # Manually apply a policy to the second account + az storage account management-policy create \ + --account-name oshieldtestpolicyon \ + --resource-group openshield-test-rg \ + --policy '{ + "rules": [{ + "name": "test-policy", + "enabled": true, + "type": "Lifecycle", + "definition": { + "filters": {"blobTypes": ["blockBlob"]}, + "actions": { + "baseBlob": { + "delete": {"daysAfterLastAccessTimeGreaterThan": 365} + } + } + } + }] + }' + +--- + +## 4. Test Cases + +--- + +### TC-001 — Rule detects non-compliant account (POSITIVE TEST) + +**Purpose:** Confirm the rule flags a storage account with no lifecycle policy. + +**Pre-condition:** oshieldtestnopolicy exists with no lifecycle policy. + +**Steps:** + + python -c " + from dotenv import load_dotenv; load_dotenv() + import os + from scanner.azure_client import AzureClient + from scanner.rules import az_stor_003 as rule + + client = AzureClient(os.environ['AZURE_SUBSCRIPTION_ID']) + findings = rule.scan(client, os.environ['AZURE_SUBSCRIPTION_ID']) + print(f'Total findings: {len(findings)}') + for f in findings: + print(f' [{f[\"severity\"]}] {f[\"resource_name\"]}') + " + +**Expected result:** +- At minimum one finding returned +- oshieldtestnopolicy appears in the findings list +- Finding has severity = MEDIUM +- Finding has rule_id = AZ-STOR-003 +- Finding dict contains all required keys: + rule_id, rule_name, severity, category, resource_id, resource_name, + resource_type, description, remediation, playbook, frameworks + +**Pass criteria:** oshieldtestnopolicy is in findings list. + +--- + +### TC-002 — Rule ignores compliant account (NEGATIVE TEST) + +**Purpose:** Confirm the rule does NOT flag accounts that already have a policy. + +**Pre-condition:** oshieldtestpolicyon exists WITH a lifecycle policy applied. + +**Steps:** Same script as TC-001. Inspect the findings list. + +**Expected result:** +- oshieldtestpolicyon does NOT appear in the findings list. + +**Pass criteria:** oshieldtestpolicyon absent from findings. + +--- + +### TC-003 — Finding dict has correct structure + +**Purpose:** Confirm every required field is present and correctly typed. + +**Steps:** + + python -c " + from dotenv import load_dotenv; load_dotenv() + import os, json + from scanner.azure_client import AzureClient + from scanner.rules import az_stor_003 as rule + + REQUIRED_KEYS = [ + 'rule_id', 'rule_name', 'severity', 'category', + 'resource_id', 'resource_name', 'resource_type', + 'description', 'remediation', 'playbook', 'frameworks' + ] + + client = AzureClient(os.environ['AZURE_SUBSCRIPTION_ID']) + findings = rule.scan(client, os.environ['AZURE_SUBSCRIPTION_ID']) + + for f in findings: + missing = [k for k in REQUIRED_KEYS if k not in f] + if missing: + print(f'FAIL — missing keys: {missing}') + else: + print(f'PASS — {f[\"resource_name\"]} has all required keys') + print(f' frameworks: {f[\"frameworks\"]}') + print(f' severity: {f[\"severity\"]}') + " + +**Expected result:** +- No missing keys reported +- severity = MEDIUM +- frameworks dict contains CIS, NIST, ISO27001 keys + +**Pass criteria:** All required keys present in every finding. + +--- + +### TC-004 — Full scan engine picks up the rule + +**Purpose:** Confirm the rule loads automatically when the engine runs — +no manual registration needed. + +**Steps:** + + python -c " + from dotenv import load_dotenv; load_dotenv() + import json, os + from scanner.engine import ScanEngine + + engine = ScanEngine(os.environ['AZURE_SUBSCRIPTION_ID']) + rule_ids = [getattr(r, 'RULE_ID', 'UNKNOWN') for r in engine.rules] + print('Loaded rules:', rule_ids) + print('AZ-STOR-003 loaded:', 'AZ-STOR-003' in rule_ids) + " + +**Expected result:** +- AZ-STOR-003 appears in the loaded rules list. + +**Pass criteria:** 'AZ-STOR-003 loaded: True' in output. + +--- + +### TC-005 — Playbook prints usage when called with no arguments + +**Purpose:** Confirm the script does not crash silently and has clear usage. + +**Steps:** + + bash playbooks/cli/fix_az_stor_003.sh + +**Expected result:** +- Prints usage instructions +- Exits with a non-zero exit code (1) +- Does NOT make any changes to Azure + +**Pass criteria:** Usage text displayed, script exits cleanly. + +--- + +### TC-006 — Playbook remediates the non-compliant account + +**Purpose:** Confirm the playbook successfully creates a lifecycle policy. + +**Pre-condition:** oshieldtestnopolicy has no lifecycle policy. + +**Steps:** + + bash playbooks/cli/fix_az_stor_003.sh \ + openshield-test-rg \ + oshieldtestnopolicy \ + 365 + + # Verify the policy was created + az storage account management-policy show \ + --account-name oshieldtestnopolicy \ + --resource-group openshield-test-rg + +**Expected result:** +- Script prints confirmation message +- az management-policy show returns a JSON policy object +- Policy contains a rule named openshield-lifecycle-rule +- Policy shows tierToCool at 30 days, tierToArchive at 90 days, + delete at 365 days + +**Pass criteria:** Policy visible in Azure portal and via CLI show command. + +--- + +### TC-007 — Rule returns zero findings after remediation + +**Purpose:** Full end-to-end — confirm the rule clears after the fix is applied. + +**Pre-condition:** TC-006 has run successfully (oshieldtestnopolicy now has a policy). + +**Steps:** Re-run TC-001 script. + +**Expected result:** +- oshieldtestnopolicy no longer appears in findings. + +**Pass criteria:** Previously flagged account no longer in findings list. + +--- + +### TC-008 — Script handles non-existent account gracefully + +**Purpose:** Confirm the script fails cleanly when given a valid-format name +that does not exist in Azure — the failure comes from the Azure CLI, not +from our validation. + +**Steps:** + + bash playbooks/cli/fix_az_stor_003.sh \ + openshield-test-rg \ + oshieldaccountxyz999 \ + 365 + + # When prompted, enter "y" to proceed past the confirmation. + +**Expected result:** +- Passes all input validation (name format is valid) +- Azure CLI returns a ResourceNotFound error +- Script exits with a non-zero exit code from set -euo pipefail +- Error from Azure CLI is visible in output + +**Pass criteria:** Script exits with Azure error, does not silently continue. + +--- + +### TC-009 — Playbook rejects invalid days-to-delete value + +**Purpose:** Confirm integer validation works — prevents broken JSON policy. + +**Steps:** + + bash playbooks/cli/fix_az_stor_003.sh \ + openshield-test-rg \ + oshieldtestnopolicy \ + "not-a-number" + +**Expected result:** +- Prints: `ERROR: days-to-delete must be a positive integer` +- Exits with code 1 +- Makes no changes to Azure + +**Pass criteria:** Error message displayed, exit code 1. + +--- + +### TC-010 — Playbook rejects shell-unsafe characters in arguments + +**Purpose:** Confirm input sanitisation prevents shell injection. + +**Steps:** + + bash playbooks/cli/fix_az_stor_003.sh \ + "my-rg; echo INJECTED" \ + oshieldtestnopolicy + +**Expected result:** +- Prints: `ERROR: resource-group contains invalid characters` +- Exits with code 1 +- The string "INJECTED" does NOT appear in output + +**Pass criteria:** Error shown, no command injection executed. + +--- + +### TC-011 — Playbook enables last access tracking before applying policy + +**Purpose:** Confirm the prerequisite step runs before the policy is created. +Without last access tracking enabled, `daysAfterLastAccessTimeGreaterThan` +policies are accepted by Azure but never fire — a silent failure. + +**Steps:** + + # Confirm tracking is OFF before the test + az storage account blob-service-properties show \ + --account-name oshieldtestnopolicy \ + --resource-group openshield-test-rg \ + --query "lastAccessTimeTrackingPolicy.enable" + # Should return: null or false + + # Run the playbook (enter "y" when prompted) + bash playbooks/cli/fix_az_stor_003.sh \ + openshield-test-rg \ + oshieldtestnopolicy \ + 365 + + # Confirm tracking is now ON + az storage account blob-service-properties show \ + --account-name oshieldtestnopolicy \ + --resource-group openshield-test-rg \ + --query "lastAccessTimeTrackingPolicy.enable" + # Must return: true + +**Expected result:** +- Before playbook: tracking disabled or null +- After playbook: tracking enabled = true +- Policy also present (verify with management-policy show) + +**Pass criteria:** `lastAccessTimeTrackingPolicy.enable` is `true` after the +playbook runs. + +--- + +## 5. Cleanup + +After all tests pass, delete the test resources to avoid charges: + + az group delete --name openshield-test-rg --yes --no-wait + +--- + +## 6. Pass / Fail Summary Table + +| Test Case | Description | Expected | Status | +|---|---|---|---| +| TC-001 | Rule detects non-compliant account | Finding returned | [ ] | +| TC-002 | Rule ignores compliant account | No finding | [ ] | +| TC-003 | Finding dict structure | All required keys present | [ ] | +| TC-004 | Engine loads rule automatically | AZ-STOR-003 in loaded list | [ ] | +| TC-005 | Playbook prints usage on no args | Usage text + exit 1 | [ ] | +| TC-006 | Playbook creates lifecycle policy | Policy visible in Azure | [ ] | +| TC-007 | Rule clears after remediation | Zero findings post-fix | [ ] | +| TC-008 | Script handles non-existent account | Exits with Azure error | [ ] | +| TC-009 | Playbook rejects non-integer days | Error + exit 1 | [ ] | +| TC-010 | Playbook rejects unsafe characters | Error, no injection | [ ] | +| TC-011 | Playbook enables last access tracking | Tracking = true after run | [ ] | + +All 11 test cases must pass before opening the PR. diff --git a/playbooks/cli/fix_az_stor_003.sh b/playbooks/cli/fix_az_stor_003.sh new file mode 100755 index 00000000..7231d7b8 --- /dev/null +++ b/playbooks/cli/fix_az_stor_003.sh @@ -0,0 +1,195 @@ +#!/bin/bash +# OpenShield Remediation Playbook +# Rule: AZ-STOR-003 — Storage Account Has No Lifecycle Management Policy +# Usage: ./fix_az_stor_003.sh [days-to-delete] +# Severity: MEDIUM +# +# What this script does: +# 1. Enables last access time tracking on the storage account (required +# prerequisite for daysAfterLastAccessTimeGreaterThan policies). +# 2. Creates a lifecycle management policy with three tiers: +# - Move to Cool tier after 30 days of no access +# - Move to Archive tier after 90 days of no access +# - Delete blobs after days (default 365) +# 3. The same delete rule applies to blob snapshots. +# +# Prerequisites: +# - Azure CLI installed and logged in (az login) +# - Contributor or Storage Account Contributor role on the target account +# - The storage account must use StorageV2 or BlobStorage kind for lifecycle +# management. Classic and premium accounts are not supported. +# +# Example: +# ./fix_az_stor_003.sh my-resource-group my-storage-account 365 + +set -euo pipefail + +RESOURCE_GROUP="${1:-}" +STORAGE_ACCOUNT="${2:-}" +DAYS_TO_DELETE="${3:-365}" + +# ── Argument validation ────────────────────────────────────────────────────── + +if [ -z "$RESOURCE_GROUP" ] || [ -z "$STORAGE_ACCOUNT" ]; then + echo "Usage: $0 [days-to-delete]" + echo "" + echo "Arguments:" + echo " resource-group Name of the Azure resource group" + echo " storage-account-name Name of the storage account to remediate" + echo " days-to-delete Days before blobs are permanently deleted (default: 365)" + echo "" + echo "Example:" + echo " $0 my-resource-group my-storage-account 365" + exit 1 +fi + +# ── Validate days-to-delete is a positive integer ──────────────────────────── + +if ! [[ "$DAYS_TO_DELETE" =~ ^[1-9][0-9]*$ ]]; then + echo "ERROR: days-to-delete must be a positive integer (got: '$DAYS_TO_DELETE')" + exit 1 +fi + +# ── Validate names contain only Azure-safe characters ─────────────────────── +# Resource group: letters, numbers, hyphens, underscores, dots, parentheses +# Storage account: lowercase letters and numbers only (Azure naming constraint) + +if ! [[ "$RESOURCE_GROUP" =~ ^[a-zA-Z0-9._()-]+$ ]]; then + echo "ERROR: resource-group contains invalid characters: '$RESOURCE_GROUP'" + exit 1 +fi + +if ! [[ "$STORAGE_ACCOUNT" =~ ^[a-z0-9]{3,24}$ ]]; then + echo "ERROR: storage-account-name must be 3-24 lowercase letters and numbers only." + exit 1 +fi + +# ── Validate DAYS_TO_DELETE range is sane ──────────────────────────────────── +# Azure requires tier transition <= delete threshold. Archive at 90 < delete. + +if [ "$DAYS_TO_DELETE" -lt 91 ]; then + echo "ERROR: days-to-delete must be at least 91 (must exceed the Archive tier at 90 days)" + exit 1 +fi + +# ── Secure temp file with guaranteed cleanup on exit or error ───────────────── + +POLICY_FILE=$(mktemp) +chmod 600 "$POLICY_FILE" + +cleanup() { + rm -f "$POLICY_FILE" +} +trap cleanup EXIT + +# ── Confirm before making changes ──────────────────────────────────────────── + +echo "============================================================" +echo " OpenShield Remediation — AZ-STOR-003" +echo "============================================================" +echo "" +echo " Storage account : $STORAGE_ACCOUNT" +echo " Resource group : $RESOURCE_GROUP" +echo " Delete after : $DAYS_TO_DELETE days" +echo "" +echo " Steps:" +echo " 1. Enable last access time tracking (required prerequisite)" +echo " 2. Create lifecycle policy with three tiers:" +echo " - Move to Cool tier after 30 days of no access" +echo " - Move to Archive after 90 days of no access" +echo " - Delete permanently after $DAYS_TO_DELETE days of no access" +echo "" +echo " NOTE: This requires StorageV2 or BlobStorage account kind." +echo " Premium and Classic accounts do not support lifecycle management." +echo "" +read -r -p "Proceed? [y/N] " CONFIRM +if [[ "$CONFIRM" != "y" && "$CONFIRM" != "Y" ]]; then + echo "Aborted. No changes were made." + exit 0 +fi + +# ── Step 1: Enable last access time tracking ───────────────────────────────── +# REQUIRED before daysAfterLastAccessTimeGreaterThan can be used in a policy. +# Without this, the Azure API accepts the policy JSON but the tier transitions +# never fire — the account stays non-compliant silently. + +echo "" +echo "[1/2] Enabling last access time tracking on: $STORAGE_ACCOUNT ..." + +az storage account blob-service-properties update \ + --account-name "$STORAGE_ACCOUNT" \ + --resource-group "$RESOURCE_GROUP" \ + --enable-last-access-tracking true + +echo " Last access tracking enabled." + +# ── Step 2: Write and apply the lifecycle policy ────────────────────────────── +# DAYS_TO_DELETE is validated as a positive integer >= 91 above. +# All variable expansions inside the heredoc are safe. + +echo "" +echo "[2/2] Applying lifecycle management policy to: $STORAGE_ACCOUNT ..." + +cat > "$POLICY_FILE" << EOF +{ + "rules": [ + { + "name": "openshield-lifecycle-rule", + "enabled": true, + "type": "Lifecycle", + "definition": { + "filters": { + "blobTypes": ["blockBlob"] + }, + "actions": { + "baseBlob": { + "tierToCool": { + "daysAfterLastAccessTimeGreaterThan": 30 + }, + "tierToArchive": { + "daysAfterLastAccessTimeGreaterThan": 90 + }, + "delete": { + "daysAfterLastAccessTimeGreaterThan": ${DAYS_TO_DELETE} + } + }, + "snapshot": { + "delete": { + "daysAfterCreationGreaterThan": ${DAYS_TO_DELETE} + } + } + } + } + } + ] +} +EOF + +az storage account management-policy create \ + --account-name "$STORAGE_ACCOUNT" \ + --resource-group "$RESOURCE_GROUP" \ + --policy "@${POLICY_FILE}" + +# Temp file removed automatically by trap on EXIT. + +# ── Confirmation ───────────────────────────────────────────────────────────── + +echo "" +echo "============================================================" +echo " Remediation complete for: $STORAGE_ACCOUNT" +echo "============================================================" +echo "" +echo " Applied:" +echo " Last access time tracking : enabled" +echo " Move to Cool after 30 days of no access" +echo " Move to Archive after 90 days of no access" +echo " Delete after $DAYS_TO_DELETE days of no access" +echo "" +echo " To verify the policy was applied:" +echo " az storage account management-policy show \\" +echo " --account-name $STORAGE_ACCOUNT \\" +echo " --resource-group $RESOURCE_GROUP" +echo "" +echo " NOTE: Adjust tier thresholds and delete day to match your" +echo " organisation's data retention and compliance policy." +echo "============================================================" diff --git a/scanner/azure_client.py b/scanner/azure_client.py index e7381b5d..bf3e3352 100644 --- a/scanner/azure_client.py +++ b/scanner/azure_client.py @@ -3,6 +3,7 @@ import logging from typing import Any, Dict, List, Optional +from azure.core.exceptions import HttpResponseError, ResourceNotFoundError from azure.identity import DefaultAzureCredential from azure.mgmt.authorization import AuthorizationManagementClient from azure.mgmt.compute import ComputeManagementClient @@ -59,6 +60,74 @@ def get_storage_accounts(self) -> List[Any]: logger.error("get_storage_accounts failed: %s", exc) return [] + def get_storage_lifecycle_policy( + self, resource_group: str, account_name: str + ) -> Optional[bool]: + """Check whether a storage account has a lifecycle management policy. + + Three-state return — the calling rule uses strict identity checks + (is False / is None) to distinguish these states: + + True — policy exists and contains at least one enabled rule. + False — ResourceNotFoundError: no policy configured (non-compliant). + None — any other error (permissions, network, SDK bug). + Caller must NOT create a finding — skip with a warning + to avoid false positives. + + The StorageManagementClient is created fresh here following the same + pattern as every other method in AzureClient (one client per call). + The credential is reused from self.credential so no new auth round- + trip occurs. + + Args: + resource_group: Resource group containing the storage account. + account_name: Name of the storage account. + + Returns: + Optional[bool] — True, False, or None as described above. + """ + try: + client = StorageManagementClient(self.credential, self.subscription_id) + policy = client.management_policies.get( + resource_group, account_name, "default" + ) + # A policy shell can exist with an empty rules list — + # treat that the same as no policy (non-compliant). + rules = getattr(getattr(policy, "policy", None), "rules", None) + return bool(rules) + + except ResourceNotFoundError: + # Expected path: the account genuinely has no lifecycle policy. + # This is the non-compliant condition — return False to flag it. + logger.debug( + "get_storage_lifecycle_policy(%s): ResourceNotFound — no policy", + account_name, + ) + return False + + except HttpResponseError as exc: + # 403 = service principal lacks + # Microsoft.Storage/storageAccounts/managementPolicies/read. + # Return None — cannot determine compliance, do not flag. + logger.error( + "get_storage_lifecycle_policy(%s) HTTP %s — " + "check service principal permissions: %s", + account_name, + exc.status_code, + exc, + ) + return None + + except Exception as exc: + # Unexpected failure (network, SDK bug, etc.). + # Return None — skip rather than create a false positive. + logger.error( + "get_storage_lifecycle_policy(%s) unexpected error: %s", + account_name, + exc, + ) + return None + # ------------------------------------------------------------------ # # Network # # ------------------------------------------------------------------ # @@ -185,4 +254,4 @@ def get_conditional_access_policies(self) -> List[Any]: return response.json().get("value", []) except Exception as exc: logger.error("get_conditional_access_policies failed: %s", exc) - return [] + return [] \ No newline at end of file diff --git a/scanner/rules/az_stor_003.py b/scanner/rules/az_stor_003.py new file mode 100644 index 00000000..7758b45a --- /dev/null +++ b/scanner/rules/az_stor_003.py @@ -0,0 +1,117 @@ +"""AZ-STOR-003: Storage account has no lifecycle management policy configured.""" + +import logging +from typing import Any, Dict, List, Optional + +logger = logging.getLogger(__name__) + +# subscription_id is received by scan() and passed to AzureClient methods +# that need explicit scope. It is not read from the environment here — +# the engine always passes it as a parameter. Never read os.environ directly. + +# ── Required module-level constants ───────────────────────────────────────── + +RULE_ID = "AZ-STOR-003" +RULE_NAME = "Storage Account Has No Lifecycle Management Policy" +SEVERITY = "MEDIUM" +CATEGORY = "Storage" +FRAMEWORKS = { + "CIS": "3.7", + "NIST": "PR.DS-3", + "ISO27001": "A.8.3.1", +} +DESCRIPTION = ( + "The storage account has no lifecycle management policy configured. " + "Without a lifecycle policy, blobs accumulate indefinitely — old data " + "that is no longer needed remains accessible, increasing storage costs " + "and the attack surface. A compromised account exposes all historical " + "data with no automatic expiry or tiering in place." +) +REMEDIATION = ( + "Create a lifecycle management policy on the storage account that " + "transitions blobs to cooler tiers (Cool, Archive) after a defined " + "number of days, and deletes blobs that exceed the organisation's " + "maximum retention period. Navigate to: Storage Account > " + "Data management > Lifecycle management > Add a rule." +) +PLAYBOOK = "playbooks/cli/fix_az_stor_003.sh" + + +# ── Required scan function ─────────────────────────────────────────────────── + +def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: + """Detect storage accounts with no lifecycle management policy. + + The Azure Storage Management SDK exposes lifecycle policies via + ``management_policies.get(resource_group, account_name)``. + A ResourceNotFound (404) response means no policy exists — this is + the condition we flag as MEDIUM severity. + + Three-state return from get_storage_lifecycle_policy(): + True — policy exists and has rules → skip (compliant) + False — no policy exists → create finding + None — permissions error or unexpected failure → skip with warning + to avoid false positives + + Args: + azure_client: An AzureClient instance with all SDK clients + pre-configured. + subscription_id: The Azure subscription ID being scanned. + + Returns: + A list of finding dicts — one per storage account that has no + lifecycle policy. Accounts that could not be checked are skipped + and logged as warnings. + """ + findings: List[Dict[str, Any]] = [] + + for account in azure_client.get_storage_accounts(): + resource_id = getattr(account, "id", "") + account_name = getattr(account, "name", "") + location = getattr(account, "location", "") + + if not resource_id or not account_name: + continue + + parsed = azure_client.parse_resource_id(resource_id) + resource_group = parsed.get("resource_group", "") + if not resource_group: + continue + + # True = compliant, False = no policy, None = could not determine + policy_status: Optional[bool] = azure_client.get_storage_lifecycle_policy( + resource_group, account_name + ) + + if policy_status is None: + # Permissions error or unexpected SDK failure. + # Skip rather than flag — never create false positives. + logger.warning( + "AZ-STOR-003: Could not determine lifecycle policy for %s " + "— skipping. Ensure the service principal has " + "Microsoft.Storage/storageAccounts/managementPolicies/read " + "permission.", + account_name, + ) + continue + + if policy_status is False: + findings.append({ + "rule_id": RULE_ID, + "rule_name": RULE_NAME, + "severity": SEVERITY, + "category": CATEGORY, + "resource_id": resource_id, + "resource_name": account_name, + "resource_type": "Microsoft.Storage/storageAccounts", + "description": DESCRIPTION, + "remediation": REMEDIATION, + "playbook": PLAYBOOK, + "frameworks": FRAMEWORKS, + "metadata": { + "resource_group": resource_group, + "location": location, + }, + }) + + return findings \ No newline at end of file From 1e7a81fffb89a928c1c6ce476794ed53be772730 Mon Sep 17 00:00:00 2001 From: Tanvir Farhad Date: Fri, 8 May 2026 11:37:29 +0100 Subject: [PATCH 13/15] docs: add SOC 2 Type II compliance framework mapping (#33) * docs: add SOC 2 Type II compliance framework mapping for all 20 rules Added SOC 2 Type II framework with detailed controls for security measures and compliance requirements. * feat: add soc2 to FRAMEWORK_FILE_MAP in finding.py add soc2.json to FRAMEWORK_FILE_MAP in finding.py * feat: add soc2 to SUPPORTED_FRAMEWORKS in compliance.py Added 'soc2' to the list of supported compliance frameworks. * Add SOC 2 controls for data protection and management --- api/models/finding.py | 1 + api/routes/compliance.py | 4 +- compliance/frameworks/soc2.json | 107 ++++++++++++++++++++++++++++++++ 3 files changed, 110 insertions(+), 2 deletions(-) create mode 100644 compliance/frameworks/soc2.json diff --git a/api/models/finding.py b/api/models/finding.py index 90b86623..8cdab3f6 100644 --- a/api/models/finding.py +++ b/api/models/finding.py @@ -20,6 +20,7 @@ "cis": "cis_azure_benchmark.json", "nist": "nist_csf.json", "iso27001": "iso27001.json", + "soc2": "soc2.json", } diff --git a/api/routes/compliance.py b/api/routes/compliance.py index e3b68a23..6a3b104f 100644 --- a/api/routes/compliance.py +++ b/api/routes/compliance.py @@ -7,7 +7,7 @@ compliance_bp = Blueprint("compliance", __name__) -SUPPORTED_FRAMEWORKS = ("cis", "nist", "iso27001") +SUPPORTED_FRAMEWORKS = ("cis", "nist", "iso27001", "soc2") def _get_db() -> DatabaseManager: @@ -20,7 +20,7 @@ def _get_db() -> DatabaseManager: def get_compliance(framework: str): """Return pass/fail compliance breakdown for a framework. - Supported frameworks: cis, nist, iso27001 + Supported frameworks: cis, nist, iso27001, soc2 Returns control-level pass/fail status mapped to current open findings. """ diff --git a/compliance/frameworks/soc2.json b/compliance/frameworks/soc2.json new file mode 100644 index 00000000..7de22578 --- /dev/null +++ b/compliance/frameworks/soc2.json @@ -0,0 +1,107 @@ +{ + "framework": "SOC 2 Type II", + "version": "2017", + "published": "2017-04", + "controls": { + "AZ-STOR-001": { + "control_id": "CC6.6", + "control_name": "Restricts Access to Information Assets", + "description": "Public blob access allows unauthenticated users from outside the network boundary to read storage data without credentials. CC6.6 requires that access from outside the network perimeter is restricted and controlled. Disabling public access enforces this boundary by requiring authentication for all storage operations." + }, + "AZ-STOR-002": { + "control_id": "CC6.7", + "control_name": "Protects Data in Transit", + "description": "Allowing unencrypted HTTP traffic to a storage account exposes data in transit to interception and tampering. CC6.7 requires that data transmitted over networks is protected using encryption. Enforcing HTTPS-only ensures all storage traffic is encrypted in transit." + }, + "AZ-STOR-003": { + "control_id": "CC8.1", + "control_name": "Change Management", + "description": "A storage account with no lifecycle management policy allows data to accumulate indefinitely with no automatic expiry or tiering. CC8.1 requires that infrastructure and data are managed through formal processes. Implementing a lifecycle policy ensures data retention is controlled and old data is automatically moved or deleted according to organisational policy." + }, + "AZ-NET-001": { + "control_id": "CC6.6", + "control_name": "Restricts Access from Outside the Network Boundary", + "description": "An NSG allowing unrestricted RDP access from the internet permits any external party to attempt remote access to virtual machines. CC6.6 requires that logical access from outside the network boundary is restricted. Limiting RDP to known IP ranges enforces this boundary and eliminates unauthorised remote access attempts." + }, + "AZ-NET-002": { + "control_id": "CC6.6", + "control_name": "Restricts Access from Outside the Network Boundary", + "description": "An NSG allowing unrestricted SSH access from the internet exposes virtual machines to brute force and credential attacks from any external party. CC6.6 requires that access from outside the network perimeter is restricted and controlled. Restricting SSH to known IP ranges or removing it in favour of Azure Bastion enforces this boundary." + }, + "AZ-NET-003": { + "control_id": "CC6.6", + "control_name": "Restricts Access from Outside the Network Boundary", + "description": "An NSG permitting unrestricted inbound access on port 443 from the internet exposes web services to automated scanning and exploitation attempts from any external source. CC6.6 requires that access from outside the network boundary is restricted to authorised sources. Public-facing services should be fronted by a WAF-enabled Application Gateway rather than exposed directly." + }, + "AZ-NET-004": { + "control_id": "CC6.6", + "control_name": "Restricts Access from Outside the Network Boundary", + "description": "A Network Security Group with no custom rules provides no meaningful boundary control and relies entirely on Azure defaults. CC6.6 requires that logical access from outside the network perimeter is explicitly restricted. Explicit least-privilege rules must be defined to enforce the network boundary." + }, + "AZ-NET-005": { + "control_id": "A1.1", + "control_name": "Capacity and Performance Monitoring", + "description": "Virtual networks without DDoS Protection Standard are vulnerable to volumetric attacks that can exhaust capacity and cause service outages. A1.1 requires that current processing capacity is monitored and resources are available to meet objectives. DDoS Protection Standard ensures network availability is maintained under attack conditions." + }, + "AZ-NET-006": { + "control_id": "CC6.6", + "control_name": "Restricts Access from Outside the Network Boundary", + "description": "Unassociated public IP addresses represent unnecessary exposure on the internet and may indicate leftover resources from decommissioned workloads. CC6.6 requires that the network boundary is tightly controlled with only necessary resources exposed. Removing unassociated public IPs reduces the external attack surface." + }, + "AZ-NET-007": { + "control_id": "CC6.6", + "control_name": "Restricts Access from Outside the Network Boundary", + "description": "An Application Gateway without WAF enabled provides no protection against web application attacks from external sources including OWASP Top 10 vulnerabilities. CC6.6 requires that access from outside the network boundary is controlled and filtered. WAF in Prevention mode enforces application-layer boundary protection for public-facing services." + }, + "AZ-NET-008": { + "control_id": "CC8.1", + "control_name": "Change Management", + "description": "A load balancer with no backend pool configured is either misconfigured or a leftover resource from a decommissioned workload that was not properly cleaned up. CC8.1 requires that infrastructure changes are managed, tracked and that unused resources are removed through a formal process. Removing empty load balancers maintains an accurate and controlled infrastructure state." + }, + "AZ-NET-009": { + "control_id": "CC6.7", + "control_name": "Protects Data in Transit", + "description": "VPN gateway connections using IKEv1 use an outdated protocol with known vulnerabilities that weaken the confidentiality and integrity of data transmitted between networks. CC6.7 requires that data transmitted over networks is protected using current secure protocols. Migrating to IKEv2 ensures VPN traffic is protected with a modern and secure key exchange mechanism." + }, + "AZ-NET-010": { + "control_id": "CC6.6", + "control_name": "Restricts Access from Outside the Network Boundary", + "description": "A subnet without an NSG attached has no network layer access controls leaving all resources in that subnet reachable from other subnets or the internet with no filtering. CC6.6 requires that logical access from outside the network boundary is restricted. Attaching an NSG with explicit rules enforces boundary protection at the subnet level." + }, + "AZ-IDN-001": { + "control_id": "CC6.1", + "control_name": "Logical Access Security Measures", + "description": "A service principal with Contributor role at subscription scope has unrestricted ability to create, modify and delete any resource in the environment. CC6.1 requires that logical access to information assets is restricted to authorised users and service accounts with least-privilege permissions. Scoping role assignments to the minimum required resource enforces this control." + }, + "AZ-IDN-002": { + "control_id": "CC6.1", + "control_name": "Logical Access Security Measures", + "description": "Without MFA enforced on privileged accounts, a single compromised password grants full administrative access to the Azure environment. CC6.1 requires that logical access controls include strong authentication mechanisms. Enforcing MFA via Conditional Access policies ensures privileged access requires multiple factors of authentication." + }, + "AZ-DB-001": { + "control_id": "CC6.7", + "control_name": "Protects Data in Transit", + "description": "SQL Server without Transparent Data Encryption stores database files in plain text on disk. CC6.7 requires that data is protected using encryption both in transit and at rest. Enabling TDE ensures database files, backups and transaction logs are encrypted and unreadable without the encryption key." + }, + "AZ-DB-002": { + "control_id": "CC6.6", + "control_name": "Restricts Access from Outside the Network Boundary", + "description": "A SQL Server firewall rule allowing all IP addresses makes the database reachable from anywhere on the internet. CC6.6 requires that access from outside the network boundary is restricted to authorised sources. Locking the firewall to specific application IP ranges ensures only authorised systems can connect to the database." + }, + "AZ-CMP-001": { + "control_id": "CC6.6", + "control_name": "Restricts Access from Outside the Network Boundary", + "description": "A virtual machine with a public IP and no NSG has unrestricted inbound network access from the internet with no filtering in place. CC6.6 requires that logical access from outside the network perimeter is restricted and controlled. Attaching an NSG with explicit rules enforces the network boundary and controls what traffic can reach the VM." + }, + "AZ-KV-001": { + "control_id": "A1.2", + "control_name": "Environmental Threats and Recovery", + "description": "Key Vault without soft delete enabled allows permanent deletion of secrets, keys and certificates with no recovery possible. A1.2 requires that environmental threats to availability are identified and mitigated including protection against accidental or malicious data loss. Enabling soft delete ensures deleted vault objects can be recovered within the retention period." + }, + "AZ-KV-002": { + "control_id": "CC6.6", + "control_name": "Restricts Access from Outside the Network Boundary", + "description": "A Key Vault accessible from the public internet allows any external party to attempt access to secrets, keys and certificates. CC6.6 requires that access from outside the network boundary is restricted and controlled. Locking Key Vault access to private endpoints or specific VNet service endpoints enforces this boundary and protects sensitive credentials from external exposure." + } + } +} From f409b67d4e12566683ea62d67ec9c00254d2e481 Mon Sep 17 00:00:00 2001 From: Tanvir Farhad Date: Sat, 9 May 2026 14:53:04 +0100 Subject: [PATCH 14/15] Refactor/azure client network methods (#22) * refactor: add get_virtual_networks() and get_public_ip_addresses() to AzureClient * Refactor DDoS protection check to use azure_client * refactor: AZ-NET-006 now uses azure_client.get_public_ip_addresses() --- scanner/azure_client.py | 18 ++++++++++++++++++ scanner/rules/az_net_005.py | 15 +-------------- scanner/rules/az_net_006.py | 15 +-------------- 3 files changed, 20 insertions(+), 28 deletions(-) diff --git a/scanner/azure_client.py b/scanner/azure_client.py index bf3e3352..e68c06c0 100644 --- a/scanner/azure_client.py +++ b/scanner/azure_client.py @@ -152,6 +152,24 @@ def get_network_interface( logger.error("get_network_interface(%s) failed: %s", nic_name, exc) return None + def get_virtual_networks(self) -> List[Any]: + """List all virtual networks in the subscription.""" + try: + client = NetworkManagementClient(self.credential, self.subscription_id) + return list(client.virtual_networks.list_all()) + except Exception as exc: + logger.error("get_virtual_networks failed: %s", exc) + return [] + + def get_public_ip_addresses(self) -> List[Any]: + """List all public IP addresses in the subscription.""" + try: + client = NetworkManagementClient(self.credential, self.subscription_id) + return list(client.public_ip_addresses.list_all()) + except Exception as exc: + logger.error("get_public_ip_addresses failed: %s", exc) + return [] + # ------------------------------------------------------------------ # # Compute # # ------------------------------------------------------------------ # diff --git a/scanner/rules/az_net_005.py b/scanner/rules/az_net_005.py index 9f6702c8..48f90bee 100644 --- a/scanner/rules/az_net_005.py +++ b/scanner/rules/az_net_005.py @@ -27,20 +27,7 @@ def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: """Detect virtual networks without DDoS protection enabled.""" findings: List[Dict[str, Any]] = [] - try: - # NOTE: This rule creates a NetworkManagementClient directly rather than - # going through azure_client. A get_virtual_networks() method should be - # added to AzureClient in a follow-up PR for consistency. - from azure.mgmt.network import NetworkManagementClient - client = NetworkManagementClient( - azure_client.credential, azure_client.subscription_id - ) - vnets = list(client.virtual_networks.list_all()) - except Exception as exc: - logger.error("Failed to list virtual networks: %s", exc) - return findings - - for vnet in vnets: + for vnet in azure_client.get_virtual_networks(): ddos = getattr(vnet, "ddos_protection_plan", None) enable_ddos = getattr(vnet, "enable_ddos_protection", False) if not ddos and not enable_ddos: diff --git a/scanner/rules/az_net_006.py b/scanner/rules/az_net_006.py index 180f49b9..26923d79 100644 --- a/scanner/rules/az_net_006.py +++ b/scanner/rules/az_net_006.py @@ -28,20 +28,7 @@ def scan(azure_client: Any, subscription_id: str) -> List[Dict[str, Any]]: """Detect public IP addresses not associated with any resource.""" findings: List[Dict[str, Any]] = [] - try: - # NOTE: This rule creates a NetworkManagementClient directly rather than - # going through azure_client. A get_public_ip_addresses() method should be - # added to AzureClient in a follow-up PR for consistency. - from azure.mgmt.network import NetworkManagementClient - client = NetworkManagementClient( - azure_client.credential, azure_client.subscription_id - ) - public_ips = list(client.public_ip_addresses.list_all()) - except Exception as exc: - logger.error("Failed to list public IP addresses: %s", exc) - return findings - - for pip in public_ips: + for pip in azure_client.get_public_ip_addresses(): ip_config = getattr(pip, "ip_configuration", None) nat_gateway = getattr(pip, "nat_gateway", None) if not ip_config and not nat_gateway: From bb477796ab3340a3a0d8bae3a008a7106dfecc57 Mon Sep 17 00:00:00 2001 From: Ritik Sah Date: Sat, 9 May 2026 15:15:14 +0100 Subject: [PATCH 15/15] feat: add CI pipeline with 6 automated checks (#34) - Python syntax check on all rule files - Rule structure validation (RULE_ID, SEVERITY, FRAMEWORKS) + RULE_ID uniqueness - Hardcoded credential scan - Playbook existence + bash syntax check for every rule - Compliance JSON validation for all four framework files (inc. soc2.json) - API syntax check - Compliance vs rule cross-reference check - CI summary step with per-check pass/fail table (if: always) - Fix duplicate DESCRIPTION assignment in az_net_003.py - Add pyyaml to requirements.txt for local YAML validation - Add docs/ci-pipeline.md with local run commands and design rationale - Update CI_PIPELINE_GUIDE.md with final PR description Closes #30 --- .github/workflows/ci.yml | 370 ++++++++++++++++++++++++++++++++++++ docs/ci-pipeline.md | 357 ++++++++++++++++++++++++++++++++++ requirements.txt | 1 + scanner/rules/az_net_003.py | 2 - 4 files changed, 728 insertions(+), 2 deletions(-) create mode 100644 .github/workflows/ci.yml create mode 100644 docs/ci-pipeline.md diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 00000000..95f55108 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,370 @@ +name: OpenShield CI + +on: + pull_request: + branches: + - dev + - main + +jobs: + ci-checks: + name: Run All CI Checks + runs-on: ubuntu-latest + + steps: + # ── 1. Checkout the code ───────────────────────────────────────── + - name: Checkout repository + uses: actions/checkout@v4 + + # ── 2. Set up Python ───────────────────────────────────────────── + - name: Set up Python 3.11 + uses: actions/setup-python@v5 + with: + python-version: "3.11" + + # ── 3. Install dependencies ─────────────────────────────────────── + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install -r requirements.txt + + # ── CHECK 1: Python syntax on all rule files ─────────────────────── + - name: Python syntax check (rule files) + id: syntax_check + run: | + echo "=== Checking Python syntax on scanner/rules/ ===" + FAIL=0 + for f in scanner/rules/az_*.py; do + if ! python -m py_compile "$f" 2>&1; then + echo "SYNTAX ERROR: $f" + FAIL=1 + else + echo "OK: $f" + fi + done + if [ "$FAIL" -eq 1 ]; then + echo "One or more rule files have syntax errors." + exit 1 + fi + + # ── CHECK 2: Rule structure validation + RULE_ID uniqueness ────── + - name: Rule structure validation + id: structure_check + run: | + echo "=== Validating rule file structure ===" + python - <<'PYEOF' + import os + import importlib.util + import sys + from collections import defaultdict + + rules_dir = "scanner/rules" + required_fields = ["RULE_ID", "SEVERITY", "FRAMEWORKS"] + valid_severities = {"CRITICAL", "HIGH", "MEDIUM", "LOW", "INFO"} + failures = [] + seen_ids = defaultdict(list) + + for filename in sorted(os.listdir(rules_dir)): + if not filename.startswith("az_") or not filename.endswith(".py"): + continue + + filepath = os.path.join(rules_dir, filename) + spec = importlib.util.spec_from_file_location("rule", filepath) + mod = importlib.util.module_from_spec(spec) + + try: + spec.loader.exec_module(mod) + except Exception as e: + failures.append(f"{filename}: import error — {e}") + continue + + for field in required_fields: + if not hasattr(mod, field): + failures.append(f"{filename}: missing field '{field}'") + + if hasattr(mod, "SEVERITY"): + if mod.SEVERITY not in valid_severities: + failures.append( + f"{filename}: SEVERITY '{mod.SEVERITY}' not in {valid_severities}" + ) + + if hasattr(mod, "FRAMEWORKS"): + if not isinstance(mod.FRAMEWORKS, dict) or len(mod.FRAMEWORKS) == 0: + failures.append(f"{filename}: FRAMEWORKS must be a non-empty dict") + + if hasattr(mod, "RULE_ID"): + seen_ids[mod.RULE_ID].append(filename) + + # Two files sharing a RULE_ID silently corrupt scan reports + for rule_id, files in seen_ids.items(): + if len(files) > 1: + failures.append( + f"DUPLICATE RULE_ID '{rule_id}' in: {', '.join(files)}" + ) + + if failures: + print("RULE STRUCTURE FAILURES:") + for f in failures: + print(f" - {f}") + sys.exit(1) + else: + print(f"All {len(seen_ids)} rule files passed structure validation.") + PYEOF + + # ── CHECK 3: Hardcoded credential scan ──────────────────────────── + - name: Hardcoded credential scan + id: cred_scan + run: | + echo "=== Scanning for hardcoded credentials ===" + PATTERNS=( + "password\s*=" + "secret\s*=" + "api_key\s*=" + "client_secret\s*=" + "AZURE_CLIENT_SECRET\s*=\s*['\"][^'\"]\+" + "-----BEGIN.*PRIVATE KEY-----" + "AccountKey=" + ) + + FAIL=0 + for pattern in "${PATTERNS[@]}"; do + matches=$(grep -rniE "$pattern" \ + --include="*.py" --include="*.sh" --include="*.json" --include="*.yml" \ + --exclude-dir=".git" \ + --exclude-dir="venv" \ + --exclude="ci.yml" \ + . 2>/dev/null | \ + grep -v "\.env" | \ + grep -v "os\.environ" | \ + grep -v "os\.getenv" | \ + grep -v "#" | \ + grep -v "example" | \ + grep -v "placeholder" || true) + + if [ -n "$matches" ]; then + echo "POTENTIAL CREDENTIAL LEAK — pattern '$pattern':" + echo "$matches" + FAIL=1 + fi + done + + if [ "$FAIL" -eq 1 ]; then + echo "Hardcoded credentials detected. Remove them and use environment variables." + exit 1 + else + echo "No hardcoded credentials found." + fi + + # ── CHECK 4: Playbook existence + bash syntax ───────────────────── + - name: Playbook existence and syntax check + id: playbook_check + run: | + echo "=== Checking playbooks exist and are valid bash ===" + FAIL=0 + for rule_file in scanner/rules/az_*.py; do + filename=$(basename "$rule_file" .py) + playbook="playbooks/cli/fix_${filename}.sh" + + if [ ! -f "$playbook" ]; then + echo "MISSING PLAYBOOK: $playbook (required for $rule_file)" + FAIL=1 + elif ! bash -n "$playbook" 2>&1; then + echo "BASH SYNTAX ERROR: $playbook" + FAIL=1 + else + echo "OK: $playbook" + fi + done + + if [ "$FAIL" -eq 1 ]; then + echo "Fix the missing or broken playbook(s) before this PR can merge." + exit 1 + fi + + # ── CHECK 5: Compliance JSON validation ─────────────────────────── + - name: Compliance JSON validation + id: json_check + run: | + echo "=== Validating compliance framework JSON files ===" + python - <<'PYEOF' + import json + import sys + import os + + framework_dir = "compliance/frameworks" + expected_files = [ + "cis_azure_benchmark.json", + "nist_csf.json", + "iso27001.json", + "soc2.json", + ] + failures = [] + + for fname in expected_files: + fpath = os.path.join(framework_dir, fname) + + if not os.path.exists(fpath): + failures.append(f"MISSING FILE: {fpath}") + continue + + try: + with open(fpath) as f: + data = json.load(f) + + if not isinstance(data, dict) or len(data) == 0: + failures.append(f"{fname}: must be a non-empty JSON object") + continue + + n_controls = len(data.get("controls", {})) + print(f"OK: {fname} ({n_controls} controls)") + + except json.JSONDecodeError as e: + failures.append(f"{fname}: invalid JSON — {e}") + + if failures: + print("COMPLIANCE JSON FAILURES:") + for f in failures: + print(f" - {f}") + sys.exit(1) + PYEOF + + # ── CHECK 6: API syntax check ────────────────────────────────────── + - name: API syntax check + id: api_check + run: | + echo "=== Checking Python syntax on API files ===" + FAIL=0 + if [ -d "api" ]; then + while IFS= read -r -d '' f; do + if ! python -m py_compile "$f" 2>&1; then + echo "SYNTAX ERROR: $f" + FAIL=1 + else + echo "OK: $f" + fi + done < <(find api/ -name "*.py" -print0) + else + echo "No api/ directory found — skipping" + fi + + if [ "$FAIL" -eq 1 ]; then + echo "One or more API files have syntax errors." + exit 1 + fi + + # ── CHECK 7: Compliance JSON ↔ rule file cross-reference ────────── + - name: Compliance rule cross-reference + id: xref_check + run: | + echo "=== Cross-referencing compliance controls against rule files ===" + python - <<'PYEOF' + import json + import os + import importlib.util + import sys + + rules_dir = "scanner/rules" + framework_dir = "compliance/frameworks" + + existing_ids = set() + for filename in os.listdir(rules_dir): + if not filename.startswith("az_") or not filename.endswith(".py"): + continue + filepath = os.path.join(rules_dir, filename) + spec = importlib.util.spec_from_file_location("rule", filepath) + mod = importlib.util.module_from_spec(spec) + try: + spec.loader.exec_module(mod) + if hasattr(mod, "RULE_ID"): + existing_ids.add(mod.RULE_ID) + except Exception: + pass + + failures = [] + + for fname in os.listdir(framework_dir): + if not fname.endswith(".json"): + continue + fpath = os.path.join(framework_dir, fname) + try: + data = json.load(open(fpath)) + except (json.JSONDecodeError, OSError): + continue + + for rule_id in data.get("controls", {}): + if rule_id not in existing_ids: + failures.append( + f"{fname}: references '{rule_id}' but no matching rule file found" + ) + + if failures: + print("COMPLIANCE CROSS-REFERENCE FAILURES:") + for f in failures: + print(f" - {f}") + print() + print("Either add the missing rule file or remove the stale control mapping.") + sys.exit(1) + else: + print(f"All compliance controls map to existing rule files. ({len(existing_ids)} rules checked)") + PYEOF + + # ── Final summary — always runs, shows per-check pass/fail ──────── + - name: CI Summary + if: always() + env: + SYNTAX: ${{ steps.syntax_check.outcome }} + STRUCTURE: ${{ steps.structure_check.outcome }} + CREDS: ${{ steps.cred_scan.outcome }} + PLAYBOOK: ${{ steps.playbook_check.outcome }} + JSON: ${{ steps.json_check.outcome }} + API: ${{ steps.api_check.outcome }} + XREF: ${{ steps.xref_check.outcome }} + run: | + python - <<'PYEOF' + import os + + checks = [ + ("Python syntax (rule files)", os.environ["SYNTAX"]), + ("Rule structure + RULE_ID uniqueness", os.environ["STRUCTURE"]), + ("Hardcoded credential scan", os.environ["CREDS"]), + ("Playbook existence + bash syntax", os.environ["PLAYBOOK"]), + ("Compliance JSON validation", os.environ["JSON"]), + ("API syntax check", os.environ["API"]), + ("Compliance vs rule cross-reference", os.environ["XREF"]), + ] + + labels = { + "success": "PASS", + "failure": "FAIL", + "skipped": "SKIP", + "cancelled": "CANCELLED", + } + + lines = [ + "## OpenShield CI Results", + "", + "| Check | Result |", + "|---|---|", + ] + + all_passed = True + for name, outcome in checks: + label = labels.get(outcome, outcome.upper()) + lines.append(f"| {name} | {label} |") + if outcome != "success": + all_passed = False + + lines.append("") + if all_passed: + lines.append("**Result: All checks passed.**") + else: + lines.append("**Result: One or more checks failed. See the step logs above for details.**") + + summary = "\n".join(lines) + print(summary) + + summary_path = os.environ.get("GITHUB_STEP_SUMMARY") + if summary_path: + with open(summary_path, "a") as f: + f.write(summary + "\n") + PYEOF diff --git a/docs/ci-pipeline.md b/docs/ci-pipeline.md new file mode 100644 index 00000000..e79edb59 --- /dev/null +++ b/docs/ci-pipeline.md @@ -0,0 +1,357 @@ +# CI Pipeline + +OpenShield runs a GitHub Actions workflow on every pull request to `dev` and `main`. The workflow contains seven checks. All seven must pass before a PR can merge. + +This document explains what each check does, how to run every check locally before opening a PR, and the reasoning behind the testing methods chosen. + +--- + +## Checks at a glance + +| # | Check | What fails | +|---|---|---| +| 1 | Python syntax (rule files) | Any `az_*.py` with a syntax error | +| 2 | Rule structure + RULE_ID uniqueness | Missing required fields, invalid SEVERITY, non-dict FRAMEWORKS, duplicate RULE_IDs | +| 3 | Hardcoded credential scan | Literal secrets, keys, or connection strings in source files | +| 4 | Playbook existence + bash syntax | Missing `.sh` for any rule file, or a `.sh` with a bash syntax error | +| 5 | Compliance JSON validation | Missing framework file, invalid JSON, empty object | +| 6 | API syntax check | Any `api/**/*.py` with a syntax error | +| 7 | Compliance rule cross-reference | A rule ID referenced in a framework JSON that has no matching rule file | + +The final step always runs and writes a per-check pass/fail table to the GitHub Actions summary panel so reviewers can see the result without reading through logs. + +--- + +## Setup for local runs + +Before running any checks locally, install the project dependencies including `pyyaml`, which is required to validate the workflow file as valid YAML. + +```bash +pip install -r requirements.txt +``` + +If you prefer to install only what the local checks need without the full Azure SDK stack: + +```bash +pip install pyyaml==6.0.1 +``` + +To verify the workflow file itself is valid YAML before pushing: + +```bash +python -c " +import yaml +with open('.github/workflows/ci.yml') as f: + yaml.safe_load(f) +print('YAML is valid') +" +``` + +This catches structural problems in the workflow file — misaligned indentation, duplicate keys, bad anchors — that GitHub Actions would reject silently or with a confusing error message. + +--- + +## Running checks locally + +Run these from the root of the repository. If any command exits non-zero, CI will also fail. + +### Check 1 — Python syntax (rule files) + +```bash +for f in scanner/rules/az_*.py; do + python -m py_compile "$f" && echo "OK: $f" || echo "FAIL: $f" +done +``` + +A clean run prints `OK:` for every file and exits 0. + +--- + +### Check 2 — Rule structure and RULE_ID uniqueness + +```python +python - <<'PYEOF' +import os, importlib.util, sys +from collections import defaultdict + +rules_dir = "scanner/rules" +required_fields = ["RULE_ID", "SEVERITY", "FRAMEWORKS"] +valid_severities = {"CRITICAL", "HIGH", "MEDIUM", "LOW", "INFO"} +failures = [] +seen_ids = defaultdict(list) + +for filename in sorted(os.listdir(rules_dir)): + if not filename.startswith("az_") or not filename.endswith(".py"): + continue + filepath = os.path.join(rules_dir, filename) + spec = importlib.util.spec_from_file_location("rule", filepath) + mod = importlib.util.module_from_spec(spec) + try: + spec.loader.exec_module(mod) + except Exception as e: + failures.append(f"{filename}: import error — {e}") + continue + for field in required_fields: + if not hasattr(mod, field): + failures.append(f"{filename}: missing field '{field}'") + if hasattr(mod, "SEVERITY") and mod.SEVERITY not in valid_severities: + failures.append(f"{filename}: SEVERITY '{mod.SEVERITY}' is not valid") + if hasattr(mod, "FRAMEWORKS") and (not isinstance(mod.FRAMEWORKS, dict) or len(mod.FRAMEWORKS) == 0): + failures.append(f"{filename}: FRAMEWORKS must be a non-empty dict") + if hasattr(mod, "RULE_ID"): + seen_ids[mod.RULE_ID].append(filename) + +for rule_id, files in seen_ids.items(): + if len(files) > 1: + failures.append(f"DUPLICATE RULE_ID '{rule_id}' in: {', '.join(files)}") + +if failures: + print("FAILURES:") + for f in failures: print(f" - {f}") + sys.exit(1) +else: + print(f"All {len(seen_ids)} rule files passed.") +PYEOF +``` + +--- + +### Check 3 — Hardcoded credential scan + +```bash +PATTERNS=( + "password\s*=" + "secret\s*=" + "api_key\s*=" + "client_secret\s*=" + "AZURE_CLIENT_SECRET\s*=\s*['\"][^'\"]\+" + "-----BEGIN.*PRIVATE KEY-----" + "AccountKey=" +) + +FAIL=0 +for pattern in "${PATTERNS[@]}"; do + matches=$(grep -rniE "$pattern" \ + --include="*.py" --include="*.sh" --include="*.json" --include="*.yml" \ + --exclude-dir=".git" --exclude-dir="venv" --exclude="ci.yml" \ + . 2>/dev/null | \ + grep -v "\.env" | grep -v "os\.environ" | grep -v "os\.getenv" | \ + grep -v "#" | grep -v "example" | grep -v "placeholder" || true) + if [ -n "$matches" ]; then + echo "POTENTIAL LEAK — pattern '$pattern':" + echo "$matches" + FAIL=1 + fi +done +[ "$FAIL" -eq 0 ] && echo "No hardcoded credentials found." || echo "FAIL" +``` + +If this flags a match in your code, replace the literal value with `os.environ["VAR_NAME"]` and store the real value in your `.env` file (which is gitignored). + +--- + +### Check 4 — Playbook existence and bash syntax + +```bash +FAIL=0 +for rule_file in scanner/rules/az_*.py; do + filename=$(basename "$rule_file" .py) + playbook="playbooks/cli/fix_${filename}.sh" + if [ ! -f "$playbook" ]; then + echo "MISSING: $playbook" + FAIL=1 + elif ! bash -n "$playbook" 2>&1; then + echo "BASH SYNTAX ERROR: $playbook" + FAIL=1 + else + echo "OK: $playbook" + fi +done +[ "$FAIL" -eq 0 ] && echo "All playbooks OK." +``` + +`bash -n` parses the script without executing it. It catches undefined syntax such as mismatched `if`/`fi`, unclosed quotes, and bad redirects. It does not execute any Azure CLI commands. + +--- + +### Check 5 — Compliance JSON validation + +```python +python - <<'PYEOF' +import json, os, sys + +framework_dir = "compliance/frameworks" +expected = ["cis_azure_benchmark.json", "nist_csf.json", "iso27001.json", "soc2.json"] +failures = [] + +for fname in expected: + fpath = os.path.join(framework_dir, fname) + if not os.path.exists(fpath): + failures.append(f"MISSING: {fpath}") + continue + try: + data = json.load(open(fpath)) + n = len(data.get("controls", {})) + print(f"OK: {fname} ({n} controls)") + except json.JSONDecodeError as e: + failures.append(f"{fname}: invalid JSON — {e}") + +if failures: + for f in failures: print(f" - {f}") + sys.exit(1) +PYEOF +``` + +--- + +### Check 6 — API syntax check + +```bash +FAIL=0 +if [ -d "api" ]; then + while IFS= read -r -d '' f; do + python -m py_compile "$f" && echo "OK: $f" || { echo "FAIL: $f"; FAIL=1; } + done < <(find api/ -name "*.py" -print0) +else + echo "No api/ directory — skipping" +fi +[ "$FAIL" -eq 0 ] && echo "API syntax OK." +``` + +--- + +### Check 7 — Compliance rule cross-reference + +```python +python - <<'PYEOF' +import json, os, importlib.util, sys + +rules_dir = "scanner/rules" +framework_dir = "compliance/frameworks" + +existing_ids = set() +for filename in os.listdir(rules_dir): + if not filename.startswith("az_") or not filename.endswith(".py"): + continue + spec = importlib.util.spec_from_file_location("rule", os.path.join(rules_dir, filename)) + mod = importlib.util.module_from_spec(spec) + try: + spec.loader.exec_module(mod) + if hasattr(mod, "RULE_ID"): + existing_ids.add(mod.RULE_ID) + except Exception: + pass + +failures = [] +for fname in os.listdir(framework_dir): + if not fname.endswith(".json"): + continue + try: + data = json.load(open(os.path.join(framework_dir, fname))) + except Exception: + continue + for rule_id in data.get("controls", {}): + if rule_id not in existing_ids: + failures.append(f"{fname}: references '{rule_id}' but no rule file found") + +if failures: + for f in failures: print(f" - {f}") + sys.exit(1) +else: + print(f"All compliance controls verified. ({len(existing_ids)} rules checked)") +PYEOF +``` + +--- + +## Testing method rationale + +### Why `py_compile` and not `flake8` or `pylint` + +`py_compile` checks only for syntax errors — the kind that prevent the file from loading at all. Linters add style and convention rules that differ across contributors and would generate noise on code written before the linter was introduced. A syntax check has a binary, objective outcome. That is the right scope for a CI gate on an open source project where contributors are writing their first rules. + +### Why `importlib` and not regex for structure validation + +Regex on Python source is fragile. A field could be assigned via a helper function, computed from a base class, or split across continuation lines. `importlib.util.spec_from_file_location` actually executes the module and then `hasattr()` checks the resulting object — the only way to be certain the attribute is present and accessible at runtime. This is the same mechanism the scanner engine uses when loading rules, so the CI check mirrors what production does. + +### Why `bash -n` and not just checking file existence + +An earlier version of this check only verified that a playbook file existed. A `.sh` file with a bash syntax error — an unclosed `if`, a bad heredoc, a missing `fi` — will crash immediately when an operator runs it in response to a real finding. `bash -n` parses without executing, so it catches structural errors at zero risk of touching any Azure resource. Existence alone is not sufficient. + +### Why the credential scan uses grep exclusions rather than an allowlist + +The patterns being scanned (`password=`, `secret=`, `api_key=`) appear legitimately in two contexts: environment variable lookups (`os.environ`, `os.getenv`) and inline comments. Both are explicitly excluded. The scan is scoped to literal assignment — the pattern that indicates a value is hardcoded in source. A grep-based approach is auditable: every exclusion is visible in one place and any contributor can read exactly what is and is not excluded. + +### Why the credential scan excludes `venv/` + +On GitHub Actions the checkout is clean with no `venv/`. Locally, `venv/` contains thousands of lines from third-party packages that match patterns like `password=None` as function arguments. Excluding `venv/` prevents false positives when contributors run the check locally without creating a confusing discrepancy between local and CI results. + +### Why the cross-reference check walks compliance JSONs rather than rule files + +The check is designed to catch a deletion scenario: a rule file is removed but its entry in one or more compliance JSONs is not. Walking the JSONs and looking up each referenced rule ID against the set of existing rule files catches stale references. The inverse check — verifying every rule file has a compliance entry — is not enforced because a rule may legitimately not map to every framework. + +--- + +## Edge cases handled + +**Rule file has syntax error but passes `py_compile`** +Not possible. `py_compile` detects all syntax errors that prevent the AST from parsing. If `py_compile` passes, the file can be imported. + +**Rule file imports a package not in `requirements.txt`** +Check 2 will fail with `import error` when `spec.loader.exec_module` raises `ModuleNotFoundError`. The error message names the missing package. Add it to `requirements.txt`. + +**Two rule files define the same `RULE_ID`** +Check 2 collects all IDs with `defaultdict(list)` before reporting, so it catches every duplicate in a single run rather than stopping at the first. The failure message names both files. + +**A playbook file exists but contains only a shebang and no logic** +`bash -n` passes — a script with only `#!/bin/bash` is syntactically valid. This is intentional: a stub playbook during development is acceptable; a broken playbook is not. + +**A compliance JSON has a `controls` key with no entries** +Check 5 reports the number of controls but does not fail on zero. An empty `controls` block is structurally valid JSON. Check 7 will simply find nothing to cross-reference. If you want to enforce minimum control counts, add a `len(controls) == 0` check to Check 5. + +**The `api/` directory does not exist** +Check 6 prints `No api/ directory found — skipping` and exits 0. The check is designed to be safe to include before the API module is added. + +**A framework JSON file references a rule ID that was renamed** +Check 7 catches this. The referenced ID will not be in `existing_ids` (which is built from the current `RULE_ID` attribute of each rule file) and CI fails with the exact JSON file and rule ID that is stale. + +**Trailing comma in a compliance JSON** +Check 5 catches this. Python's `json.load` raises `json.JSONDecodeError` on trailing commas, and the failure message includes the line number from the decoder. + +**Local `venv/` directory triggers credential scan false positives** +The scan excludes `--exclude-dir=venv`. On GitHub Actions there is no `venv/` to exclude, so the flag is harmless there. + +--- + +## How the CI summary works + +The final step uses `if: always()` so it runs regardless of whether earlier steps passed or failed. Each check step has a unique `id`. The summary step reads the outcome of every step via environment variables: + +```yaml +- name: CI Summary + if: always() + env: + SYNTAX: ${{ steps.syntax_check.outcome }} + STRUCTURE: ${{ steps.structure_check.outcome }} + ... +``` + +GitHub Actions sets `outcome` to `success`, `failure`, `skipped`, or `cancelled`. The summary step writes a markdown table to `$GITHUB_STEP_SUMMARY`, which GitHub renders as a panel on the Actions run page. This means a reviewer can see which check failed without opening any log. + +When running locally (no `$GITHUB_STEP_SUMMARY` environment variable), the summary is printed to stdout only. + +--- + +## Fixing common failures + +| Failure message | Cause | Fix | +|---|---|---| +| `SYNTAX ERROR: scanner/rules/az_xxx_000.py` | Invalid Python syntax | Open the file, find the syntax error, fix it | +| `missing field 'RULE_ID'` | Rule file does not define `RULE_ID` at module level | Add `RULE_ID = "AZ-XXX-000"` at the top of the file | +| `SEVERITY 'MEDIUM-HIGH' not in {...}` | SEVERITY value is not one of the five allowed strings | Change to `CRITICAL`, `HIGH`, `MEDIUM`, `LOW`, or `INFO` | +| `DUPLICATE RULE_ID 'AZ-NET-003'` | Two rule files declare the same ID | Assign a unique ID to the newer file | +| `POTENTIAL CREDENTIAL LEAK` | A literal secret is present in source | Replace with `os.environ["VAR_NAME"]` | +| `MISSING PLAYBOOK: playbooks/cli/fix_az_xxx_000.sh` | No playbook created for the new rule | Create `playbooks/cli/fix_az_xxx_000.sh` | +| `BASH SYNTAX ERROR: playbooks/cli/fix_az_xxx_000.sh` | Shell script has invalid syntax | Run `bash -n playbooks/cli/fix_az_xxx_000.sh` locally to see the error | +| `invalid JSON — ...` | Trailing comma or other JSON error in a framework file | Open the file, find the bad line (error message includes line number), fix it | +| `references 'AZ-XXX-000' but no matching rule file found` | A compliance JSON references a rule that does not exist | Either create the rule file or remove the entry from the compliance JSON | diff --git a/requirements.txt b/requirements.txt index ee813479..74c911fb 100644 --- a/requirements.txt +++ b/requirements.txt @@ -14,3 +14,4 @@ psycopg2-binary==2.9.9 python-dotenv==1.0.0 pyjwt==2.8.0 requests==2.31.0 +pyyaml==6.0.1 diff --git a/scanner/rules/az_net_003.py b/scanner/rules/az_net_003.py index 54d2ca14..a0a18e0f 100644 --- a/scanner/rules/az_net_003.py +++ b/scanner/rules/az_net_003.py @@ -9,7 +9,6 @@ CATEGORY = "Network" FRAMEWORKS = {"CIS": "9.3", "NIST": "SC-7", "ISO27001": "A.13.1.1"} DESCRIPTION = ( - DESCRIPTION = ( "A Network Security Group has an inbound rule allowing unrestricted access " "on port 443 from any source (0.0.0.0/0). While HTTPS traffic is encrypted, " "exposing port 443 to the entire internet unnecessarily increases the attack " @@ -18,7 +17,6 @@ "Review manually before remediating — do not auto-remediate without confirming " "the service is not meant to be publicly accessible." ) -) REMEDIATION = ( "Restrict the inbound rule on port 443 to known IP ranges or use an " "Application Gateway with WAF to front any public-facing HTTPS services. "