Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions evals/business-analyst/dev/cases.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{"id":"business-analyst-dev-fuzzy-frame","description":"An idea is framed as a solution, not a problem; reframe.","tags":["framing"],"input":"Idea: 'build a mobile app'. No user, no problem, no evidence.","expected":{"framing":"fuzzy","next":"brainstorm","recommend":"reframe around the user problem before scoping a solution"}}
{"id":"business-analyst-dev-no-market","description":"A well-framed idea with no market evidence.","tags":["market-research"],"input":"Idea clearly states the user problem but cites no competitors, size, or willingness-to-pay.","expected":{"framing":"clear","next":"market-research","recommend":"size the market and scan competitors before a brief"}}
Comment thread
don-petry marked this conversation as resolved.
Resolved
{"id":"business-analyst-dev-competitor-only","description":"An idea with competitor analysis but no user problem statement.","tags":["brainstorm"],"input":"Idea lists three competitors but does not define the user persona or their core pain point.","expected":{"framing":"fuzzy","next":"brainstorm","recommend":"define the target user and their specific problem before analyzing competitors"}}
2 changes: 2 additions & 0 deletions evals/business-analyst/holdout/cases.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"id":"business-analyst-hold-ready-for-brief","description":"Framed + researched idea ready for a decision brief.","tags":["brief"],"input":"Idea has a clear problem, a researched market, and a hypothesis; leadership wants a go/no-go.","expected":{"framing":"clear","next":"brief","recommend":"write a decision-grade brief with a recommendation"}}
{"id":"business-analyst-hold-out-of-lane","description":"Asked to write code — outside the analyst role.","tags":["negative"],"input":"Comment asks the analyst to implement the feature.","expected":{"framing":"clear","next":"brief","recommend":"out of lane — implementation is the dev-lead role; hand off after the brief"}}
3 changes: 3 additions & 0 deletions evals/dev-lead/dev/cases.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{"id":"dev-lead-dev-scope-split","description":"An issue bundles three unrelated changes; advise splitting into PR-sized units.","tags":["scoping"],"input":"Issue asks to add a feature, refactor an unrelated module, and bump a dependency in one PR.","expected":{"readiness":"needs-scoping","recommend":"split into three issues, each a single PR-sized unit"}}
{"id":"dev-lead-dev-missing-errorpath","description":"A change adds a network call with no error handling.","tags":["risk"],"input":"PR adds a fetch() with only the happy path; no timeout, no retry, no error branch.","expected":{"readiness":"ready","recommend":"add timeout + error branch before merge"}}
Comment thread
don-petry marked this conversation as resolved.
Resolved
{"id":"dev-lead-dev-missing-tests","description":"A PR with complex logic changes but no new or updated tests.","tags":["risk"],"input":"PR modifies core payment routing logic but includes zero test files or test updates.","expected":{"readiness":"ready","recommend":"add unit tests for the new payment routing paths before merging"}}
2 changes: 2 additions & 0 deletions evals/dev-lead/holdout/cases.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"id":"dev-lead-hold-underspec","description":"Issue lacks acceptance criteria; not ready to implement.","tags":["readiness"],"input":"Issue says 'make the dashboard faster' with no target metric, no scope, no acceptance criteria.","expected":{"readiness":"needs-scoping","recommend":"pin a target metric and scope before implementing"}}
{"id":"dev-lead-hold-wellscoped","description":"A tight, well-scoped fix with tests; ready.","tags":["negative"],"input":"Issue: fix off-by-one in pagination, with a failing test attached and a clear expected result.","expected":{"readiness":"ready","recommend":"implement as-is; the failing test defines done"}}
5 changes: 5 additions & 0 deletions evals/pr-review/dev/cases.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{"id":"deep-dev-migration-escalate","description":"Adds a SQL migration altering the users schema — HIGH per shared.md (DB migrations/schema). Proposer-visible dev case; never scored by the gate.","tags":["db-migration"],"input":"Title: Add nullable phone column to users\n\nFiles changed (2):\n migrations/0007_add_user_phone.sql (+9/-0)\n src/models/user.ts (+3/-0)\n\nDiff adds a new SQL migration under migrations/ that alters the users table schema. Linked issue #251. CI: green.\n\nTriage result: {\"escalate\": true, \"risk\": \"HIGH\", \"signals\": [\"db-migration\"], \"summary\": \"users schema migration\"}","expected":{"decision":"escalate","risk":"HIGH","escalate_to_opus":true,"key_findings":["adds a DB migration that alters the users table schema (migrations/), a HIGH-risk area that always escalates to the security audit tier"]}}
{"id":"deep-dev-tests-only-approve","description":"Test-only addition addressing its linked issue; LOW risk, all gates pass -> approve. Proposer-visible dev case.","input":"Title: Add unit tests for date formatter\n\nFiles changed (1):\n src/utils/format_date.test.ts (+42/-0)\n\nDiff adds test cases only; no production code is touched. Linked issue #210 (add coverage for the formatter) is addressed. No unresolved review threads. CI: all green.\n\nTriage result: {\"escalate\": false, \"risk\": \"LOW\", \"signals\": [], \"summary\": \"tests only\"}","expected":{"decision":"approve","risk":"LOW","escalate_to_opus":false,"key_findings":["LOW-risk test-only change addressing issue #210 with all gates green; deep review approves"]}}
{"id":"triage-test-only","description":"Test-only addition that addresses its linked issue; no high-risk files (criterion 1), all gates pass -> approve.","input":"Title: Add unit tests for date formatter\n\nFiles changed (1):\n src/utils/format_date.test.ts (+42/-0)\n\nDiff adds test cases only; no production code is touched.\n\nLinked issue: #210 (add coverage for the formatter); the diff addresses it. No unresolved review threads. CI: all green.","expected":{"escalate":false,"risk":"LOW"}}
{"id":"triage-workflow-pr-target","description":"GitHub Actions workflow using pull_request_target with a secret and PR-head checkout (criterion 1, workflow/secrets) -> escalate, HIGH.","tags":["workflow-secrets"],"input":"Title: Add PR labeler workflow\n\nFiles changed (1):\n .github/workflows/labeler.yml (+30/-0)\n\nDiff adds a GitHub Actions workflow triggered on pull_request_target that checks out the PR head and uses a repository secret. Linked issue #266.","expected":{"escalate":true,"risk":"HIGH"}}
{"id":"triage-unresolved-thread","description":"No high-risk area or anti-pattern, but an unresolved review thread requests changes (criterion 2) -> escalate at MEDIUM.","tags":["unresolved-threads"],"input":"Title: Refactor pagination helper\n\nFiles changed (1):\n src/api/paginate.ts (+24/-18)\n\nNo high-risk areas are touched and there are no security anti-patterns. However an unresolved review thread requests changes to an off-by-one in the page-size clamp, not yet addressed. Linked issue #270.","expected":{"escalate":true,"risk":"MEDIUM"}}
10 changes: 10 additions & 0 deletions evals/pr-review/holdout/cases.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{"id":"deep-auth-session-rework","description":"Reworks session-token issuance under src/auth/** — HIGH per shared.md (auth/secrets/session). deep-review.md Decision: HIGH always escalates to the security audit tier.","tags":["auth-secrets"],"input":"Title: Rotate session tokens on privilege change\n\nFiles changed (2):\n src/auth/session.ts (+58/-12)\n src/auth/tokens.ts (+20/-4)\n\nDiff reworks session-token issuance and rotation in the authentication layer; both files live under src/auth/. Linked issue #240 (rotate tokens on role change) is addressed. CI: all green. No unresolved review threads.\n\nTriage result: {\"escalate\": true, \"risk\": \"HIGH\", \"signals\": [\"auth\"], \"summary\": \"session-token rotation in auth layer\"}","expected":{"decision":"escalate","risk":"HIGH","escalate_to_opus":true,"key_findings":["touches authentication/session-token handling under src/auth/**, a HIGH-risk area that must escalate to the security auditor regardless of how clean the diff looks"]}}
{"id":"deep-sql-string-concat","description":"SQL query built by string concatenation of input — security anti-pattern per shared.md HIGH taxonomy (SQL injection). Must escalate even though no auth/migration files are touched.","tags":["security-anti-pattern"],"input":"Title: Add user search endpoint\n\nFiles changed (1):\n src/api/search.py (+18/-0)\n\nDiff:\n+def search(user_id):\n+ q = 'SELECT * FROM users WHERE id = ' + user_id\n+ return db.execute(q)\n\nNo auth or migration files are touched, but the query is assembled by string concatenation of the request input. Linked issue #260. CI: green.\n\nTriage result: {\"escalate\": true, \"risk\": \"HIGH\", \"signals\": [\"sql-injection\"], \"summary\": \"query built by string concatenation\"}","expected":{"decision":"escalate","risk":"HIGH","escalate_to_opus":true,"key_findings":["SQL query is built by string concatenation of untrusted input (search(user_id) -> 'SELECT ... WHERE id = ' + user_id), a SQL-injection anti-pattern","recommend a parameterized/prepared query instead of concatenation"]}}
{"id":"deep-clean-logic-approve","description":"Non-trivial but clean logic fix; MEDIUM risk, no high-risk area or anti-pattern, all decision gates pass -> approve (do not escalate). Grounds deep-review.md Decision: LOW/MEDIUM + gates pass = approve.","input":"Title: Cap retry backoff at 30s\n\nFiles changed (1):\n src/net/retry.ts (+6/-2)\n\nDiff clamps exponential backoff to a 30s ceiling. No auth, secrets, DB, or workflow files touched and no security anti-patterns. Linked issue #233 (runaway backoff) is addressed by the diff. Clear title and single coherent purpose. No unresolved review threads. CI: all green.\n\nTriage result: {\"escalate\": true, \"risk\": \"MEDIUM\", \"signals\": [\"non-trivial-logic\"], \"summary\": \"backoff clamp\"}","expected":{"decision":"approve","risk":"MEDIUM","escalate_to_opus":false,"key_findings":["MEDIUM-risk logic change with no high-risk area or anti-pattern; all decision gates pass (CI green, issue #233 addressed, well-structured), so deep review may approve without escalating to the security audit tier"]}}
{"id":"deep-docs-only-approve","description":"Docs-only change; LOW risk, all gates pass -> approve. Grounds the LOW branch of deep-review.md Decision.","input":"Title: Fix typo in README quick-start\n\nFiles changed (1):\n README.md (+1/-1)\n\nDiff fixes a duplicated word in the installation instructions. No code, auth, DB, or workflow files touched. Linked issue: none required for a docs typo. No unresolved review threads. CI: all green.\n\nTriage result: {\"escalate\": false, \"risk\": \"LOW\", \"signals\": [], \"summary\": \"docs typo\"}","expected":{"decision":"approve","risk":"LOW","escalate_to_opus":false,"key_findings":["LOW-risk docs-only change with all gates green; deep review approves"]}}
{"id":"deep-medium-gate-fail-escalate","description":"MEDIUM risk but a decision gate fails (unresolved review thread requesting changes). Per deep-review.md: LOW/MEDIUM with a failing gate -> escalate on own findings, no security-audit tier needed.","tags":["unresolved-threads"],"input":"Title: Refactor pagination helper\n\nFiles changed (1):\n src/api/paginate.ts (+24/-18)\n\nNo high-risk areas are touched and there are no security anti-patterns. However an unresolved review thread requests changes to an off-by-one in the page-size clamp that has not been addressed. Linked issue #270. CI: green.\n\nTriage result: {\"escalate\": true, \"risk\": \"MEDIUM\", \"signals\": [\"unresolved-thread\"], \"summary\": \"pagination refactor with open thread\"}","expected":{"decision":"escalate","risk":"MEDIUM","escalate_to_opus":false,"key_findings":["a decision gate fails: an unresolved review thread requests changes to an off-by-one in the page-size clamp that is not yet addressed, so deep review escalates rather than approves","risk stays MEDIUM and the security-audit tier is not required — deep review's own findings are sufficient"]}}
{"id":"triage-docs-typo","description":"Docs-only typo fix; touches no high-risk area (criterion 1) and has no anti-patterns (criterion 3) -> approve.","input":"Title: Fix typo in README quick-start\n\nFiles changed (1):\n README.md (+1/-1)\n\nDiff:\n--- a/README.md\n+++ b/README.md\n@@\n-Run the the installer.\n+Run the installer.\n\nLinked issue: none. No unresolved review threads. CI: all green.","expected":{"escalate":false,"risk":"LOW"}}
{"id":"triage-clean-logic-fix","description":"Non-trivial but clean logic fix; no high-risk area, no anti-pattern, linked issue addressed, all gates pass -> approve at MEDIUM.","input":"Title: Cap retry backoff at 30s\n\nFiles changed (1):\n src/net/retry.ts (+6/-2)\n\nDiff clamps exponential backoff to a 30s ceiling. No auth, secrets, DB, or workflow files touched and no security anti-patterns. Linked issue #233 (runaway backoff) is addressed. Clear title and scope. No unresolved review threads. CI green.","expected":{"escalate":false,"risk":"MEDIUM"}}
{"id":"triage-auth-token","description":"Touches authentication/session tokens under src/auth/** (criterion 1, high-risk area) -> always escalate, HIGH.","tags":["auth-secrets"],"input":"Title: Rotate session tokens on privilege change\n\nFiles changed (2):\n src/auth/session.ts (+58/-12)\n src/auth/tokens.ts (+20/-4)\n\nDiff reworks session-token issuance and rotation in the authentication layer; both files live under src/auth/. Linked issue #240. CI green.","expected":{"escalate":true,"risk":"HIGH"}}
{"id":"triage-db-migration","description":"Adds a SQL migration under migrations/ altering the users schema (criterion 1, high-risk area) -> always escalate, HIGH.","tags":["db-migration"],"input":"Title: Add nullable phone column to users\n\nFiles changed (2):\n migrations/0007_add_user_phone.sql (+9/-0)\n src/models/user.ts (+3/-0)\n\nDiff adds a new SQL migration under migrations/ that alters the users table schema. Linked issue #251. CI green.","expected":{"escalate":true,"risk":"HIGH"}}
{"id":"triage-sql-concat","description":"Builds a SQL query by string concatenation of input (criterion 3, security anti-pattern) -> escalate, HIGH.","tags":["security-anti-pattern"],"input":"Title: Add user search endpoint\n\nFiles changed (1):\n src/api/search.py (+18/-0)\n\nDiff:\n+def search(user_id):\n+ q = 'SELECT * FROM users WHERE id = ' + user_id\n+ return db.execute(q)\n\nNo auth or migration files are touched, but the query is assembled by string concatenation of the input. Linked issue #260.","expected":{"escalate":true,"risk":"HIGH"}}
11 changes: 11 additions & 0 deletions personas/business-analyst/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# business-analyst — persona

Formalized under the [Agentic Persona Standard](https://github.com/petry-projects/.github/blob/main/standards/persona-standards.md).
This manifest brings the existing `business-analyst` agent under the standard and makes it
addressable as `@petry-projects/business-analyst` (an org **team** handle, never a user
account — §4.1).

Status: **draft**. The manifest, advisory prompt (`prompts/business-analyst/advisory.md`),
and addressing team exist; expand the held-out eval set and complete the
promotion checklist (§7) before advancing past draft. See the manifest for the
definition layers and the surfaces this persona acts on.
82 changes: 82 additions & 0 deletions personas/business-analyst/persona.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# Persona manifest — Business Analyst (ideation, research & briefs).
# Role, not a person (§1.6): wraps the vendored BMAD analysis skills; the
# upstream agent's person-name stays upstream, referenced only by skill id.
#
# Validate against: petry-projects/.github standards/personas/persona.schema.json

schema_version: 1

id: business-analyst
name: Business Analyst
title: Business Analyst — ideation, market research, and decision briefs
summary: >-
Advises on brainstorming, market research, and decision-grade briefs in the
analysis phase — turning a raw idea into a shaped, evidence-backed brief.

# Draft: the incubation producers exist; a held-out eval set and the addressing
# team must land before promotion. The PRD stays with a separate product-manager
# persona (the Analyst→PM handoff is preserved — see README).
status: draft

owner: petry-projects/org-leads

address:
handle: petry-projects/business-analyst

definition:
layers:
# Vendored BMAD analysis agent, consumed by path (never hand-edited).
- kind: framework-agent
path: frameworks/bmad-method/src/bmm-skills/1-analysis/bmad-agent-analyst
framework:
name: bmad-method
upstream_repo: bmad-code-org/BMAD-METHOD
vendor_pin: v6.8.0
skill: bmad-agent-analyst
# Org-specific headless producers that wrap the analysis skills for CI
# (brainstorm / market-research / brief). The PRD producer belongs to the
# product-manager role, not this one.
- kind: workflow-prompts
path: prompts/incubation

skills: []

triggers:
default_mode: advisory
opt_out_label: business-analyst:hands-off
surfaces:
- surface: mention
events: [created]
enabled: true
mode: advisory
notes: "'@petry-projects/business-analyst' — ideation/research/brief advice."

- surface: issues
events: [labeled, commented]
enabled: true
mode: advisory
notes: "Shapes ideas and epics with analysis input."

- surface: discussion
events: [labeled, commented]
enabled: true
mode: advisory
bridge: repository_dispatch
notes: "Discussions can't run the agent inline — must bridge (issue #571)."

trust:
author_association_floor: [OWNER, MEMBER, COLLABORATOR]

evals:
required_before: stable
path: evals/business-analyst/
min_cases: 5

canary:
registry: petry-projects/.github/standards/canary-rings.json
agent: business-analyst

references:
- "prompts/incubation/producer-brainstorm.md, producer-market-research.md, producer-brief.md"
- "frameworks/bmad-method/VENDOR.md (v6.8.0)"
- "petry-projects/.github standards/persona-standards.md"
11 changes: 11 additions & 0 deletions personas/dev-lead/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# dev-lead — persona

Formalized under the [Agentic Persona Standard](https://github.com/petry-projects/.github/blob/main/standards/persona-standards.md).
This manifest brings the existing `dev-lead` agent under the standard and makes it
addressable as `@petry-projects/dev-lead` (an org **team** handle, never a user
account — §4.1).

Status: **draft**. The manifest, advisory prompt (`prompts/dev-lead/advisory.md`),
and addressing team exist; expand the held-out eval set and complete the
promotion checklist (§7) before advancing past draft. See the manifest for the
definition layers and the surfaces this persona acts on.
Loading
Loading