Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 102 additions & 0 deletions .github/workflows/system-chains.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
# System chain tests (trinity-enterprise#794) — the 1.0 system test's chains,
# run against a freshly booted stack of the chosen ref (a release candidate tag,
# or `dev`). Manual: the chains create agents, push to a test skills repo and
# wait minutes for staleness, so they gate a release cut, never a PR.
#
# Every run uploads `chain-report` (json + md): per chain, passed / failed at a
# named step / partial / not run. Attach the markdown to #783 as the go / no-go
# evidence. The job is red unless EVERY chain passed — a chain that could not
# run is not a pass.
#
# Optional configuration (without it, J18 reports NOT RUN):
# vars.CHAIN_SKILLS_REPO owner/name of a github.com test skills repo
# secrets.CHAIN_SKILLS_GITHUB_TOKEN token with push access to that repo
name: system-chains

on:
workflow_dispatch:
inputs:
ref:
description: "Ref to boot and test (a release candidate tag, or dev)"
required: true
default: dev

permissions:
contents: read

concurrency:
group: system-chains-${{ inputs.ref }}
cancel-in-progress: false

jobs:
system-chains:
name: system-chains
runs-on: ubuntu-latest
timeout-minutes: 60

steps:
- uses: actions/checkout@v7
with:
ref: ${{ inputs.ref }}
submodules: false

- uses: actions/setup-python@v7
with:
# Must equal the image pin (#1891, test_1891_python_version_parity.py).
python-version: '3.13'

- name: Install test dependencies
run: pip install -r tests/requirements-test.txt

- name: Generate a throwaway admin password
run: |
set -euo pipefail
PASS="$(openssl rand -base64 24 | tr -d '/+=\n')Aa1!"
echo "::add-mask::$PASS"
echo "ADMIN_PASSWORD=$PASS" >> "$GITHUB_ENV"

- name: Boot the stack
run: |
set -uo pipefail
cp .env.example .env
echo "ADMIN_PASSWORD=$ADMIN_PASSWORD" >> .env
echo "SECRET_KEY=$(openssl rand -hex 32)" >> .env
echo "TRINITY_DEFAULT_SYSTEM_MANIFEST=disabled" >> .env
./scripts/deploy/start.sh || { echo "::error title=Stack failed to start::the chains never ran"; exit 1; }
for i in $(seq 1 120); do
if curl -fsS http://localhost:8000/health >/dev/null 2>&1; then exit 0; fi
sleep 2
done
echo "::error title=Backend never became healthy::/health did not answer within 240s"
exit 1

- name: Run the chains
env:
TRINITY_API_URL: http://localhost:8000
TRINITY_TEST_PASSWORD: ${{ env.ADMIN_PASSWORD }}
CHAIN_SKILLS_REPO: ${{ vars.CHAIN_SKILLS_REPO }}
CHAIN_SKILLS_GITHUB_TOKEN: ${{ secrets.CHAIN_SKILLS_GITHUB_TOKEN }}
run: |
set -uo pipefail
rc=0
scripts/system/run_chains.sh --report-dir "$RUNNER_TEMP/chain-report" || rc=$?
if [ -f "$RUNNER_TEMP/chain-report/chain-report.md" ]; then
cat "$RUNNER_TEMP/chain-report/chain-report.md" >> "$GITHUB_STEP_SUMMARY"
fi
exit "$rc"

- name: Upload the report
if: always()
uses: actions/upload-artifact@v7
with:
name: chain-report
path: ${{ runner.temp }}/chain-report
if-no-files-found: warn

- name: Stack logs on failure
if: failure()
run: docker compose logs --tail 200 backend || true

- name: Tear down
if: always()
run: docker compose down -v --remove-orphans || true
7 changes: 6 additions & 1 deletion docs/testing/JOURNEYS.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,13 @@ coverage debt is queryable rather than rediscovered.
| **J12** | My agent's own declaration of what it measures is what Trinity actually knows about it. | user | local | live-stack | **no** | `tests/journeys/test_j12_declared_metrics_journey.py` | L-02, L-03 | #477, #478, #479, #666 |
| **J13** | My agent tells me what I can do with it and what is waiting on me — and only things that are true. | user | local | live-stack | **no** | `tests/journeys/test_j13_suggestions_journey.py` | — | #465 |
| **J14** | I can tag the agent in our group chat and it already knows what we've been talking about. | user | local | live-stack | **no** | — | — | abilityai/trinity-enterprise#600 |
| **J15** | A fleet I roll out from one definition comes up holding its skills and its canon, and the orchestrator sees what I deployed. | operator | local | live-stack | **no** | `tests/system_chains/test_j15_rollout_chain.py` | L-02, SK-02 | abilityai/trinity-enterprise#794 |
| **J16** | A fact I publish to the canon reaches the agents that use it, and they say which version they read. | user | local | live-stack | **no** | `tests/system_chains/test_j16_canon_fact_chain.py` | — | abilityai/trinity-enterprise#794 |
| **J17** | A fact that lands at the wrong agent reaches the agent that owns it, and I hold a receipt. | user | local | live-stack | **no** | `tests/system_chains/test_j17_fact_routing_chain.py` | — | abilityai/trinity-enterprise#794 |
| **J18** | A change I make to a library skill reaches every agent that holds it, and an agent without the permission cannot change skills. | operator | local | live-stack | **no** | `tests/system_chains/test_j18_library_change_chain.py` | SK-01, SK-02, SK-03 | abilityai/trinity-enterprise#794 |
| **J19** | My agent's objective reaches me: the number it records shows against the target, and reads stale when it stops. | user | local | live-stack | **no** | `tests/system_chains/test_j19_objective_to_person_chain.py` | L-02 | abilityai/trinity-enterprise#794 |

**0 of 14 journeys built.** Green/red is NOT recorded here — regenerate with `--junit <dir>` against CI artifacts for a coverage figure.
**0 of 19 journeys built.** Green/red is NOT recorded here — regenerate with `--junit <dir>` against CI artifacts for a coverage figure.

## Variants

Expand Down
1 change: 1 addition & 0 deletions docs/testing/STRATEGY.md
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,7 @@ The rule, and what enforces it:
| how to run the backend suites: env vars, tier wall times, friction recovery | [tests/README.md](../../tests/README.md) |
| how to run and tag the Playwright specs, and the fixture-agent contract | [src/frontend/e2e/README.md](../../src/frontend/e2e/README.md) |
| what each test file covers | [tests/registry.json](../../tests/registry.json) — it indexes tests; the journey catalog indexes promises, and the two are linked, never merged |
| the 1.0 system chains (J15–J19) — how to run them and read their go / no-go report | [tests/system_chains/README.md](../../tests/system_chains/README.md) |
| the click-through scenarios and their run reports | [phases/INDEX.md](phases/INDEX.md); `ui-sweep/` (created by the first sweep) |
| a feature's own test record | beside that feature's docs — e.g. [PULL_MIGRATION_TESTING.md](../planning/PULL_MIGRATION_TESTING.md) next to the pull-migration status |
| the manual runbook for one feature | that feature flow's `## Testing` section under `docs/memory/feature-flows/` |
Expand Down
57 changes: 57 additions & 0 deletions scripts/system/run_chains.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
#!/usr/bin/env bash
# Run the 1.0 system chain tests (trinity-enterprise#794) against a live stack —
# a release candidate, or any instance you can afford to mutate.
#
# TRINITY_API_URL=http://localhost:8000 \
# TRINITY_TEST_PASSWORD=<admin password> \
# [CHAIN_SKILLS_REPO=owner/name CHAIN_SKILLS_GITHUB_TOKEN=…] \
# scripts/system/run_chains.sh [--report-dir DIR] [pytest args…]
#
# Writes DIR/chain-report.json and DIR/chain-report.md (default ./chain-report):
# per chain, passed / failed at a named step / partial / not run. The markdown
# is what you attach to #783 as go / no-go evidence.
#
# Exit code: 0 only when EVERY chain passed; 1 when any chain FAILED (or no
# chain was reported at all); 3 when
# nothing failed but a chain was partial or not run. A chain that did not run,
# or ran only in part, is not a pass — read the report for why.
set -uo pipefail

REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
REPORT_DIR="${CHAIN_REPORT_DIR:-$PWD/chain-report}"
if [ "${1:-}" = "--report-dir" ]; then REPORT_DIR="$2"; shift 2; fi
mkdir -p "$REPORT_DIR"
REPORT_DIR="$(cd "$REPORT_DIR" && pwd)"

: "${TRINITY_API_URL:=http://localhost:8000}"
if [ -z "${TRINITY_TEST_PASSWORD:-}" ] && [ -z "${ADMIN_PASSWORD:-}" ]; then
echo "run_chains: set TRINITY_TEST_PASSWORD (or ADMIN_PASSWORD) to the target's admin password" >&2
exit 2
fi

export TRINITY_CHAIN_TESTS=1 TRINITY_API_URL CHAIN_REPORT_DIR="$REPORT_DIR"

cd "$REPO_ROOT/tests"
rc=0
python -m pytest system_chains -v -rs -p no:randomly --tb=short "$@" || rc=$?
if [ "$rc" = "5" ]; then
echo "run_chains: no chains collected — nothing ran, which is not a pass" >&2
exit 1
fi

if [ ! -f "$REPORT_DIR/chain-report.json" ]; then
echo "run_chains: no report was written — treat this run as failed" >&2
exit 1
fi
cat "$REPORT_DIR/chain-report.md"
python - "$REPORT_DIR/chain-report.json" <<'PY'
import json, sys
s = json.load(open(sys.argv[1]))["summary"]
# 0 every chain passed · 1 a chain FAILED (a defect, or the run itself broke)
# · 3 nothing failed, but a chain was partial or not run (not yet evidence)
if s["total"] == 0:
sys.exit(1) # an empty report is a broken run, never "not yet"
if s["all_passed"]:
sys.exit(0)
sys.exit(1 if s["counts"]["failed"] else 3)
PY
77 changes: 77 additions & 0 deletions tests/journeys/catalog.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -275,3 +275,80 @@ journeys:
# Mode, so the harness needs a bot with Privacy Mode off in a real group —
# nothing in the invariant catalog covers a channel adapter yet.
invariants: []

# --- trinity-enterprise#794: the 1.0 system chains -----------------------------
# A chain checks that a change in one part of a rolled-out system shows up in
# another part, on records the system wrote. Harnesses live in
# tests/system_chains/ (collected only under TRINITY_CHAIN_TESTS=1, run by
# scripts/system/run_chains.sh against a release candidate) and every run writes
# a per-chain report: passed / failed at a step / partial / not run. All
# `built: false` until a release-candidate run shows them green.

- id: J15
issue: "abilityai/trinity-enterprise#794"
promise: "A fleet I roll out from one definition comes up holding its skills and its canon, and the orchestrator sees what I deployed."
actor: operator
lanes: [local]
tier: live-stack
harness: tests/system_chains/test_j15_rollout_chain.py
built: false
owner: trinity
# Skeleton: reports NOT RUN until the reference company (ent#793) exists.
# L-02: create is atomic; SK-02: assigned ⇒ delivered, per agent of the fleet.
invariants: [L-02, SK-02]

- id: J16
issue: "abilityai/trinity-enterprise#794"
promise: "A fact I publish to the canon reaches the agents that use it, and they say which version they read."
actor: user
lanes: [local]
tier: live-stack
harness: tests/system_chains/test_j16_canon_fact_chain.py
built: false
owner: trinity
# Skeleton: NOT RUN until ent#793 and the change-propagation work (ent#804).
# No invariant covers canon-version provenance yet — an empty list, not an
# invented id.
invariants: []

- id: J17
issue: "abilityai/trinity-enterprise#794"
promise: "A fact that lands at the wrong agent reaches the agent that owns it, and I hold a receipt."
actor: user
lanes: [local]
tier: live-stack
harness: tests/system_chains/test_j17_fact_routing_chain.py
built: false
owner: trinity
# Skeleton: NOT RUN until ent#793 and fact routing (ent#804).
invariants: []

- id: J18
issue: "abilityai/trinity-enterprise#794"
promise: "A change I make to a library skill reaches every agent that holds it, and an agent without the permission cannot change skills."
actor: operator
lanes: [local]
tier: live-stack
harness: tests/system_chains/test_j18_library_change_chain.py
built: false
owner: trinity
# Needs a github.com skills repo the run can push to (CHAIN_SKILLS_REPO /
# CHAIN_SKILLS_GITHUB_TOKEN); without one it reports NOT RUN.
# SK-01..03: the assignment resolves, the assigned skill is delivered, the
# source does not silently stop syncing.
invariants: [SK-01, SK-02, SK-03]

- id: J19
issue: "abilityai/trinity-enterprise#794"
promise: "My agent's objective reaches me: the number it records shows against the target, and reads stale when it stops."
actor: user
lanes: [local]
tier: live-stack
harness: tests/system_chains/test_j19_objective_to_person_chain.py
built: false
owner: trinity
# Runs on any stack with Docker. The brief half reports NOT RUN: the platform
# puts no objective number in a brief (model-written only), so the chain is
# `partial` until it does.
# L-02: the declared metric is in the registry the create wrote.
invariants: [L-02]
64 changes: 64 additions & 0 deletions tests/registry.json
Original file line number Diff line number Diff line change
Expand Up @@ -5139,6 +5139,70 @@
"unit"
],
"description": "trinity-enterprise#753: list_skill_gates / set_skill_gate / clear_skill_gate reach their routes with names url-encoded; set_skill_gate sends only the fields given (an omitted field keeps its stored value, null resets the deadline, stray keys dropped); the zod schema refuses a bad approver or a deadline outside 1-168 (incl. bool/string); list_skills carries the library's approval recommendation; each tool has a TOOL_ACCESS_POLICY row and fits the 2,048-character description cap."
},
{
"file": "unit/test_ent794_chain_report.py",
"feature": "abilityai/trinity-enterprise#794",
"added": "2026-10-07",
"categories": [
"unit",
"system-test",
"reporting"
],
"description": "The system chain report's verdicts: every step passing is passed; a failing step is failed at that step by name; a skip is not run with its reason; a skip inside a step marks that step not run; a step that could not run makes the chain partial, never passed; a pass with no step recorded is a failure; the summary is a go only when every chain passed (an empty run is not evidence); JSON and markdown carry every chain and escape a pipe in a reason."
},
{
"file": "system_chains/test_j18_library_change_chain.py",
"feature": "abilityai/trinity-enterprise#794",
"added": "2026-10-07",
"categories": [
"system-test",
"live-stack",
"skills"
],
"description": "J18 (live, TRINITY_CHAIN_TESTS=1): a skill committed to a test github.com repo is synced into the library, assigned to two agents, and carried at the library's version in each container's platform-written .trinity-skill.json; a change to it reaches both holders through fleet re-inject (turned on for the run when the shipped default is off, and said so) with no per-agent action; an agent key without skills.manage is refused on itself and on another agent and changes nothing. NOT RUN without CHAIN_SKILLS_REPO / CHAIN_SKILLS_GITHUB_TOKEN."
},
{
"file": "system_chains/test_j19_objective_to_person_chain.py",
"feature": "abilityai/trinity-enterprise#794",
"added": "2026-10-07",
"categories": [
"system-test",
"live-stack",
"objectives",
"metrics"
],
"description": "J19 (live, TRINITY_CHAIN_TESTS=1): a canon objective names a target; the agent records the number with its own key; the Workspace role card and the operator objectives read show it owned against the target and not stale; with recording stopped it reads stale on the card past 2x cadence, keeping the last value. The brief step is recorded NOT RUN (the platform puts no objective number in a brief), so the chain reports partial."
},
{
"file": "system_chains/test_j15_rollout_chain.py",
"feature": "abilityai/trinity-enterprise#794",
"added": "2026-10-07",
"categories": [
"system-test",
"skeleton"
],
"description": "J15 skeleton: a fleet rolls out from one definition — reports NOT RUN, blocked by abilityai/trinity-enterprise#793 (the reference company)."
},
{
"file": "system_chains/test_j16_canon_fact_chain.py",
"feature": "abilityai/trinity-enterprise#794",
"added": "2026-10-07",
"categories": [
"system-test",
"skeleton"
],
"description": "J16 skeleton: a canon fact reaches its consumers — reports NOT RUN, blocked by ent#793 and the change-propagation work (ent#804); no canon-version or staleness record exists on dev."
},
{
"file": "system_chains/test_j17_fact_routing_chain.py",
"feature": "abilityai/trinity-enterprise#794",
"added": "2026-10-07",
"categories": [
"system-test",
"skeleton"
],
"description": "J17 skeleton: a fact reaches its owner — reports NOT RUN, blocked by ent#793 and fact routing (ent#804)."
}
]
}
4 changes: 3 additions & 1 deletion tests/run-full.sh
Original file line number Diff line number Diff line change
Expand Up @@ -239,8 +239,10 @@ run_tier journeys 900 journeys/
# TIER_DIRS is the single list. `NON_TIER_DIRS` are directories `api` must also
# not collect but that no tier owns: `manual/` and `deploy/` are opt-in, and
# `harness/` and the support packages hold no tests of their own.
# `system_chains/` is opt-in too: the 1.0 system chains run against a rolled-out
# fleet through `scripts/system/run_chains.sh` (TRINITY_CHAIN_TESTS=1), never here.
TIER_DIRS=(unit integration git_sync security scheduler_tests agent_server journeys)
NON_TIER_DIRS=(manual deploy harness fixtures testing_utils testkit)
NON_TIER_DIRS=(manual deploy harness fixtures testing_utils testkit system_chains)

# And the list is CHECKED against the tree, because a derived list is only as
# good as its inputs: a new directory under tests/ that nobody wired would still
Expand Down
Loading
Loading