Skip to content

fix: preserve observed engine session initialization - #76815

Merged
pelikhan merged 4 commits into
mainfrom
pelikhan/turbo-winner
Nov 11, 2026
Merged

pelikhan merged 4 commits into
mainfrom
pelikhan/turbo-winner

Conversation

@pelikhan

Copy link
Copy Markdown
Collaborator

Recent engine runs retained initialization metadata in raw logs or canonical traces but lost it from the unified session artifact. This preserves that observed evidence without inventing tool activity, accounting, or execution outcomes.

Changes

  • Retain agentVersion and provider in unified session.init/session.start, with matching types and generated schemas.
  • Extract Crush model, local provider, and session identifiers only from its first harness-framed, recognized startup prefix. Preserve diagnostics separately and leave quoted examples and unknown level-prefixed answer text untouched.
  • Update focused regressions, documentation, changeset, and compiled Crush/Kiro workflows.

Observed evidence

Validation

  • make agent-report-progress passed, including type checking, lint, schema freshness, focused tests, and workflow-lock drift checks.
  • Four focused JavaScript suites passed: 305 tests.
  • Replayed the existing Kiro and Crush native artifacts and confirmed the missing metadata survives canonical and unified projection.
  • make recompile compiled all 365 workflows. No workflows were dispatched or rerun.

Review status

Draft pending final review. The earlier automated security review found no confirmed vulnerabilities, but could not review the final revised Crush parser and generated locks because its synchronous reviewer could not accept follow-up. This PR does not claim that review gate passed.

@pelikhan
pelikhan marked this pull request as ready for review November 11, 2026 08:53
Copilot AI dynamic review requested due to automatic review settings November 11, 2026 08:53
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

❌ PR Specification Reviewer failed to deliver outputs. Please review the logs for details.

Generated by PR Specification Reviewer for #76815

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · routed: haiku55 xhigh · ~1 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 7.2K · ◷

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Ponytail Reviewer · routed: gpt61 medium · ~4 AIC · ⌖ ~1 AIC · ◇ ~1 AIC · ⊞ 181 · ◷

Generated by Ponytail Reviewer for #76815

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

PR Code Quality Reviewer

Outcome: reported_action

The agent completed and reported one or more actions. The no-op output is a completion signal, not a claim that no action was taken.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from PR Code Quality Reviewer · gpt56 · ~4 AIC · ⌖ ~1 AIC · ◇ ~1 AIC · ⊞ 1.6K · ◷

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · routed: haiku55 xhigh · ~2 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 8.5K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments that could not be inline-anchored

.github/workflows/shared/crush.md:322

This early return drops `nativeInit` (and its diagnostic) whenever the harness-framed preamble is followed by a valid canonical JSON array/JSONL session, because only `records` and `terminal` are returned here. That violates T-UAS-009 and the new Section 3.5 initialization obligation to preserve observed initialization fields in canonical/full unified sessions. Include the startup events before returning the recognized records, and add a regression fixture for preamble + canonical session output...

Generated by PR Specification Reviewer for #76815 · copilot · gpt50 · ~3 AIC · ⌖ ~2 AIC · ⊞ 1.6K
Comment /review-specs to run again

@github-actions

Copy link
Copy Markdown
Contributor

Comment Memory

Peek at saved memory (pr-code-quality-reviewer)
reviewed_at: 2026-11-11T08:56:29Z
review_event: REQUEST_CHANGES
top_themes: [fragile exact-wording parser allowlist, all-or-nothing metadata promotion]
files_reviewed: [.github/workflows/shared/crush.md, actions/setup/js/unified_session_payload.cjs, docs/public/schemas/unified-session.schema.json]
comment_count: 2

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~4 AIC · ⌖ ~1 AIC · ⊞ 1.6K · ◷
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found two correctness risks in the Crush startup parser. The exact-wording allowlist can silently drop all metadata after a CLI wording change, and the all-or-nothing model/session gate discards valid partial observations. Please make parsing resilient and emit independently observed fields. Static review only; repository code was not executed per review instructions.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~4 AIC · ⌖ ~1 AIC · ⊞ 1.6K
Comment /review to run again

Comment thread .github/workflows/shared/crush.md Outdated
const nativeLine = /^(?:INFO {2}(?:Running in non-interactive mode|Overriding large model for non-interactive run provider=\S+ model=\S+|Created session for non-interactive run session_id=[a-f0-9-]+|ModelProvider called provider=\S+ model=\S+)|WARN {2}(?:Failed to parse skill file path=.+ error="no YAML frontmatter found"|Ripgrep \(rg\) not found in \$PATH\. Some grep features might be limited or slower\.|Using large model as small model for unknown provider provider=\S+ model=\S+))\r?$/;
const firstAnswer = nativeLines.findIndex(line => !nativeLine.test(line));
const nativePreamble = nativeLines.slice(0, firstAnswer < 0 ? nativeLines.length : firstAnswer).join("\n");
const model = nativePreamble.match(/^INFO {2}Overriding large model for non-interactive run provider=(\S+) model=(\S+)\r?$/m);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The all-or-nothing gate here discards valid partial evidence: if the model/provider line is observed but the session-id line is absent or changes format, nativeInit is omitted entirely. Since all fields in SessionInitData are optional, emit session.init with whichever independently validated fields were observed and keep the remaining startup lines in the diagnostic event.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 96d94dd: provider/model and session ID are extracted independently, preserving whichever metadata is observed; framed startup lines remain diagnostics. Both Crush locks were regenerated, and final validation passed.

Comment thread .github/workflows/shared/crush.md Outdated
const executionStart = logContent.match(/^\[crush-harness\] starting Crush CLI\r?\n/m);
const preamble = executionStart ? logContent.slice(executionStart.index).match(/^\[crush-harness\] starting Crush CLI\r?\n(INFO {2}Running in non-interactive mode(?:\r?\n(?:INFO|WARN|ERROR) {2}[^\r\n]*)*)/) : undefined;
const nativeLines = preamble?.[1].split("\n") ?? [];
const nativeLine = /^(?:INFO {2}(?:Running in non-interactive mode|Overriding large model for non-interactive run provider=\S+ model=\S+|Created session for non-interactive run session_id=[a-f0-9-]+|ModelProvider called provider=\S+ model=\S+)|WARN {2}(?:Failed to parse skill file path=.+ error="no YAML frontmatter found"|Ripgrep \(rg\) not found in \$PATH\. Some grep features might be limited or slower\.|Using large model as small model for unknown provider provider=\S+ model=\S+))\r?$/;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This parser only promotes metadata when the startup lines match an exact wording allowlist. A minor Crush wording/version change therefore makes nativeInit undefined and silently drops all observed model/provider/session metadata—the regression this change is intended to prevent. Please retain the recognized harness preamble as diagnostics and either parse fields independently or emit a warning when the framed preamble is present but does not match.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 96d94dd: metadata extraction no longer depends on the exact startup-message allowlist, and changed startup logger lines are retained as diagnostics. Both Crush locks were regenerated, and final validation passed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

After startup-prefix removal, the quiet-mode decoder can misclassify JSON answers as saved-session tool activity, while [] drops all events, including initialization and diagnostics.

1 open issue
What changed in this PR

This PR preserves observed engine initialization metadata in gh-aw’s unified session artifacts and separates recognized Crush startup diagnostics from assistant output.

Changes:

  • Retain CLI version and provider fields in unified initialization events, types, and schemas.
  • Extract Crush startup model, provider, and session identifiers.
  • Add regressions, update documentation, and regenerate Crush and Kiro workflow locks.
File Description
docs/​src/​content/​docs/​specs/​unified-agent-session-specification.md Document retained initialization fields.
docs/​public/​schemas/​unified-session.schema.json Add version and provider to unified schemas.
docs/​public/​schemas/​agent-session.schema.json Add version and provider to canonical schema.
actions/​setup/​js/​unified_session_payload.test.cjs Test initialization field retention and aliases.
actions/​setup/​js/​unified_session_payload.cjs Preserve version and provider during projection.
actions/​setup/​js/​types/​unified_session.d.ts Extend unified initialization types.
actions/​setup/​js/​types/​agent_session.d.ts Extend canonical initialization types.
actions/​setup/​js/​parse_deepseek_log.test.cjs Verify DeepSeek metadata survives projection.
actions/​setup/​js/​kiro_session_pipeline.test.cjs Verify Kiro initialization survives the pipeline.
actions/​setup/​js/​crush_session.test.cjs Cover startup extraction and answer preservation.
.github/​workflows/​smoke-kiro.lock.yml Refresh generated workflow hashes.
.github/​workflows/​smoke-crush.lock.yml Include the updated Crush parser.
.github/​workflows/​shared/​kiro.md Update comments describing version retention.
.github/​workflows/​shared/​crush.md Extract startup metadata and separate diagnostics.
.github/​workflows/​engine-conformance-kiro.lock.yml Refresh generated workflow hashes.
.github/​workflows/​engine-conformance-crush.lock.yml Include the updated Crush parser.
.changeset/​patch-observed-session-initialization.md Record the patch release change.

Review effort: Balanced (auto)

Comment thread .github/workflows/shared/crush.md Outdated
Comment on lines +330 to +332
if (nativeInit) {
logEntries.push({ type: "session.init", data: nativeInit });
logEntries.push({ type: "crush.diagnostic", data: { content: nativePreamble } });

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /diagnosing-bugs, /tdd and /codebase-design. No blocking issues. Two suggestions are posted inline.

📋 Key Themes & Highlights

Key Themes

  • Partial preamble handling (shared/crush.md): session.init needs both the model and session lines. If either is missing, the preamble leaks into answer text. This predates the PR, so it is not a regression, but it is the gap this PR targets.
  • Untyped new fields (unified_session_payload.cjs): provider and agentVersion skip the type guards used by neighbouring fields.
  • Undocumented event: crush.diagnostic is not in the unified types, EVENT_FIELDS, or the spec.
  • Test gap: no test covers a partial preamble (only the session line or only the model line missing).

Positive Highlights

  • ✅ The preamble is framed and matched against an allowlist, so quoted or unframed text is not promoted to session.init.
  • ✅ The Kiro test now compares against the raw parse, which is a cleaner assertion.
  • ✅ A patch changeset is included.

Scope

Reviewed against HEAD ae44735. The local working tree is dirty and differs from the PR head, so I did not review those edits. Vitest was not run in this environment.

@copilot please address the review comments above.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~7 AIC · ⌖ ~2 AIC · ⊞ 6.6K
Comment /matt to run again

Comment thread .github/workflows/shared/crush.md Outdated
const nativePreamble = nativeLines.slice(0, firstAnswer < 0 ? nativeLines.length : firstAnswer).join("\n");
const model = nativePreamble.match(/^INFO {2}Overriding large model for non-interactive run provider=(\S+) model=(\S+)\r?$/m);
const session = nativePreamble.match(/^INFO {2}Created session for non-interactive run session_id=([a-f0-9-]+)\r?$/m);
const nativeInit = model && session ? { sourceEngine: "crush", provider: model[1], model: model[2], sessionId: session[1] } : undefined;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/diagnosing-bugs] nativeInit requires both the Overriding large model and the Created session lines. If either is missing, session.init is dropped and the harness preamble lines stay in assistant.message text. That is the leak this PR is meant to prevent. Consider emitting session.init from whichever fields are present, and always stripping the framed preamble once [crush-harness] starting Crush CLI matches.

@copilot please address this.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 96d94dd: initialization retains partial observed fields, and framed startup output is separated from assistant text. Both Crush locks were regenerated, and final validation passed.

"session.init": {
sourceEngine: ["sourceEngine"],
agentVersion: ["agentVersion", "agent_version"],
provider: ["provider"],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/codebase-design] provider and agentVersion are copied by selectFields without a type check. The neighbouring fields are guarded (for example isNonBlank). Add a string guard so a non-string value from a parser cannot reach the unified session.

@copilot please address this.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 96d94dd: non-string agentVersion and provider values are omitted from both session.init and session.start projections, with regressions added.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot

🍳 A little prep to help this PR along! Copilot assignment requested for the following outstanding work:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
  2. Review (.github/workflows/shared/crush.md:291): address unresolved feedback - Review comment
  3. Review (.github/workflows/shared/crush.md:288): address unresolved feedback - Review comment
  4. Review (.github/workflows/shared/crush.md:293): address unresolved feedback - Review comment
  5. Review (actions/setup/js/unified_session_payload.cjs:225): address unresolved feedback - Review comment

Keeping the changes focused; human review and CI get the final taste test. No merging or CI triggering.

Structured data
{
  "completed_task": 0,
  "head": "681a2632f8f3e84827817bbc8b6a330c282dd7e0",
  "kind": "requested",
  "schema_version": "1",
  "state": "8270951a1e6e6414f5b45397c79c7bc23e4f284f845f350e706c37d533a2b1c1",
  "work": [
    "06e3a7ad1b2a8d03f715ee2c2da9a424737a14053eab55e24fb90dfc96ea4010",
    "847a92e67259c6e7507cd70a9a23253dd618e85dbcf4d03e31a3f0e46a35ef41",
    "d7de3a9f32e6e2e597af85948b5537e83ae94726abfbb4fed9c2d9f22e7fa511",
    "ec7788b22e1517bbb2739e43b81b8e1c11a841f5e69b1ac79241eb3c2fb615b0"
  ],
  "workflow": "pr-sous-chef"
}

Generated by 👨🍳 PR Sous Chef · pi · routed: haiku55 xhigh · ~2 AIC · ⌖ ~2 AIC · ⊞ 6.9K · ◷
Comment /souschef to run again

Copilot AI and others added 2 commits November 11, 2026 09:38
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot AI commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

@copilot

🍳 A little prep to help this PR along! Copilot assignment requested for the following outstanding work:...

Merged latest main in d7f5d0c and addressed the four review comments in 96d94dd. Final validation passed, including 642 impacted setup JavaScript tests and workflow-lock drift checks for all 365 workflows. CodeQL timed out and its tool prohibited retry, so that review remains incomplete. No workflows were dispatched.

Copilot AI requested a review from gh-aw-bot November 11, 2026 09:53
@pelikhan
pelikhan merged commit a653869 into main Nov 11, 2026
68 checks passed
@pelikhan
pelikhan deleted the pelikhan/turbo-winner branch November 11, 2026 15:25
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v1.2.3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants