[Feature] Add lineage fields to the event log for per-run trace reconstruction #841

Activity

1. addu390

   added

   feature[Issue Type] New features or improvements to existing features.[Issue Type] New features or improvements to existing features.

   priority/majorDefault priority of the PR or issue.Default priority of the PR or issue.

   on Jun 11, 2026
2. zxs1633079383 commented on Jul 6, 2026

   

   zxs1633079383

   on Jul 6, 2026

   Contributor

   More actions

   I took a pass through the current event-log/runtime shape and I think this can be a narrow first PR if the lineage data stays on the log record/context rather than inside user event attributes.

   Current insertion-point observation:

   • EventLogRecord is already the right wire object for the log; it currently carries EventContext + Event.
   • EventContext currently has only eventType and timestamp.
   • EventRouter.notifyEventProcessed(Event) only receives the event, so it cannot know which action emitted it.
   • RunnerContextImpl.sendEvent(Event) does know the current actionName, but that metadata is lost before the event reaches the logger.

   My suggested first-PR shape:

   1. Add a small lineage/context object to EventLogRecord or extend EventContext with optional fields:
      • logSchemaVersion
      • runId / traceId
      • eventId
      • sourceEventId or parentEventId
      • emittingAction
      • optionally the current key/sequence number if we can expose them without leaking unstable internals
   2. Keep these fields outside event.attributes, so the event payload remains user-owned and cross-language JSON-compatible.
   3. Treat each input event as the root of a run: runId = inputEvent.id, no parent/source event, no emitting action.
   4. For events emitted from an action, set runId from the source event, sourceEventId = actionTask.event.id, and emittingAction = action.name.
   5. Leave reverse edges out of the log, as proposed. They can still be derived from the agent plan's trigger rules during offline reconstruction.

   This keeps the first implementation scoped to recording and replay/debuggability. Visualization, OTel export, and trace-tree reconstruction can be follow-ups once the log has stable causal fields.
3. rosemarYuan commented on Jul 7, 2026

   

   rosemarYuan

   on Jul 7, 2026

   Contributor

   More actions

   Thanks for sharing the design direction @addu390 @zxs1633079383 . I took a look at the current runtime/event-log path, and one identity detail around runId seems worth settling before the schema lands.

   Using runId = inputEvent.id may not be stable enough. In Java, InputEvent is freshly created in wrapToInputEvent with a random UUID, so source replay after failover can give the same logical input a different run id. In Python, event ids are content-derived, so two distinct inputs with identical payloads can share the same run id and merge into one run.

   A more replay-stable option could be deriving runId from the stable job identifier, key, and per-key sequence number, for example jobIdentifier:key:sequenceNumber with a well-defined key encoding. This does not add a new identity concept: key + sequenceNumber is already part of the identity ActionStateUtil.generateKey relies on to match action state across failover.

   I am currently working on Memory observability, which seems closely related to event-log trace work. Do you have any near-term plans to work on this issue? If not, I would be happy to pick it up after finishing Memory observability.

   👍React with 👍1Reacted by Adesh Nalpet Adimurthy
4. zxs1633079383 commented on Jul 7, 2026

   

   zxs1633079383

   on Jul 7, 2026

   Contributor

   More actions

   Thanks @rosemarYuan, good catch. I agree that runId = inputEvent.id is not the right stable identity.

   I also checked the current Java/Python event identity behavior:

   • Java input events can get a fresh random UUID when wrapped, so replay/failover can change the id for the same logical input.
   • Python event ids are content-derived, so identical payloads can collide across distinct logical inputs.

   A cleaner split may be:

   • eventId: the concrete event id, preserving current event identity semantics.
   • runId / traceId: a replay-stable run identity derived from stable runtime coordinates, e.g. job identifier + key + per-key sequence number, with a defined key encoding.
   • sourceEventId / parentEventId: causal edge from the event being processed to the event emitted by an action.
   • emittingAction: the action that emitted the event.

   For the first PR, I think the important regression tests are:

   1. Java replay/wrapping should not produce a different run id for the same logical input position.
   2. Python identical payloads under different key/sequence positions should not collapse into one run.
   3. The lineage fields should stay outside user event attributes.

   Since @addu390 opened the issue and offered to submit a PR, it would be good to confirm whether they still plan to take it. I am happy to help review the design/tests or contribute a focused follow-up if useful.

   👍React with 👍1Reacted by Adesh Nalpet Adimurthy
5. addu390 commented on Jul 7, 2026

   

   addu390

   on Jul 7, 2026

   ContributorAuthor

   More actions

   Hi @zxs1633079383 @rosemarYuan, thanks for the interest on this issue, I haven't started working on it quite yet. Feel free to take up the issue and implementation. I can help with the review of the design.

   👍React with 👍1Reacted by YuanFeng
6. rosemarYuan commented on Jul 7, 2026

   

   rosemarYuan

   on Jul 7, 2026

   Contributor

   More actions

   Thanks @addu390, happy to take this up after I wrap up the current Memory observability work. I’ll keep the first PR scoped to recording-only and share the design/implementation for review.

   Agreed on the split for @zxs1633079383:

   • eventId: the concrete event id, preserving current event identity semantics.
   • runId: a replay-stable run identity derived from stable runtime coordinates.
   • sourceEventId / emittingAction: the causal edge from the processed event to the event emitted by an action.

   One small addition I think would help reconstruction:

   • emitIndex: the ordinal of an emitted event within the parent action invocation.

   The reason is that sourceEventId + emittingAction identifies which action invocation emitted an event, but it does not order multiple sibling events emitted by that same invocation. Recording emitIndex makes the sibling order explicit, instead of relying on log layout.

   This is useful because:

   • Python generator/async actions can drain emitted events across multiple polls.
   • Per-subtask event-log files may later be merged, where cross-file line order is not a strong contract.
   • (sourceEventId, emittingAction, emitIndex) gives v1 a stable emitted-event occurrence key and leaves a clean path for a later deterministic event-address scheme.

   I would define emitIndex as the ordinal within the parent action invocation, rather than just the local batch index.@zxs1633079383 thanks for the field shape, would welcome your review
7. wenjin272 commented on Jul 8, 2026

   

   wenjin272

   on Jul 8, 2026

   Contributor

   More actions

   Thanks to @addu390, @zxs1633079383, and @rosemarYuan for the great discussion.

   I noticed a few detailed questions around lineage tracking and field design, so I think it would be helpful to align on the design before moving forward with development. @rosemarYuan, could you please post a detailed design proposal in the GitHub discussion for everyone to review?

   👀React with 👀1Reacted by YuanFeng
8. rosemarYuan commented on Jul 8, 2026

   

   rosemarYuan

   on Jul 8, 2026

   Contributor

   More actions

   Thanks for assigning this to me. I’d be happy to take it on.

   I agree it would be better to align on the field semantics and lineage model before starting the implementation. I’ll prepare a detailed design proposal in the GitHub discussion soon, covering the proposed field set, run identity derivation, causal edge semantics, and the scope of the first recording-only PR.

   Looking forward to everyone’s feedback there.
9. wenjin272

   closed this as completedon Aug 5, 2026
10. wenjin272

    added

    fixVersion/0.4.0

    on Aug 5, 2026
11. wenjin272

    assigned

    rosemarYuan

    on Aug 5, 2026