[docs] Restructure the observability chapter by sangkyoonnam · Pull Request #1262 · apache/flink-agents

sangkyoonnam · 2026-10-15T10:43:49Z

Linked issue: #1235

Purpose of change

Covers all of #1235 in one PR, as suggested in review. monitoring.md now reads as one observability chapter:

Overview: what metrics, logs and the Trace Log each show, and how they fit together in an investigation.
Metrics: how to reach them (reporters, the Web UI, with a link to Flink's dashboard docs) and what the Scope column means, then the built-in tables with shorter descriptions. Qualifications that matter to users, such as how restore affects input-run counts and latency or how replayed results are counted, move to numbered notes under each table, and units are stated once.
Logs: minimal Python and Java examples for logging inside an Action.
Trace Log: the SLF4J and file outputs in one section, internal reporting and timing details trimmed, and the compatibility notes removed, since the release notes will cover migration.

The trace-log, json-format and recording-level anchors that other pages link to are kept. The Chinese docs aren't touched.

Tests

Docs only. The site builds with Hugo 0.110.0, the version docs/setup_hugo.sh pins, with no warnings, and every in-page anchor resolves. Restated metric and Trace Log behavior was checked against the runtime, for example that an input run already active at restore records neither an outcome nor latency (BuiltInInputRunMetricsTest) and that the default Trace Log appender writes plain JSON (%msg%n in Slf4jTraceLogger).

API

No.

Documentation

doc-needed
doc-not-needed
doc-included

Was this patch authored or co-authored using generative AI tooling?

Yes
No

Generated-by: Claude Code 2.1.301 (Claude Opus 5.5)

…ance Generated-by: Claude Code 2.1.296 (Claude Opus 5.5)

wenjin272 · 2026-10-16T13:14:33Z

Thanks for the overview and scope improvements! Since #1219 has merged, could you also include the remaining metrics and logging work from #1235 in this PR?

Simplify the metrics descriptions, keeping their meanings, scopes, units, and essential boundaries. Put necessary qualifications next to the relevant metric or in numbered notes, and add the Flink Web UI metrics documentation link.
Restructure the logging and Trace Log sections to fit the observability overview, trim internal timing details, and add minimal Java/Python examples for logging inside an Action.
Remove the remaining compatibility sections; we’ll cover migration details in the release notes.

This would complete the observability chapter restructuring in one PR.

… Log Generated-by: Claude Code 2.1.301 (Claude Opus 5.5)

sangkyoonnam · 2026-10-16T17:52:30Z

@wenjin272 Done in c4837fc, so this now covers the rest of #1235. The metric tables keep each metric's meaning, scope and unit, with the qualifications moved to numbered notes under each table and a link to Flink's Web UI metrics docs. Logging has minimal Python and Java examples inside an Action, the Trace Log sections drop the internal timing and reporting details, and the compatibility notes are gone. I've updated the title and description to match.

wenjin272

Thanks for restructuring the observability chapter and addressing the earlier feedback! The metrics and logging guidance is much clearer now. I have three small clarification suggestions below to help users locate metrics and choose Trace Log targets.

wenjin272 · 2026-10-19T06:05:00Z

+| **Model Resource**, **Tool**, **Skill**, **MCP Server**, **Subagent** | `action.<action_name>.<kind>.<name>`, where `<kind>` is `model_resource`, `tool`, `skill`, `mcp_server` or `subagent` | `action.tool_call_action.tool.get_weather.numOfToolCallsSucceeded` |

-Execution latency is measured from matching start and outcome observations of the same call in the same task attempt. Assigning a call identity before preparation or submission does not start the timer. Invoked Tools retain their own start and completion timestamps, even when the reports arrive after a durable call or parallel batch returns. Queueing and time spent waiting for other calls in the batch are excluded. Outcome counters still update when no matching start is available.
+Each group below the Agent scope is a key-value group: `<action_name>` is the Action's name, `<model_name>` the model a chat model reports usage for, and `<name>` the resource, Tool, Skill, MCP Server or sub-agent name. Reporters that support key-value groups, such as Prometheus, expose them as labels.


Could you briefly explain that chat_model_action and tool_call_action are built-in Action names and link to the relevant documentation? Users may not know which name to look for in metric paths.

Updated in bb48488: the paragraph after the scope table now identifies chat_model_action and tool_call_action as built-in Actions.

wenjin272 · 2026-10-19T06:05:00Z

+#### Token Usage Metrics

-Metric names listed in the built-in tables above are reserved in their corresponding scopes. Custom metrics must use different names within the same scope.
+Token usage metrics are recorded when chat models are invoked through `ChatModelConnection`, which helps track LLM API usage and cost.


Please clarify that token usage in the built-in chat flow appears under action.chat_model_action.model.<model_name>, rather than the user Action that sends ChatRequestEvent.

Clarified in bb48488: the built-in chat flow records token usage metrics under action.chat_model_action.model.<model_name>, not under the Action that sends ChatRequestEvent.

wenjin272 · 2026-10-19T06:05:00Z

-#### Event and Action Metrics
+- **Metrics** show how much and how fast: Event and Action throughput, input-run outcomes and latency, LLM, Tool and sub-agent calls, and token usage. They go through [Flink's metric system](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/ops/metrics/), so they reach the same reporters and dashboards as the rest of the job.
+- **Logs** carry what your Action code and the framework write through Flink's logging framework, such as warnings and stack traces.
+- **The [Trace Log](#trace-log)** records Events, Action executions and the calls made within Actions as structured JSON, so you can follow what one input did. By default it records only Events; configure `trace-log.targets` to include Actions and component calls.


Could you introduce EVENT_ONLY (Events only, the default) and ALL (all entity types observed by the framework) before the Recording Targets example? Detailed matching rules can remain in the linked configuration reference.

Updated in bb48488: Recording Targets now introduces EVENT_ONLY (Events only, the default) and ALL (all entity types the framework observes) before the example. Matching rules remain in the configuration reference.

…Log scope presets Generated-by: Claude Code 2.1.302 (Claude Opus 5.5)

wenjin272

Thanks for addressing my comments. LGTM

[docs] Add an observability overview and metric access and scope guid…

47474c5

…ance Generated-by: Claude Code 2.1.296 (Claude Opus 5.5)

github-actions Bot added doc-included Your PR already contains the necessary documentation updates. fixVersion/0.4.0 priority/major Default priority of the PR or issue. labels Oct 15, 2026

sangkyoonnam added 3 commits October 15, 2026 22:15

[ci] Retrigger CI

7e89956

[ci] Retrigger CI

554327c

[ci] Retrigger CI

d5e5475

[docs] Simplify metric descriptions and restructure logging and Trace…

c4837fc

… Log Generated-by: Claude Code 2.1.301 (Claude Opus 5.5)

sangkyoonnam changed the title ~~[docs] Add an observability overview and metric access and scope guidance~~ [docs] Restructure the observability chapter Oct 16, 2026

github-actions Bot added doc-included Your PR already contains the necessary documentation updates. and removed doc-included Your PR already contains the necessary documentation updates. labels Oct 16, 2026

wenjin272 reviewed Oct 19, 2026

View reviewed changes

[docs] Name the built-in Actions in metric paths and introduce Trace …

bb48488

…Log scope presets Generated-by: Claude Code 2.1.302 (Claude Opus 5.5)

wenjin272 approved these changes Oct 19, 2026

View reviewed changes

wenjin272 merged commit 0ff883e into apache:main Oct 19, 2026
28 checks passed

Navigation Menu

Repository navigation

[docs] Restructure the observability chapter - #1262

[docs] Restructure the observability chapter#1262
wenjin272 merged 6 commits into
apache:mainfrom
sangkyoonnam:docs/1235-observability-overview-v2

sangkyoonnam commented Oct 15, 2026 •

edited

Loading

Uh oh!

wenjin272 commented Oct 16, 2026

Uh oh!

sangkyoonnam commented Oct 16, 2026

Uh oh!

wenjin272 left a comment

Uh oh!

wenjin272 Oct 19, 2026

Uh oh!

sangkyoonnam Oct 19, 2026

Uh oh!

wenjin272 Oct 19, 2026

Uh oh!

sangkyoonnam Oct 19, 2026

Uh oh!

wenjin272 Oct 19, 2026

Uh oh!

sangkyoonnam Oct 19, 2026

Uh oh!

wenjin272 left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

sangkyoonnam commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of change

Tests

API

Documentation

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

wenjin272 commented Oct 16, 2026

Uh oh!

sangkyoonnam commented Oct 16, 2026

Uh oh!

wenjin272 left a comment

Choose a reason for hiding this comment

Uh oh!

wenjin272 Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

sangkyoonnam Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

wenjin272 Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

sangkyoonnam Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

wenjin272 Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

sangkyoonnam Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

wenjin272 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sangkyoonnam commented Oct 15, 2026 •

edited

Loading