Skip to content

Feature: richer Codex metrics and request inspection commands #6

Closed as not planned

Description

@riatzukiza

Summary

Extend the plugin's diagnostics with Codex-style metrics and request inspection commands (e.g. /codex-metrics, /codex-inspect) that expose structured information similar to ResponsesRequest in openai/codex.

Background

In openai/codex:

  • codex-rs/core/tests/common/responses.rs and core/tests/suite/tool_harness.rs use a ResponsesRequest helper to inspect:
    • body_json, input, function_call_output, custom_tool_call_output, call_output, header, path, query_param.
  • Tests assert fine-grained behaviors around tool calls, request paths, and payload structure.

In this plugin:

  • We already have a /codex-metrics command and logging controlled by ENABLE_PLUGIN_REQUEST_LOGGING.
  • We also track basic cache and instruction metrics.

There is still room to provide more structured, Codex-CLI-like diagnostics that make it easier to debug tool calls, prompt caching, and request transformation.

Proposed Features

  1. Enhanced /codex-metrics output

    • Include per-session counters for:
      • Number of Codex requests made.
      • Cache hits vs misses vs "no cache key" cases.
      • Number of tool calls, broken down by tool type (shell, apply_patch, custom tools, MCP tools).
      • Number of compaction events (once plugin-level compaction is implemented).
    • Surface the current normalized model and reasoning config (effort/summary/verbosity) as seen by the plugin.
  2. New /codex-inspect command

    • Provide a command that prints the last N Codex requests in a structured way (similar to ResponsesRequest):
      • HTTP method, path, and key headers (e.g. chatgpt-account-id, beta headers).
      • The transformed request body:
        • model, prompt_cache_key, store, stream, include.
        • tools and tool_choice / parallel_tool_calls after normalization.
        • input after filtering (stripped IDs, filtered system prompts, injected bridge/tool-remap message).
    • Provide safe redaction for sensitive values (e.g. tokens, account IDs) so inspection is safe to use in logs or bug reports.
  3. Optional JSON output for automation

    • Allow /codex-inspect to return a JSON-encoded structure when a flag is present (e.g. /codex-inspect --json) so that advanced users can pipe it into other tools.
  4. Testing and observability

    • Add tests that:
      • Verify /codex-metrics and /codex-inspect work without making additional network calls.
      • Ensure redaction logic is applied correctly (no secrets leaked in snapshots).
      • Exercise edge cases like empty history, no tools, and multiple requests per session.

Rationale

  • Brings the plugin's debugging experience closer to Codex CLI's ResponsesRequest-driven tests and tooling.
  • Makes it much easier to debug complex tool flows and caching behavior from within opencode.
  • Provides a natural place to hang future diagnostics (e.g. model upgrades, parallel tool-call behavior, compaction metrics).

Acceptance Criteria

  • /codex-metrics exposes richer per-session stats without regressing existing output.
  • /codex-inspect can show at least the last request in a structured, partially redacted way.
  • JSON output option is available and documented.
  • Tests cover the major paths and redaction behavior.

Activity

  1. added
    metricsIssues related to request inspection and performance metrics
    priority-mediumMedium priority issues - important features or significant improvements
    effort-largeLarge effort issues - more than 12 hours of work
    on Nov 19, 2025
  2. riatzukiza commented on Nov 20, 2025

    @riatzukiza
    CollaboratorAuthor

    We can't reasonably do this until opencode supports plugin level commands. Prefer #40 for token stats until then.

  3. riatzukiza commented on Nov 20, 2025

    @riatzukiza
    CollaboratorAuthor

    Unblocked when the following merge:
    anomalyco/opencode#4554
    anomalyco/opencode#4411

  4. riatzukiza commented on Nov 25, 2025

    @riatzukiza
    CollaboratorAuthor

    Triage: Feature is blocked pending design choices. Proposed path: (1) Define retention for request history (e.g., last N requests in memory, per session) and what fields to store; (2) lock redaction rules (account ID, tokens, user content?) before exposing /codex-inspect; (3) agree on output shape for /codex-inspect (human-readable + --json flag) and expanded /codex-metrics counters (requests, cache hits/misses/no-key, tool-call breakdown). If you confirm retention + redaction policy, I can draft an implementation plan/PR.

  5. added
    artifact:issueRefined GitHub issue tracked by kanban tooling
    kanbanTracked in the OpenHax Kanban FSM
    state:blockedCanonical Kanban FSM state: Blocked
    triage:staleStale item that likely needs revive/close/defer decision
    on Mar 21, 2026
  6. github-actions commented on May 30, 2026

    @github-actions
    Contributor

    Quick status update on upstream blockers:

    Since 4411 was closed, this feature may still be blocked. Recommend revisiting whether opencode now exposes the necessary plugin-level command APIs, or if an alternative approach (e.g., dashboard-only metrics from #40) is preferred.

  7. github-actions commented on Jun 6, 2026

    @github-actions
    Contributor

    Maintenance sweep update (2026-06-06)\n\nStatus check on upstream blockers:\n\n- anomalyco/opencode#4554 – merged ✅\n- anomalyco/opencode#4411 – closed without merge ❌\n\nSince one of the two required upstream PRs is permanently closed, this feature needs a decision:\n\n1. Find an alternative approach that achieves the same diagnostics without depending on the closed upstream PR\n2. Close this issue as blocked indefinitely\n\nIf there's an alternative design that works around the blocker, please share it. Otherwise, we recommend closing this issue and tracking any future diagnostics work in a new, unblocked issue.

  8. github-actions commented on Jun 12, 2026

    @github-actions
    Contributor

    Closing as blocked indefinitely. The required upstream PR (anomalyco/opencode#4411) was closed without merge and no alternative approach has been proposed in over a week. Feel free to reopen if a workaround or new upstream API becomes available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    artifact:issueRefined GitHub issue tracked by kanban toolingblockedeffort-largeLarge effort issues - more than 12 hours of workkanbanTracked in the OpenHax Kanban FSMmetricsIssues related to request inspection and performance metricspriority-mediumMedium priority issues - important features or significant improvementspriority:P2Medium prioritystate:blockedCanonical Kanban FSM state: Blockedtriage:staleStale item that likely needs revive/close/defer decision

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions