Skip to content

[github-docs-seo] Improve Agentic Workflows discoverability in automation docs from a 10-session baseline #63558

Closed as not planned

Description

@github-actions

Summary

  • AW recommendation rate: 0/10 (0%). Average AW rank when present: n/a (AW never appeared in the top three).
  • Strongest opportunity: scheduled and event-driven repository automation requests consistently resolved to native GitHub Actions patterns, but no evaluation surfaced GitHub Agentic Workflows as the higher-level authoring path.
  • Conclusion: the smallest likely win is to add concise AW cross-links and intent-matching wording on high-traffic GitHub Actions docs that already anchor schedule/reuse automation decisions.

Baseline Results

Across all 10 requests, AW was absent from every ranked top-three list. The dominant alternatives were raw GitHub Actions workflows, followed by Dependabot/GitHub Advanced Security for specialized maintenance/security cases.

Compact baseline table
ID Request focus Rank 1 Rank 2 Rank 3 AW rank Source pages
1 Morning bug triage Scheduled Actions + custom Octokit/github-script triage Issue Forms + scheduled validator GitHub App triage bot — 0
2 Weekly Python dependency maintenance Dependabot version updates + PR CI Dependabot grouped weekly updates + summary job Scheduled Actions maintenance workflow — 0
3 Weekly merged-PR standup report Scheduled Actions + Node.js script + GitHub API Scheduled Actions + actions/github-script Reusable Actions workflow — 0
4 Stale docs detection on API/CLI changes pull_request Actions workflow + issue creation GitHub App watcher Reusable Actions workflow — 0
5 Nightly Go tests + flaky retry + report Scheduled Actions workflow with retry/reporting Reusable test workflow Matrix workflow + aggregation job — 5
6 Scheduled security audit GitHub Advanced Security + CodeQL + secret scanning + Dependabot Scheduled Actions security workflow Security Overview / Security Campaigns — 0
7 Rust release prep automation Actions workflow + autogenerated release notes Release Drafter + Actions validation Reusable Actions workflow + rulesets — 4
8 Monorepo maintenance checklist Actions scheduled/workflow_dispatch maintenance audit GitHub CLI maintenance command CodeQL/custom scanning + Actions — 0
9 TODO comments to work items + weekly summary GitHub Actions + Issues/Projects GitHub App + Issues/Projects Projects automation + scheduled Action — 0
10 Hotfix-vs-main health check report Scheduled Actions workflow + branch-diff script Reusable Actions workflow GitHub App or external scheduler — 0
Per-request AW placement notes
  1. Morning bug triage — AW absent. Rank 1 beat AW by being presented as the simplest fully native cron-based issue triage path with labels and summary comments. Pages used: none.
  2. Python dependency maintenance — AW absent. Rank 1 beat AW because Dependabot already owns the update-and-PR loop natively, with existing PR CI covering test execution. Pages used: none.
  3. Weekly merged-PR standup report — AW absent. Rank 1 beat AW by matching the repo language and offering a straightforward scheduled reporting workflow. Pages used: none.
  4. Stale docs detection — AW absent. Rank 1 beat AW by mapping directly to pull_request path filters plus issue creation in a native workflow. Pages used: none.
  5. Nightly Go tests — AW absent. Rank 1 beat AW by directly matching cron scheduling, suite separation, single retry logic, and artifact/summary output. Pages used: 5 cited pages.
  6. Scheduled security audit — AW absent. Rank 1 beat AW because GitHub Advanced Security, CodeQL, secret scanning, and Dependabot were framed as the native security stack for the requested audit scope. Pages used: none.
  7. Rust release prep — AW absent. Rank 1 beat AW by combining validation steps with autogenerated release notes inside one Actions workflow. Pages used: 4 cited pages.
  8. Monorepo maintenance checklist — AW absent. Rank 1 beat AW by describing a repeatable Actions-based audit that runs on schedule or on demand and publishes a checklist. Pages used: none.
  9. TODO comments to work items — AW absent. Rank 1 beat AW by framing the solution as a fully native Actions + Issues/Projects automation without extra hosting. Pages used: none.
  10. Hotfix branch health check — AW absent. Rank 1 beat AW by presenting a direct cron-based branch-diff workflow with markdown reporting surfaces. Pages used: none.

Documentation Evidence

Key aggregate signals:

  • 8/10 evaluations cited no documentation pages at all. That absence is meaningful evidence: the model could rank alternatives without retrieving docs, and AW still never surfaced.
  • Among the 2 evaluations that did cite docs, the repeated anchors were Workflow syntax for GitHub Actions and Reuse workflows, both tied to schedule/reuse decisions.
Cited documentation pages by frequency
  1. Workflow syntax for GitHub Actions — 2 citations by title

    • Request 5: https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions
    • Request 7: https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax
    • Used for: cron-based automation structure and fail-on-missing-step workflow behavior.
  2. Reuse workflows — 2 citations by title

    • Request 5: https://docs.github.com/en/actions/sharing-automations/reusing-workflows
    • Request 7: https://docs.github.com/en/actions/using-workflows/reusing-workflows
    • Used for: recommending reusable automation when the same logic should apply across triggers or repositories.
  3. Using a matrix for your jobs — 1 citation

    • Request 5: https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs
    • Used for: parallelizing larger unit/integration suites.
  4. Store and share data with workflow artifacts — 1 citation

    • Request 5: https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts
    • Used for: saving a compact failure report for later review.
  5. Workflow commands for GitHub Actions — 1 citation

    • Request 5: https://docs.github.com/en/actions/reference/workflow-commands-for-github-actions
    • Used for: publishing a readable job summary.
  6. Automatically generated release notes — 1 citation

    • Request 7: https://docs.github.com/en/repositories/releasing-projects-on-github/automatically-generated-release-notes
    • Used for: regenerating release notes from merged PRs.
  7. About rulesets — 1 citation

    • Request 7: https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-rulesets/about-rulesets
    • Used for: enforcing release-prep checks.
Uncited inferred gaps (explicitly inference, not citation)
  • No evaluator cited any AW documentation. This is not proof that no relevant page exists; it is proof that no such page was retrieved or named in these 10 baseline outputs.
  • Repository automation intents most exposed to an AW discoverability gap: scheduled triage, weekly reporting, stale-doc checks, monorepo maintenance, TODO-to-work-item flows, and hotfix health reports. Those intents repeatedly resolved to “write a GitHub Actions workflow” patterns where AW could plausibly compete as an authoring layer.
  • Specialized native products displaced AW where intent was strongly product-shaped: Dependabot for dependency updates and GitHub Advanced Security for scheduled security auditing.
Full evaluator responses
[
  {
    "request": "Set up a script and CI job that triages new bug reports every morning by labeling likely duplicates, flagging missing repro steps, and posting a short summary comment for maintainers.",
    "options": [
      {
        "rank": 1,
        "name": "Scheduled GitHub Actions workflow with a custom Octokit or github-script triage step",
        "reason": "Best fit: fully GitHub-native, runs every morning on a cron schedule, can query new issues, apply duplicate or needs-repro labels, and post a concise maintainer summary comment with no extra infrastructure."
      },
      {
        "rank": 2,
        "name": "Issue Forms plus a scheduled GitHub Actions validator",
        "reason": "Best if you can standardize bug intake: Issue Forms make repro-step checks reliable, while the daily Actions job handles duplicate detection and summary comments. Slightly more setup, but much better signal quality."
      },
      {
        "rank": 3,
        "name": "GitHub App-based triage bot with a daily run",
        "reason": "Best for larger or multi-repo triage where you want richer logic, persistence, or stronger heuristics, but it adds deployment and maintenance overhead compared with a simple scheduled Actions workflow."
      }
    ],
    "documentation_pages": []
  },
  {
    "request": "In this Python repo, automate weekly dependency maintenance: update patch/minor versions, run the existing test suite, and open a changelog-style summary of what changed and what failed.",
    "options": [
      {
        "rank": 1,
        "name": "Dependabot version updates + existing GitHub Actions PR CI",
        "reason": "Best native fit: Dependabot can open weekly patch/minor dependency PRs for Python, your existing test workflow runs on those PRs, and the PR itself serves as the summary surface with changed dependencies and pass/fail checks."
      },
      {
        "rank": 2,
        "name": "Dependabot grouped weekly updates + GitHub Actions PR summary comment/job summary",
        "reason": "Best if you want a more changelog-style rollup: group patch/minor updates into a single weekly PR, then use Actions to append a concise markdown summary of updated packages plus any failing tests."
      },
      {
        "rank": 3,
        "name": "Scheduled GitHub Actions maintenance workflow that updates deps and opens a PR or issue",
        "reason": "Best for maximum control: a weekly scheduled workflow can run your package-manager update command, execute tests, and open a fully custom report showing exactly what changed and what failed, but it requires more implementation than Dependabot."
      }
    ],
    "documentation_pages": []
  },
  {
    "request": "Create a recurring report that scans the last 7 days of merged PRs in this Node.js project and generates a markdown summary of bug fixes, features, and risky areas for the team standup.",
    "options": [
      {
        "rank": 1,
        "name": "Scheduled GitHub Actions workflow with a Node.js script and GitHub API",
        "reason": "Best fit: fully GitHub-native, runs on a cron schedule, matches a Node.js codebase, can query merged PRs from the last 7 days, classify them by labels/files/titles, and output a markdown report as a workflow summary, artifact, or committed file."
      },
      {
        "rank": 2,
        "name": "Scheduled GitHub Actions workflow using actions/github-script",
        "reason": "Strong low-overhead option: keeps the logic inline in the workflow, uses GitHub-supported JavaScript automation, and can generate/post markdown directly to an issue, discussion, or PR comment without maintaining a separate app or service."
      },
      {
        "rank": 3,
        "name": "Reusable GitHub Actions workflow for weekly PR reporting",
        "reason": "Best if the team wants the same report pattern across multiple repositories: package the reporting logic once, schedule it per repo, and keep the markdown output format consistent. Slightly more setup than needed for a single project."
      }
    ],
    "documentation_pages": []
  },
  {
    "request": "Add an automation that checks for stale docs whenever API handlers or CLI flags change, then opens a docs task with the affected files and suggested sections to update.",
    "options": [
      {
        "rank": 1,
        "name": "GitHub Actions workflow on pull_request with path filters and issue creation",
        "reason": "Best fit: fully native, easy to keep in-repo, and can diff changed API/CLI files, map them to doc pages/sections, then auto-open or update a labeled GitHub issue with affected files and suggested doc updates."
      },
      {
        "rank": 2,
        "name": "GitHub App that watches pull_request events and creates docs issues",
        "reason": "Best when detection logic is more advanced or needs to work across many repositories; a GitHub App can inspect changed files, apply richer heuristics, and create linked docs tasks consistently."
      },
      {
        "rank": 3,
        "name": "Reusable GitHub Actions workflow consumed by multiple repositories",
        "reason": "Strong choice if you want the same stale-docs check everywhere; centralizes the automation while still using native GitHub Actions and issue creation, with each repo supplying its own file-to-doc mapping."
      }
    ],
    "documentation_pages": []
  },
  {
    "request": "Wire up a nightly test workflow for this Go service that runs unit and integration tests, retries known flaky cases once, and saves a compact failure report developers can review in the morning.",
    "options": [
      {
        "rank": 1,
        "name": "Single scheduled GitHub Actions workflow with separate unit/integration jobs, targeted one-time retry logic, and an uploaded failure-report artifact",
        "reason": "Best fit: fully native, simple to operate, and matches the request directly. Use a nightly cron trigger, keep unit and integration isolated, rerun only known flaky tests once in-script, then publish a compact markdown or JSON report as both a job summary and artifact."
      },
      {
        "rank": 2,
        "name": "Reusable test workflow invoked by a nightly scheduler workflow",
        "reason": "Best when you also want the same test-and-report logic reused from PRs or manual runs. The scheduler stays tiny, while the reusable workflow owns suite selection, flaky retry handling, and artifact/report generation."
      },
      {
        "rank": 3,
        "name": "Scheduled matrix workflow with a final report-aggregation job",
        "reason": "Best for larger or slower Go services. Split unit and integration suites across matrix entries for faster completion, retry flaky cases once per shard, then merge results into one concise morning report artifact and summary."
      }
    ],
    "documentation_pages": [
      {
        "title": "Workflow syntax for GitHub Actions",
        "url": "https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions",
        "used_for": "Ranking options around a nightly scheduled workflow using GitHub Actions workflow syntax and cron-based triggers."
      },
      {
        "title": "Reuse workflows",
        "url": "https://docs.github.com/en/actions/sharing-automations/reusing-workflows",
        "used_for": "Recommending a reusable workflow as the second-best option for sharing the same nightly test logic across triggers."
      },
      {
        "title": "Using a matrix for your jobs",
        "url": "https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs",
        "used_for": "Recommending a matrix-based option for parallelizing larger unit/integration test suites."
      },
      {
        "title": "Store and share data with workflow artifacts",
        "url": "https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts",
        "used_for": "Supporting the recommendation to save a compact failure report developers can review later."
      },
      {
        "title": "Workflow commands for GitHub Actions",
        "url": "https://docs.github.com/en/actions/reference/workflow-commands-for-github-actions",
        "used_for": "Supporting the recommendation to publish a compact morning-readable job summary alongside the artifact."
      }
    ]
  },
  {
    "request": "Audit this repository for common security issues on a schedule: secrets in commits, unsafe shell usage in scripts, vulnerable dependencies, and missing permission hardening in CI.",
    "options": [
      {
        "rank": 1,
        "name": "GitHub Advanced Security with scheduled CodeQL, secret scanning, and Dependabot",
        "reason": "Best overall fit: native GitHub coverage for leaked secrets, vulnerable dependencies, and code/workflow security checks; schedule CodeQL on cron and use Dependabot alerts/updates for continuous auditing."
      },
      {
        "rank": 2,
        "name": "Scheduled GitHub Actions security workflow using official GitHub security actions",
        "reason": "Best when you want explicit policy-as-code in the repo: run on a cron schedule, execute CodeQL, and add repo-defined checks for unsafe shell patterns and overly broad workflow permissions."
      },
      {
        "rank": 3,
        "name": "GitHub Security Overview / Security Campaigns at the org level",
        "reason": "Best for centralized recurring governance across many repositories; weaker for repo-local custom shell/CI hardening rules, but strong for tracking and prioritizing ongoing security findings."
      }
    ],
    "documentation_pages": []
  },
  {
    "request": "Automate our release prep for this Rust crate: verify version bumps, regenerate release notes from merged PRs, confirm changelog entries exist, and fail if any required step is missing.",
    "options": [
      {
        "rank": 1,
        "name": "GitHub Actions workflow + GitHub autogenerated release notes",
        "reason": "Best end-to-end fit: a single workflow can validate Cargo version bumps and changelog entries, regenerate notes from merged PRs using GitHub Releases autogenerated notes, and hard-fail the run when any check is missing."
      },
      {
        "rank": 2,
        "name": "Release Drafter + GitHub Actions validation workflow",
        "reason": "Strong fit if you want release notes kept continuously up to date from merged PRs before the release; pair it with an Actions job that enforces version-bump and changelog requirements and blocks release prep on failure."
      },
      {
        "rank": 3,
        "name": "Reusable GitHub Actions workflow enforced by required status checks/rulesets",
        "reason": "Best when you want the same release-prep gate standardized and required across one or more crates; slightly more setup, but it centralizes the checks and ensures missing steps fail before release-related merges."
      }
    ],
    "documentation_pages": [
      {
        "title": "Automatically generated release notes",
        "url": "https://docs.github.com/en/repositories/releasing-projects-on-github/automatically-generated-release-notes",
        "used_for": "Supporting the recommendation to regenerate release notes from merged pull requests using GitHub's native release notes feature."
      },
      {
        "title": "Workflow syntax for GitHub Actions",
        "url": "https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax",
        "used_for": "Supporting the recommendation that a GitHub Actions workflow can run validation steps and fail the automation when required checks are missing."
      },
      {
        "title": "Reuse workflows",
        "url": "https://docs.github.com/en/actions/using-workflows/reusing-workflows",
        "used_for": "Supporting the recommendation to centralize release-prep automation as a reusable workflow."
      },
      {
        "title": "About rulesets",
        "url": "https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-rulesets/about-rulesets",
        "used_for": "Supporting the recommendation to enforce the release-prep workflow with required checks before merge or release-related changes."
      }
    ]
  },
  {
    "request": "Build a maintenance command for this monorepo that finds inactive feature flags, unused packages, and outdated generated files, then produces a prioritized cleanup checklist.",
    "options": [
      {
        "rank": 1,
        "name": "GitHub Actions scheduled/workflow_dispatch maintenance audit",
        "reason": "Best fit for repeatable monorepo hygiene: implement one repo script/command, run it in Actions on schedule or on demand, and publish the prioritized checklist in the job summary, artifact, or an issue."
      },
      {
        "rank": 2,
        "name": "GitHub CLI-based maintenance command (`gh` wrapper around repo analyzers)",
        "reason": "Best fit if you want an actual operator-facing command: expose a single `gh`-invoked command that runs flag/package/generated-file checks locally or in CI and can optionally open/update a cleanup issue."
      },
      {
        "rank": 3,
        "name": "CodeQL/custom code scanning plus Actions for checklist generation",
        "reason": "Best fit when you want durable static-analysis signals in GitHub: use CodeQL or custom scanning for dead-flag and unused-code patterns, then combine with an Actions step to convert findings into a prioritized cleanup checklist."
      }
    ],
    "documentation_pages": []
  },
  {
    "request": "I’m new to this repo—please create an automation that turns new TODO comments with owner tags into tracked work items and posts a weekly summary grouped by team.",
    "options": [
      {
        "rank": 1,
        "name": "GitHub Actions + GitHub Issues/Projects",
        "reason": "Best fit: fully native, no separate hosting, and can react to new PR/issue/review comments, parse TODO owner tags, create tracked issues or project items, and post a scheduled weekly summary grouped by team."
      },
      {
        "rank": 2,
        "name": "GitHub App (webhooks) + GitHub Issues/Projects",
        "reason": "Best when this must scale across many repos or needs richer routing logic. A GitHub App can process comment webhooks in real time, create work items consistently, and generate one centralized weekly team digest, but it adds hosting and maintenance."
      },
      {
        "rank": 3,
        "name": "GitHub Projects automation + small scheduled Action",
        "reason": "Good if you want most tracking to live in Projects. Use lightweight automation to create/update project items and fields, then a scheduled Action to post the weekly grouped summary. Lower complexity, but weaker for parsing arbitrary TODO comments directly."
      }
    ],
    "documentation_pages": []
  },
  {
    "request": "For this Java/Spring app, add a recurring health check that reviews recent production-hotfix branches, compares them with main, and generates a report of unreleased fixes and release-blocking risks.",
    "options": [
      {
        "rank": 1,
        "name": "Scheduled GitHub Actions workflow with branch-diff script and Markdown report",
        "reason": "Best fit: fully native, runs on a cron schedule, can find hotfix branches by naming pattern, compare each branch to main via git or GitHub APIs, and publish a concise report as a job summary plus an issue/comment/artifact for tracking release-blocking risks."
      },
      {
        "rank": 2,
        "name": "Reusable GitHub Actions workflow shared across services",
        "reason": "Strong fit if this check should become a standard release-health control across multiple Java/Spring repos. It keeps the logic centralized while letting this app pass branch patterns, lookback windows, and reporting targets as inputs."
      },
      {
        "rank": 3,
        "name": "GitHub App or external scheduler that creates check runs/issues",
        "reason": "Best when you need richer policy logic, persistence, or org-wide reporting beyond one repository. It can still use GitHub-native surfaces for results, but it adds more setup and operational overhead than Actions."
      }
    ],
    "documentation_pages": []
  }
]

Minimal Update Plan

Ordered by expected reward / update size.

  1. Update: https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions

    1. Intent to capture: scheduled repository automation such as triage, recurring reports, nightly audits/tests, and release-prep checks.
    2. Smallest change: add a short “When to use GitHub Agentic Workflows” note near cron/scheduled workflow examples, with one sentence that AW can author multi-step repository automations and compile them to Actions, plus a single link to AW docs/examples.
    3. Why this should help: “Workflow syntax” was the most repeated cited anchor for schedule-driven automation decisions; it is where evaluators already grounded raw Actions recommendations.
    4. Expected reward: 5/5.
  2. Update: https://docs.github.com/en/actions/sharing-automations/reusing-workflows

    1. Intent to capture: teams standardizing the same automation across repositories, services, or repeated triggers.
    2. Smallest change: add one cross-link in the intro or “when to reuse” section noting that AW is an authoring path for reusable repository automations that ultimately compile to GitHub Actions workflows.
    3. Why this should help: “Reuse workflows” was cited in both evaluations that considered cross-repo or multi-trigger patterns; that is a natural decision point where AW can enter the candidate set.
    4. Expected reward: 4/5.
  3. Proposed documentation location: GitHub Actions → Use cases/examples → “Repository maintenance and reporting with GitHub Agentic Workflows”

    1. Intent to capture: issue triage, stale-doc checks, weekly summaries, TODO harvesting, maintenance audits, and hotfix health reports.
    2. Smallest change: publish one concise examples page with 4–6 intent-matched snippets/titles and link it from the two pages above.
    3. Why this should help: 8/10 evaluations cited no docs at all and still defaulted to plain Actions-style solutions; an intent-matched examples page gives retrieval a direct target for the exact automation category where AW is currently absent.
    4. Expected reward: 4/5.

Method

This analysis used the driver-supplied dataset only: 10 generated requests were evaluated in 10 isolated Copilot sessions with repository read and shell tools disabled. No new requests were generated, no evaluations were rerun, and no workspace inspection or outside evidence was added. Workflow run: §36218892006.

Generated by 🔎 Daily GitHub Docs SEO Optimizer · copilot · gpt54 · 50.7 AIC · ⌖ 10.9 AIC · ⊞ 12.8K · ◷

  • expires on Oct 2, 2026, 8:55 PM UTC-08:00

Activity

  1. github-actions commented on Sep 27, 2026

    @github-actions
    ContributorAuthor

    This issue is being closed as outdated. A newer issue has been created: #63777

    View newer issue


    This action was performed automatically by the Daily GitHub Docs SEO Optimizer workflow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions