Skip to content

Firefox API watch: a review issue for each new Firefox release - #674

Closed
Nawid3333 wants to merge 6 commits into
mainfrom
feature/firefox-api-watch
Closed

Nawid3333 wants to merge 6 commits into
mainfrom
feature/firefox-api-watch

Conversation

@Nawid3333

@Nawid3333 Nawid3333 commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

Draft, not for merging yet: this waits for the owner's review.

What it adds

firefox-api-watch.yml runs daily (07:40 UTC) and by hand. For each new Firefox Stable release it reads:

  • MDN's notes for developers
  • browser-compat-data
  • Bugzilla's fixed bugs
  • the add-ons blog

It keeps what touches FastStream and posts it to one review issue, one section per release. Relevance comes from tools/firefox-api-watch.topics.json plus the change's names searched in the tracked files under chrome/.

Each finding is a box to tick and shows:

  • area
  • what changed
  • the FastStream files it touches
  • priority and confidence
  • links to Mozilla/MDN

Issue lifecycle

  • A. No open issue → the first release with findings opens Firefox API Watch: starting with Firefox N.
  • B. An open issue → each later release is a comment on it, in version order.
  • C. The bot never edits or deletes anything it posted, so ticked boxes and the owner's notes stay.
  • D. A closed issue is never touched. Its hidden markers still count, so the next issue copies nothing from it.
  • E. Nothing relevant → no post; the run's summary says so.

Catching up: missed releases are reviewed (the current release and the two before it). Repeated runs post nothing twice. Notes Mozilla adds to a release later come as an "added after its first review" section.

Waiting and failing:

  • Missing or work-in-progress MDN notes wait up to 10 days after the release.
  • An outage waits for the next run.
  • A source that answers wrongly fails the run. Failures open "Firefox API watch failed"; the next clean run closes it.

Safety

  • permissions: {} for the workflow; the job has contents: read and issues: write.
  • No secret and no environment. It never pushes or dispatches, so it cannot start a release.
  • Outside text goes through md(), so a post carries no HTML, no mentions, no issue references, no autolinks and no links outside Mozilla/MDN. It also can't forge a marker; this is property-tested.

Checked locally

  • pnpm run lint: 0 problems. pnpm run typecheck: 0 errors.
  • Full unit suite on the last commit (a647967): 231 files and 3602 tests passed, 6 skipped as on main.
  • actionlint: clean. zizmor: no findings.
  • In WSL: tests/workflows/firefox-api-watch.test.sh, no-mentions.test.sh and keepalive.test.sh pass. The full workflow suite was not run locally; CI runs it here.
  • A live dry run over Firefox 153–160 gave 96 findings: 6 high, 43 medium, 47 low. The 6 high ones were checked by hand and are real.

Review

An independent review by a separate agent found these real problems (eight findings; one had two parts), all fixed with tests in the last commit:

  • a marker could be forged inside a code span;
  • a merged note was hidden;
  • a later change to a reported feature was never reported;
  • sections could be posted out of version order;
  • false failure alarms;
  • a long outage of a shared source never failed the run;
  • links to any github.com repository were allowed;
  • folding counted <details> inside code;
  • a lagging issue list could open a second issue.

After the fixes: the 50 watcher unit tests and the workflow test (r1–r8) pass, and so do lint and typecheck. The live dry run's counts are unchanged.

The usual glm review was not possible: the local Ollama was not running.

🤖 Generated with Claude Code

Nawid3333 and others added 3 commits November 13, 2026 15:06
tools/firefox-api-watch.mjs reads what a Firefox Stable release changed
(MDN's developer notes, browser-compat-data, Bugzilla's fixed bugs, the
add-ons blog), keeps what touches FastStream (topics in
firefox-api-watch.topics.json, names searched in the tracked files under
chrome/) and posts one section per release to a review issue: the first
release opens the issue, later ones are comments. It never edits or
deletes a post, so ticked boxes and the owner's notes stay; what it
reported is read back from hidden markers in its own posts, open and
closed issues alike, so a closed issue is never touched and nothing is
listed twice.

Outside text goes through md(): no HTML, mentions, issue references,
autolinks or chosen links, and no forged marker (property-tested).
Missing or work-in-progress notes wait up to 10 days; an outage waits
for the next run; a source that answers wrongly fails the run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Daily at 07:40 UTC and by hand (dry_run, from). One run at a time,
permissions {} with contents: read + issues: write for the job, no
secret; it never pushes or dispatches, so it cannot start a release.
An Actions cache entry remembers the reviewed version so quiet days
stop after the version check. A failed run opens "Firefox API watch
failed", closed by the next clean run (the upstream-watch pair).

tests/workflows/firefox-api-watch.test.sh runs the real step with a
stub gh and fixture sources.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread tools/firefox-api-watch.mjs Fixed
Comment thread tools/firefox-api-watch.mjs Fixed
Comment thread tools/firefox-api-watch.mjs Fixed
Comment thread tools/firefox-api-watch.mjs Fixed
Comment thread tools/firefox-api-watch.mjs Fixed
Comment thread tools/firefox-api-watch.mjs Fixed
Comment thread tools/firefox-api-watch.mjs Fixed
Comment thread tools/firefox-api-watch.mjs Fixed
Comment thread tools/firefox-api-watch.mjs Fixed
Comment thread tools/firefox-api-watch.mjs Fixed
- A marker could be forged in a code span: one pass of removing comment
  openers left "<!<!---- ffw-item bug:N --->->" as a marker that read
  back as "reported". Openers are now removed until none is left.
- Two notes merged into one finding showed only the first; the second
  (possibly the compatibility risk the kind came from) is now shown too.
- A later change to a reported feature was never reported: a BCD key is
  now the change (feature, release, added or removed), and a finding is
  news when one of its merged items has a key of its own not reported
  before (a note reworded under the same bug is not news).
- Sections could be posted out of version order: a release never posted
  now holds back the ones after it.
- An older release timing out while looked at again failed the run at
  once, and a shared source down for weeks never did: only the newest
  release unreviewed past the 10 grace days fails it now. Notes MDN never
  writes no longer block: past the grace days the release is reviewed
  from the other sources, which the section says.
- github.com links only to the specs' and Mozilla's owners (a link to
  this repository's issue would reference it); short commit hashes of
  letters only are broken up too.
- The folded list counted "<details>" in a finding's code.
- A run tries the issue it just opened as well (the list can lag).
- A Bugzilla answer of 1000 bugs or more fails the run (cut short).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread tools/firefox-api-watch.mjs Fixed
Nawid3333 and others added 2 commits November 13, 2026 17:11
CodeQL here runs the extended suite with the remote-and-local threat model, so the command
line, the environment and files read at run time count as untrusted. Each alert is fixed where
it starts, not silenced:

- Requests (SSRF, file data in a request): FFW_URLS, which let the environment point every
  source at any URL or file: URL, is gone. The workflow test names a fixtures folder instead
  (FFW_FIXTURES, answered by fixtureFetch). getUrl asks only https URLs on the five sources'
  hosts, the MDN URL is made from a constant prefix and a number, and the BCD tarball's URL is
  made from its file name, not taken from the registry's answer.
- Paths: the memo folder (--memo) and the fixtures folder must lie in RUNNER_TEMP or the
  system's temporary folder (tempFolder); FFW_BCD_FILE is gone (bcd.json in the fixtures).
- Regular expressions from data: topics.json becomes topics.mjs with RegExp literals (all 186
  patterns checked identical to the old strings); identifiers made into patterns go through
  escapeRegExp, which escapes the backslash too.
- HTML comments: stripped until none is left (stripComments; a single pass left
  "<!<!-- a -->-- b -->" a comment of its own), --!> ends one, an unclosed one hides the rest;
  the code-span defusing removes --!> too.

Tests: unit 51 + 5 (new: hosts, temporary folders, fixture answers, comment stripping, escaping,
the topics' patterns), firefox-api-watch.test.sh all 8 scenarios in WSL, a live dry run for
Firefox 160 (10 findings, as before).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… only inside its temp folder

CodeQL's two alerts left on #674: GITHUB_STEP_SUMMARY and GITHUB_OUTPUT are paths from the
environment, appended to as they came. They go through the same guard as the memo and the
fixtures now (tempFolder, renamed tempPath): the runner keeps both in RUNNER_TEMP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Nawid3333

Copy link
Copy Markdown
Owner Author

Folded into the test branch test/all-prs (347a3b6), which carries every open pull request except #634 (mpv) and goes to main as one merge after the owner's test. Closed so that only main, mpv-update and the test branch remain.

@Nawid3333 Nawid3333 closed this Nov 14, 2026
@Nawid3333
Nawid3333 deleted the feature/firefox-api-watch branch November 14, 2026 15:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants