Skip to content

test(skills): 400+ behavioral tests for 15 skills + extend test:py to cover skills/*/tests/ - #1207

Closed
bassilkhilo-ag2 wants to merge 16 commits into
sonichi:mainfrom
bassilkhilo-ag2:test/regression-search-coverage
Closed

bassilkhilo-ag2 wants to merge 16 commits into
sonichi:mainfrom
bassilkhilo-ag2:test/regression-search-coverage

Conversation

@bassilkhilo-ag2

Copy link
Copy Markdown
Collaborator

Summary

Adds behavioral tests for 15 skills that had zero test coverage, and wires them into npm test via an extended test:py glob.

Tests added (16 commits, ~400 tests)

Skill / module Test file Tests
skills/regression-search/find-regression.py find-regression.test.py 30
skills/regression-search/diagnose-call.py diagnose-call.test.py 42
skills/quota-tracker/read-quota.py read-quota.test.py 19
skills/call-diagnostics/diagnose.py diagnose.test.py 43
skills/cross-node-sync/regenerate-memory-index.py regenerate-memory-index.test.py 23
skills/make-viral-video/validate_asset.py validate_asset.test.py 21
skills/make-viral-video/source_tag.py test_source_tag.test.py 9
skills/make-viral-video/asset_cache.py asset_cache.test.py 18
skills/make-viral-video/verify_video.py verify_video.test.py 27 (approx)
skills/submit-use-case/submit_use_case.py submit_use_case.test.py 29
skills/agent-registry/registry-client.py registry-client.test.py 13
skills/agent-registry/registry-service.py registry-service.test.py 40
skills/deal-finder/scan.py scan.test.py 40
skills/discord-voice/share_screen_modal.py share_screen_modal.test.py ~10
skills/image-generation/generate.py generate.test.py ~10
skills/screen-record/subtitle.py subtitle.test.py ~25
skills/screen-record/record.py record.test.py ~25
skills/macos-tools/calendar_reader.py calendar_reader.test.py ~15
skills/macos-tools/email_sender.py email_sender.test.py ~15
skills/macos-tools/contacts.py contacts.test.py ~10
skills/macos-tools/reminders.py reminders.test.py ~10
skills/tests/publish.py publish.test.py 14

package.json change

Extends test:py glob from tests/*.test.py → tests/*.test.py skills/tests/*.test.py skills/*/tests/*.test.py so all skill tests are discovered by npm test. Adds test:sh for future shell test files. Without this change the skill tests exist but npm test never runs them.

All tests are pure unit tests — no network calls, no filesystem side-effects, no external dependencies beyond stdlib.

Bassil Khilo and others added 16 commits May 25, 2026 01:52
…test:py

Covers parse_transcript (role normalization, continuation merging),
classify_call (refusal/error/silence/repeat patterns, no-match, case-insensitive),
and find_snippet (first match, 120-char truncation, empty fallback).
Documents that "I cannot" alone doesn't trigger refusal — only "sorry, I cannot" does.
Extends test:py to pick up skills/*/tests/*.test.py and adds test:sh script.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…find_metrics, _build_transcript

Covers _truncate (5), _ts_iso (3), analyze_turns (15), _build_transcript (6),
find_call (7), find_metrics (6). Uses _make_db() to create conversation.sqlite
schema in-memory; patches dc.DB_FILE per test class for isolation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers status/remaining_pct display (human), all JSON fields + available flag,
gate exits (0=allowed, 1=blocked), missing reset headers, 0%/100% utilization.
Injects fake workspace_default so module loads without live credential proxy.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…es + categorize/repair

Covers merge_timeline (5), parse_ts (4), _ts_short (2), diagnose rules 1-4/8-9
(12), categorize_issue (9), _classify_repair (6), analyze_patterns_and_repair (4).
Injects fake workspace_default; points SQLITE_PATH/METRICS_PATH at nonexistent
paths so tests run without live data.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…llect, render, clobber guard

Covers parse_frontmatter (5), collect_entries (6), render (6), main (6).
Validates clobber guard rejects overwrite when 0 frontmatter entries but
MEMORY.md has non-trivial content. No external deps required.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…cache (18)

test_source_tag.test.py: unittest wrapper that discovers the 9 existing tests
in scripts/test_source_tag.py so they run under npm test:py.

asset_cache.test.py: covers _basename_from_url (6), cache_get (3), cache_put (4),
preload_run_from_cache (3), promote_run_to_cache (2). Documents that trailing-slash
URLs yield the dir segment (not "asset") and root paths yield the domain name.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…s, render, chi-fleet

Covers slugify (6), validate_title (11 — length, trailing period, all reject
patterns, valid outcomes), suggest_reframes (4), render_long_description (4),
is_chi_fleet (4 — env var, case-insensitive, no match). No subprocess calls.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…omain whitelist

Covers md5_of_file (3), validate() all 6 rejection rules + valid path (15),
main() exit codes and --known-404-hash flag (3). Mocks detect_image_dims and
ocr_text so no subprocess/PIL required.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…on, listing filter

Covers parse_price (5), extract_chip (9), extract_ram_gb (8), extract_storage_gb (5),
passes() (13 — accessory/Intel/no-chip/excluded/wrong-chip/low-ram/low-storage/high-price
/full-match/soft-match/no-price/empty-chips). Version guard exits cleanly on Python
<3.10 (scan.py uses PEP 604 union syntax). Run with python3.10.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers resolve_workspace (env override, default, tilde expansion),
pid_alive (zero/None/own/dead/PermissionError/OSError/negative),
and read_discovery (valid JSON, missing file, malformed JSON).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers resolve_workspace, make_id (format + uniqueness), row_to_agent
(active/stale/stopped detection, meta parsing), prune (old stopped/stale
removed, active/recent kept), and all op_ functions (register with upsert,
heartbeat with stopped-guard and 404, deregister idempotency, agents with
inline prune, health count+uptime).

Uses in-memory SQLite injected into rs._db for isolation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
verify_video.test.py (19): all 7 ffprobe checks (file-not-found,
zero-byte, codec, dims, audio, duration-tolerance, av-sync), main().

reminders.test.py (21): list_reminder_lists, list_reminders
(missing-value due/body, completed flag, error/empty), add_reminder,
complete_reminder — all via mocked run_applescript.

contacts.test.py (11): search_contacts parsing — multi-email, dedup,
empty, error, separator edge cases — via mocked subprocess.run.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
calendar_reader.test.py (18): read_events — event parsing, missing-value
location/notes, all_day, sorted output, timeout, access-denied and generic
errors, empty/short-line edge cases, days param. format_for_humans — error,
no events, title/location, all-day label.

email_sender.test.py (9): escape — backslash, double-quote, newline,
combined. send — success, draft mode, cc recipient, error exits 1.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
subtitle.test.py (14): find_latest_recording (found/missing/empty),
transcribe (SRT success, no SRT, wav cleanup), burn_subtitles (success,
missing output, suffix), main (no recording/transcription/burn failure,
success, explicit video arg).

record.test.py (14): _list_audio_devices (single/multi device, no audio
section, ffmpeg exception, index+name, video devices excluded),
_has_audio_device, _pick_audio_device (macbook mic wins, non-virtual
fallback, all-virtual fallback, none when empty, air mic matches).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tests)

generate.test.py (9): load_env — plain key-value, double/single-quoted
value stripping, comment skip, blank line skip, equals in value, missing
file no crash, overwrites stale shell env, key whitespace stripped.

share_screen_modal.test.py (10): COORDS structure (keys present,
values are int 2-tuples), main() — dry-run exits 0 and prints all keys,
stop clicks once, modal clicks 3 times and skips activate_mcp_chrome,
full mode clicks 4 times. Quartz injected as fake module at load time.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
skills/tests/publish.test.py — covers generate_readme():
  - returns "" when SKILL.md absent
  - readme header includes skill dir name
  - install section with clone + symlink instructions
  - zero-scripts count when no scripts/ dir
  - counts and lists scripts from scripts/ subdir
  - script stem hyphen/underscore → space formatting
  - "## When to Use" extraction and next-heading stop
  - fallback to "See SKILL.md" when section absent
  - MIT license + Sutando credit present

package.json test:py glob extended: skills/tests/*.test.py added
alongside existing skills/*/tests/*.test.py so root-level skill scripts
(publish.py) have a natural home for their tests.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


Bassil Khilo seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@VasiliyRad VasiliyRad left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid infrastructure expansion. The test:py glob now covers tests/, skills/tests/, and skills/*/tests/ — single npm test exercises everything. [ -f "" ] && ... correctly skips unmatched globs. test:sh slot reserved for shell tests. Spot-checked registry-client.test.py — proper importlib.util.spec_from_file_location for hyphenated filenames, env-var-driven workspace, surgical mocks of os.kill for pid_alive edges. Quality looks consistent.

This is 22 new test files / 4560 lines across 15 skills — too much to review line-by-line. Question: does npm test pass green on your machine right now? If yes, ship it; if there's any flake or env dep, please call it out so we know what to expect on first CI run.

Risk note: once this lands, every CI run iterates 22 more test files. Any test that depends on host state (network, specific dirs, env vars beyond what the test sets) becomes a recurring failure. The skip-on-glob-miss pattern means we won't notice if a test file is accidentally deleted, only if a test asserts wrong. Worth a future follow-up to add a "counted test files" assertion ("at least N skills have tests") so we can't silently lose coverage.

23 files, 1 production change (package.json scripts entry). LGTM 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants