Repository navigation
test(skills): 400+ behavioral tests for 15 skills + extend test:py to cover skills/*/tests/ - #1207
bassilkhilo-ag2 wants to merge 16 commits into
Conversation
…test:py Covers parse_transcript (role normalization, continuation merging), classify_call (refusal/error/silence/repeat patterns, no-match, case-insensitive), and find_snippet (first match, 120-char truncation, empty fallback). Documents that "I cannot" alone doesn't trigger refusal — only "sorry, I cannot" does. Extends test:py to pick up skills/*/tests/*.test.py and adds test:sh script. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…find_metrics, _build_transcript Covers _truncate (5), _ts_iso (3), analyze_turns (15), _build_transcript (6), find_call (7), find_metrics (6). Uses _make_db() to create conversation.sqlite schema in-memory; patches dc.DB_FILE per test class for isolation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers status/remaining_pct display (human), all JSON fields + available flag, gate exits (0=allowed, 1=blocked), missing reset headers, 0%/100% utilization. Injects fake workspace_default so module loads without live credential proxy. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…es + categorize/repair Covers merge_timeline (5), parse_ts (4), _ts_short (2), diagnose rules 1-4/8-9 (12), categorize_issue (9), _classify_repair (6), analyze_patterns_and_repair (4). Injects fake workspace_default; points SQLITE_PATH/METRICS_PATH at nonexistent paths so tests run without live data. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…llect, render, clobber guard Covers parse_frontmatter (5), collect_entries (6), render (6), main (6). Validates clobber guard rejects overwrite when 0 frontmatter entries but MEMORY.md has non-trivial content. No external deps required. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…cache (18) test_source_tag.test.py: unittest wrapper that discovers the 9 existing tests in scripts/test_source_tag.py so they run under npm test:py. asset_cache.test.py: covers _basename_from_url (6), cache_get (3), cache_put (4), preload_run_from_cache (3), promote_run_to_cache (2). Documents that trailing-slash URLs yield the dir segment (not "asset") and root paths yield the domain name. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…s, render, chi-fleet Covers slugify (6), validate_title (11 — length, trailing period, all reject patterns, valid outcomes), suggest_reframes (4), render_long_description (4), is_chi_fleet (4 — env var, case-insensitive, no match). No subprocess calls. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…omain whitelist Covers md5_of_file (3), validate() all 6 rejection rules + valid path (15), main() exit codes and --known-404-hash flag (3). Mocks detect_image_dims and ocr_text so no subprocess/PIL required. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…on, listing filter Covers parse_price (5), extract_chip (9), extract_ram_gb (8), extract_storage_gb (5), passes() (13 — accessory/Intel/no-chip/excluded/wrong-chip/low-ram/low-storage/high-price /full-match/soft-match/no-price/empty-chips). Version guard exits cleanly on Python <3.10 (scan.py uses PEP 604 union syntax). Run with python3.10. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers resolve_workspace (env override, default, tilde expansion), pid_alive (zero/None/own/dead/PermissionError/OSError/negative), and read_discovery (valid JSON, missing file, malformed JSON). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers resolve_workspace, make_id (format + uniqueness), row_to_agent (active/stale/stopped detection, meta parsing), prune (old stopped/stale removed, active/recent kept), and all op_ functions (register with upsert, heartbeat with stopped-guard and 404, deregister idempotency, agents with inline prune, health count+uptime). Uses in-memory SQLite injected into rs._db for isolation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
verify_video.test.py (19): all 7 ffprobe checks (file-not-found, zero-byte, codec, dims, audio, duration-tolerance, av-sync), main(). reminders.test.py (21): list_reminder_lists, list_reminders (missing-value due/body, completed flag, error/empty), add_reminder, complete_reminder — all via mocked run_applescript. contacts.test.py (11): search_contacts parsing — multi-email, dedup, empty, error, separator edge cases — via mocked subprocess.run. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
calendar_reader.test.py (18): read_events — event parsing, missing-value location/notes, all_day, sorted output, timeout, access-denied and generic errors, empty/short-line edge cases, days param. format_for_humans — error, no events, title/location, all-day label. email_sender.test.py (9): escape — backslash, double-quote, newline, combined. send — success, draft mode, cc recipient, error exits 1. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
subtitle.test.py (14): find_latest_recording (found/missing/empty), transcribe (SRT success, no SRT, wav cleanup), burn_subtitles (success, missing output, suffix), main (no recording/transcription/burn failure, success, explicit video arg). record.test.py (14): _list_audio_devices (single/multi device, no audio section, ffmpeg exception, index+name, video devices excluded), _has_audio_device, _pick_audio_device (macbook mic wins, non-virtual fallback, all-virtual fallback, none when empty, air mic matches). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tests) generate.test.py (9): load_env — plain key-value, double/single-quoted value stripping, comment skip, blank line skip, equals in value, missing file no crash, overwrites stale shell env, key whitespace stripped. share_screen_modal.test.py (10): COORDS structure (keys present, values are int 2-tuples), main() — dry-run exits 0 and prints all keys, stop clicks once, modal clicks 3 times and skips activate_mcp_chrome, full mode clicks 4 times. Quartz injected as fake module at load time. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
skills/tests/publish.test.py — covers generate_readme(): - returns "" when SKILL.md absent - readme header includes skill dir name - install section with clone + symlink instructions - zero-scripts count when no scripts/ dir - counts and lists scripts from scripts/ subdir - script stem hyphen/underscore → space formatting - "## When to Use" extraction and next-heading stop - fallback to "See SKILL.md" when section absent - MIT license + Sutando credit present package.json test:py glob extended: skills/tests/*.test.py added alongside existing skills/*/tests/*.test.py so root-level skill scripts (publish.py) have a natural home for their tests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Bassil Khilo seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
VasiliyRad
left a comment
There was a problem hiding this comment.
Solid infrastructure expansion. The test:py glob now covers tests/, skills/tests/, and skills/*/tests/ — single npm test exercises everything. [ -f "" ] && ... correctly skips unmatched globs. test:sh slot reserved for shell tests. Spot-checked registry-client.test.py — proper importlib.util.spec_from_file_location for hyphenated filenames, env-var-driven workspace, surgical mocks of os.kill for pid_alive edges. Quality looks consistent.
This is 22 new test files / 4560 lines across 15 skills — too much to review line-by-line. Question: does npm test pass green on your machine right now? If yes, ship it; if there's any flake or env dep, please call it out so we know what to expect on first CI run.
Risk note: once this lands, every CI run iterates 22 more test files. Any test that depends on host state (network, specific dirs, env vars beyond what the test sets) becomes a recurring failure. The skip-on-glob-miss pattern means we won't notice if a test file is accidentally deleted, only if a test asserts wrong. Worth a future follow-up to add a "counted test files" assertion ("at least N skills have tests") so we can't silently lose coverage.
23 files, 1 production change (package.json scripts entry). LGTM 👍
Summary
Adds behavioral tests for 15 skills that had zero test coverage, and wires them into
npm testvia an extendedtest:pyglob.Tests added (16 commits, ~400 tests)
skills/regression-search/find-regression.pyfind-regression.test.pyskills/regression-search/diagnose-call.pydiagnose-call.test.pyskills/quota-tracker/read-quota.pyread-quota.test.pyskills/call-diagnostics/diagnose.pydiagnose.test.pyskills/cross-node-sync/regenerate-memory-index.pyregenerate-memory-index.test.pyskills/make-viral-video/validate_asset.pyvalidate_asset.test.pyskills/make-viral-video/source_tag.pytest_source_tag.test.pyskills/make-viral-video/asset_cache.pyasset_cache.test.pyskills/make-viral-video/verify_video.pyverify_video.test.pyskills/submit-use-case/submit_use_case.pysubmit_use_case.test.pyskills/agent-registry/registry-client.pyregistry-client.test.pyskills/agent-registry/registry-service.pyregistry-service.test.pyskills/deal-finder/scan.pyscan.test.pyskills/discord-voice/share_screen_modal.pyshare_screen_modal.test.pyskills/image-generation/generate.pygenerate.test.pyskills/screen-record/subtitle.pysubtitle.test.pyskills/screen-record/record.pyrecord.test.pyskills/macos-tools/calendar_reader.pycalendar_reader.test.pyskills/macos-tools/email_sender.pyemail_sender.test.pyskills/macos-tools/contacts.pycontacts.test.pyskills/macos-tools/reminders.pyreminders.test.pyskills/tests/publish.pypublish.test.pypackage.jsonchangeExtends
test:pyglob fromtests/*.test.py→tests/*.test.py skills/tests/*.test.py skills/*/tests/*.test.pyso all skill tests are discovered bynpm test. Addstest:shfor future shell test files. Without this change the skill tests exist butnpm testnever runs them.All tests are pure unit tests — no network calls, no filesystem side-effects, no external dependencies beyond stdlib.