Repository navigation
scenario: add web-to-md tool for converting web pages to markdown - #61
Merged
Merged
Conversation
Implements a modular web-to-markdown converter that: - Converts web pages to clean markdown with YAML frontmatter - Organizes content by domain (saves to content_dirs/sites/) - Downloads and manages images in domain-specific directories - Detects and rejects paywall/auth-walled content (Medium, NYT, etc.) - Generates searchable index of all saved pages - Supports resume for interrupted sessions - Integrates with amplifier.config.paths for workspace consistency Architecture follows "bricks and studs" pattern with 8 focused modules: fetcher, converter, validator, image_handler, enhancer, organizer, indexer, and state management. Also fixes stub violation in knowledge_assist output_generator. Usage: make webtomd URL=https://example.com 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Rename directory: scenarios/webtomd/ → scenarios/web_to_md/ - Update Makefile command: webtomd → web-to-md (consistent with dot-to-mermaid pattern) - Update package name and Python module references - Update state file paths: .data/webtomd → .data/web_to_md - Update logger names and documentation - Add make command documentation to README - Preserve git history for all renamed files 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Marc Goodner (robotdad)
force-pushed
the
webtomd
branch
from
October 7, 2025 17:15
29799d8 to
a283f22
Compare
5 tasks done
Joi Ito (Joi)
pushed a commit
to Joi/amplifier
that referenced
this pull request
Oct 31, 2025
…crosoft#61) * feat: add webtomd tool for converting web pages to markdown Implements a modular web-to-markdown converter that: - Converts web pages to clean markdown with YAML frontmatter - Organizes content by domain (saves to content_dirs/sites/) - Downloads and manages images in domain-specific directories - Detects and rejects paywall/auth-walled content (Medium, NYT, etc.) - Generates searchable index of all saved pages - Supports resume for interrupted sessions - Integrates with amplifier.config.paths for workspace consistency Architecture follows "bricks and studs" pattern with 8 focused modules: fetcher, converter, validator, image_handler, enhancer, organizer, indexer, and state management. Also fixes stub violation in knowledge_assist output_generator. Usage: make webtomd URL=https://example.com 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: rename webtomd scenario to web_to_md - Rename directory: scenarios/webtomd/ → scenarios/web_to_md/ - Update Makefile command: webtomd → web-to-md (consistent with dot-to-mermaid pattern) - Update package name and Python module references - Update state file paths: .data/webtomd → .data/web_to_md - Update logger names and documentation - Add make command documentation to README - Preserve git history for all renamed files 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Features
Usage