Skip to content

scenario: add web-to-md tool for converting web pages to markdown - #61

Merged
Brian Krabach (bkrabach) merged 2 commits into
microsoft:mainfrom
robotdad:webtomd
Oct 7, 2025
Merged

Brian Krabach (bkrabach) merged 2 commits into
microsoft:mainfrom
robotdad:webtomd

Conversation

@robotdad

Copy link
Copy Markdown
Member

Summary

  • Add web-to-md tool: converts web pages to markdown with image handling and validation
  • Update Codespaces optimization documentation

Features

  • Clean HTML to markdown conversion
  • Pages saved to AMPLIFIER_CONTENT_DIR
  • Automatic image downloading and organization
  • Content validation and quality checks
  • Page indexing and cross-linking
  • Persistent state for resumable sessions
  • Modular "bricks and studs" architecture

Usage

  make web-to-md URL=https://example.com [URL2=...] [OUTPUT=path]

Implements a modular web-to-markdown converter that:
- Converts web pages to clean markdown with YAML frontmatter
- Organizes content by domain (saves to content_dirs/sites/)
- Downloads and manages images in domain-specific directories
- Detects and rejects paywall/auth-walled content (Medium, NYT, etc.)
- Generates searchable index of all saved pages
- Supports resume for interrupted sessions
- Integrates with amplifier.config.paths for workspace consistency

Architecture follows "bricks and studs" pattern with 8 focused modules:
fetcher, converter, validator, image_handler, enhancer, organizer,
indexer, and state management.

Also fixes stub violation in knowledge_assist output_generator.

Usage: make webtomd URL=https://example.com

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Rename directory: scenarios/webtomd/ → scenarios/web_to_md/
- Update Makefile command: webtomd → web-to-md (consistent with dot-to-mermaid pattern)
- Update package name and Python module references
- Update state file paths: .data/webtomd → .data/web_to_md
- Update logger names and documentation
- Add make command documentation to README
- Preserve git history for all renamed files

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@bkrabach
Brian Krabach (bkrabach) merged commit 66fe5cf into microsoft:main Oct 7, 2025
@robotdad
Marc Goodner (robotdad) deleted the webtomd branch October 8, 2025 21:43
Joi Ito (Joi) pushed a commit to Joi/amplifier that referenced this pull request Oct 31, 2025
…crosoft#61)

* feat: add webtomd tool for converting web pages to markdown

Implements a modular web-to-markdown converter that:
- Converts web pages to clean markdown with YAML frontmatter
- Organizes content by domain (saves to content_dirs/sites/)
- Downloads and manages images in domain-specific directories
- Detects and rejects paywall/auth-walled content (Medium, NYT, etc.)
- Generates searchable index of all saved pages
- Supports resume for interrupted sessions
- Integrates with amplifier.config.paths for workspace consistency

Architecture follows "bricks and studs" pattern with 8 focused modules:
fetcher, converter, validator, image_handler, enhancer, organizer,
indexer, and state management.

Also fixes stub violation in knowledge_assist output_generator.

Usage: make webtomd URL=https://example.com

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: rename webtomd scenario to web_to_md

- Rename directory: scenarios/webtomd/ → scenarios/web_to_md/
- Update Makefile command: webtomd → web-to-md (consistent with dot-to-mermaid pattern)
- Update package name and Python module references
- Update state file paths: .data/webtomd → .data/web_to_md
- Update logger names and documentation
- Add make command documentation to README
- Preserve git history for all renamed files

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants