Async Python crawling library with sitemap discovery, browser-based extraction, and LLM-powered structured output — one Settings object drives it all.
Onecrawler helps you build maintainable crawling and extraction workflows without turning every project into a custom scraping script. It provides a shared configuration model, async execution, sitemap discovery, browser-backed link extraction, heuristic content extraction, and optional GenAI extraction for typed outputs.
Recommended workflow:
- Use sitemaps first whenever possible.
- Fall back to browser link extraction when sitemap coverage is missing or dynamic.
- Scrape the final URL list with heuristic extraction by default.
- Use GenAI extraction when you need structured output in a Pydantic schema.
async with LinkExtractor(settings) as link_engine:
links = await link_engine.run("https://example.com")
async with Scraper(settings) as scraper_engine:
records = await scraper_engine.run(links) # [{"url": ..., "result": ...}, ...]| Capability | Details |
|---|---|
| Sitemap discovery | Resolves robots.txt, common sitemap paths, nested indexes, .xml.gz, feeds, and HTML fallback |
| Browser link extraction | Shallow and deep Playwright-backed discovery for JavaScript-rendered or sitemap-poor sites |
| URL filtering | Wildcard path filters with include_link_patterns |
| Content filtering | Composable post-extraction filters by date, keywords, file type, and cosine similarity with AND/OR/NOT logic |
| Async performance | Tunable concurrency, retries, timeouts, and crawl limits |
| Content extraction | Heuristic extraction with trafilatura for fast article-like content |
| Markdown conversion | Whole-page HTML-to-Markdown fallback for pages heuristic extraction can't handle (non-article, e-commerce, dashboards) |
| GenAI extraction | Optional model-assisted extraction for strongly typed Pydantic outputs |
| Output formats | markdown, json, xml, xmltei |
| Proxy support | Single proxy or rotating proxy pools for browser and sitemap workflows |
| Browser controls | Viewport, user agent, locale, timezone, storage state, and runtime settings |
| Need | Use | Why |
|---|---|---|
| Fast URL discovery from a public site | SiteMap |
Simplest, fastest, and least expensive way to collect URLs |
| Links from one listing page | Shallow LinkExtractor |
Reads direct same-site links from the page |
| Recursive discovery through navigation | Deep LinkExtractor |
Follows internal links until your configured limit |
| Bulk article or page text extraction | Heuristic Scraper |
Deterministic and avoids model cost |
| Non-article pages (product, docs, dashboards) | markdownify Scraper |
Never returns empty; heuristic extraction is article-biased |
| Typed fields or semantic normalization | GenAI extraction | Produces schema-shaped output for downstream systems |
pip install onecrawlerInstall Playwright browser binaries when you use browser-backed crawling or scraping:
python -m playwright install chromiumNote
GenAI extraction works out of the box — no extra install is needed. It only requires an API key from your chosen provider (OpenAI, Google) or a running Ollama instance. See GenAI Extraction for details.
For local development (requires uv):
git clone https://github.com/sayedshaun/onecrawler.git
cd onecrawler
uv sync --extra dev
uv run playwright install chromiumOneCrawler provides an optimized Docker image that includes all necessary browser dependencies. This is the recommended way to run onecrawler in production or CI/CD environments.
docker pull ghcr.io/sayedshaun/onecrawler:latestTip
You can rename it for convenience
docker tag ghcr.io/sayedshaun/onecrawler:latest onecrawlerdocker run -it --rm -v $(pwd):/app onecrawler python your_script.pyNote
The script must be located at the root of the mounted volume.
docker run --rm onecrawler python -m onecrawler crawl https://example.com --limit 20For a quick one-off crawl without writing a script:
python -m onecrawler crawl https://example.com --limit 20 --concurrency 5Each extracted page is printed to stdout as a JSON line. Run python -m onecrawler crawl --help for the full flag list (--strategy, --format, --include, --exclude). The CLI covers heuristic-strategy crawling only — proxies, GenAI extraction, and human-behavior simulation need the Python API below.
from onecrawler import Settings, Crawler
from onecrawler.utils import writter
async def main():
settings = Settings(link_extraction_limit=100, concurrency=5)
async with Crawler(settings) as engine:
results = await engine.run("https://www.prothomalo.com/")
writter.dump_json(results, "output.json")
if __name__ == "__main__":
import asyncio
asyncio.run(main())from onecrawler import HumanBehaviorSettings, LinkExtractor, Scraper, Settings
from onecrawler.utils import writter
async def main():
settings = Settings(
link_extraction_strategy="deep",
link_extraction_limit=10,
concurrency=7,
scraping_strategy="heuristic",
scraping_output_format="json",
human_behavior_settings=HumanBehaviorSettings(),
)
async with LinkExtractor(settings) as link_engine:
links = await link_engine.run("https://www.example.com/")
async with Scraper(settings) as scraper_engine:
results = await scraper_engine.run(links) # [{"url": ..., "result": ...}, ...]
writter.dump_json(results, "output.json")
if __name__ == "__main__":
import asyncio
asyncio.run(main())Tip
Always set link_extraction_limit when crawling broad sites. Without it, discovery can run indefinitely on large domains.
Use browser extraction when sitemaps are incomplete, unavailable, or unable to expose JavaScript-rendered links.
import asyncio
from onecrawler import Settings, LinkExtractor
async def main():
settings = Settings(
link_extraction_strategy="deep",
link_extraction_limit=250,
include_link_patterns=["/news/*"],
concurrency=5,
)
async with LinkExtractor(settings) as engine:
links = await engine.run("https://example.com/news")
print(f"Collected {len(links)} links")
if __name__ == "__main__":
asyncio.run(main())Tip
Use include_link_patterns to keep discovery focused on relevant paths. For example, ["/blog/*", "/docs/*"] prevents the crawler from wandering into auth pages, admin routes, or unrelated sections.
Note
Deep extraction follows internal links recursively. Use shallow strategy when you only need links visible on a single listing page — it's significantly faster.
Filter crawled results by date, keywords, file type, or semantic similarity. Filters are passed to Crawler.run() or Crawler.stream() and applied after content extraction.
import asyncio
from onecrawler import Crawler, Settings
from onecrawler.filters import AND, by_date, by_keywords
async def main():
settings = Settings(
link_extraction_limit=50,
concurrency=5,
)
# Keep only pages from 2025 that mention "python" or "async"
content_filter = AND(
by_date(start="2025-01-01", end="2025-12-31"),
by_keywords(["python", "async"]),
)
async with Crawler(settings) as engine:
results = await engine.run(
"https://example.com/blog",
filters=content_filter,
)
print(f"Matched {len(results)} pages")
if __name__ == "__main__":
asyncio.run(main())| Filter | Import | Purpose |
|---|---|---|
by_date(start, end) |
onecrawler.filters |
Keep items within a YYYY-MM-DD date range |
by_keywords(keywords) |
onecrawler.filters |
Keep items whose text contains any keyword |
by_files(types) |
onecrawler.filters |
Keep items by logical file type (pdf, image, docx, text) |
by_extension(extensions) |
onecrawler.filters |
Keep items by URL file extension (.pdf, .jpg) |
by_cosine_similarity(query, threshold) |
onecrawler.filters |
Keep items whose text is semantically similar to a query |
Use AND, OR, and NOT from onecrawler.filters to combine filters:
from onecrawler.filters import AND, OR, NOT, by_date, by_keywords, by_files
# Pages from 2025 that mention "python" but are not PDFs
f = AND(
by_date(start="2025-01-01"),
by_keywords(["python"]),
NOT(by_files(["pdf"])),
)
# Pages that mention "AI" or are from 2025
f = OR(
by_keywords(["AI"]),
by_date(start="2025-01-01", end="2025-12-31"),
)Filters work with Crawler.stream() for real-time filtered output:
async with Crawler(settings) as engine:
async for item in engine.stream(
"https://example.com/news",
filters=by_cosine_similarity("climate policy", threshold=0.3),
):
print(item["title"])Tip
Filters run after content extraction, so they work with any scraping strategy. Use by_cosine_similarity for topic-focused crawls and by_date to keep results fresh.
Note
by_date reads the date or filedate field from extracted content. Pages without a parseable date are excluded when a date filter is active.
Use GenAI extraction when you need a strongly typed response shape instead of plain content.
import asyncio
from typing import Optional
from pydantic import BaseModel
from onecrawler import Settings, LLMSettings, Scraper
class ArticleSummary(BaseModel):
title: str
author: Optional[str] = None
published_at: Optional[str] = None
summary: str
topics: list[str]
async def main():
settings = Settings(
scraping_strategy="genai",
scraping_output_format="json",
genai=LLMSettings(
provider="openai",
model_name="gpt-4o-mini",
api_key="YOUR_API_KEY",
output_schema=ArticleSummary,
),
concurrency=2,
request_timeout=30,
)
async with Scraper(settings) as scraper:
item = await scraper.run("https://example.com/articles/story")
result = item["result"]
print(result.model_dump() if hasattr(result, "model_dump") else result)
if __name__ == "__main__":
asyncio.run(main())Tip
Keep concurrency low (2–4) for GenAI extraction. Each page triggers a model call; high concurrency can exhaust rate limits quickly and inflate costs.
Warning
Never hardcode your API key in source files. Use environment variables or a secrets manager instead:
import os
api_key=os.environ["OPENAI_API_KEY"]| Provider | Requires | Models |
|---|---|---|
| OpenAI | api_key (optional with a custom base_url) |
GPT-4o, GPT-4o-mini, or any OpenAI-compatible server |
api_key |
Gemini models | |
| Ollama | base_url (no key needed) |
Any locally hosted model |
Note
Set base_url on the openai provider to target any OpenAI-compatible server (llama.cpp, vLLM, LM Studio, LocalAI, …). api_key is optional for those keyless endpoints — it is only required for the real api.openai.com default.
settings = Settings(
scraping_strategy="genai",
genai=LLMSettings(
provider="ollama",
model_name="llama3:8b",
base_url="http://localhost:11434/",
output_schema=ArticleSummary,
),
)Note
Ollama requires a running local instance. Install it from ollama.com and pull your model (ollama pull llama3:8b) before running.
Warning
For "thinking" models (qwen3, deepseek-r1, etc.), keep LLMSettings(think=False) (the default). Ollama returns an empty response for schema-constrained structured output when thinking is enabled. Set think=True only for free-form, non-schema use.
Attach one proxy or a rotating proxy pool directly to Settings.
from onecrawler import Settings, ProxySettings
settings = Settings(
proxies=[
ProxySettings(server="http://proxy-1.example:8080"),
ProxySettings(
server="http://proxy-2.example:8080",
username="user",
password="pass",
),
],
proxy_rotation_method="round_robin",
)Use proxies=[ProxySettings(...)] for a single proxy (a one-element list), or add more entries with proxy_rotation_method for a pool.
Tip
round_robin rotation distributes requests evenly across your proxy pool. For rate-limited targets, pair this with a modest concurrency value and a retry_delay to avoid triggering bans.
Important
Split URL discovery and scraping into separate pipeline steps. Collecting all URLs first gives you a checkpoint to resume from if scraping fails partway through — without re-running discovery.
Tip
Start with SiteMap before reaching for browser extraction. Sitemap-based discovery is faster, cheaper, and more complete on well-maintained sites. Fall back to LinkExtractor only when sitemaps are missing or stale.
Tip
Use heuristic scraping (scraping_strategy="heuristic") for bulk content extraction. Reserve GenAI extraction for cases where you genuinely need structured, schema-shaped output — it adds latency and cost at scale.
Tip
If heuristic extraction returns None or very little on non-article pages (product listings, dashboards, docs), switch to scraping_strategy="markdownify". It never returns empty for a rendered page, at the cost of including page chrome (nav, footers) in the output.
Caution
Respect robots.txt and a site's terms of service before crawling. Onecrawler does not enforce crawl policies automatically — you are responsible for staying within allowed access patterns.
Released under the MIT License. See LICENSE for full terms.