Common Crawl CLI

Command-line tool for querying Common Crawl's URL index. Two backends:

• Athena — SQL against the columnar ccindex.ccindex Parquet table. Right for domain-scale audits (thousands to millions of URLs). Billed per TB scanned; guardrails baked in.
• CDX API — the free index.commoncrawl.org HTTP service. Right for single-URL and small-URL-set lookups. No AWS, no bill.

Built to support DFRLab's LLM-grooming research (see [[llm-grooming]], [[pravda-network]], [[training-data-poisoning]]). The default query catalog answers "how many URLs from these domains is Common Crawl carrying, per monthly crawl, and where are the raw WARC records?" — the shape of the April 2026 Pravda-in-the-Pipeline audit — plus "which crawls captured this exact URL?" via the CDX backend.

Every Athena command requires --crawl and --subset partition filters (or --all-crawls) so scan volume — and cost — stays predictable. A domain-filtered audit across four monthly crawls typically scans under 100 MB and costs under one cent.

---

Prerequisites

1. AWS account with:
   • Athena access in us-east-1 (Common Crawl's S3 bucket lives there; cross-region reads are billed).
   • An S3 bucket you own for Athena query results.
   • IAM permissions: athena:*, glue:*Database, glue:*Table, s3:GetObject on commoncrawl/*, and s3:PutObject/s3:GetObject on your result bucket.
2. Python 3.11+ — via the workspace narrative-blueprint conda env or your own venv.
3. AWS credentials discoverable by boto3 (env vars, ~/.aws/credentials, IAM role — anything the default chain finds).

Install

conda activate narrative-blueprint      # workspace default env
cd projects/common-crawl
pip install -r requirements.txt

Configure

cp config.example.toml ~/.common-crawl-cli.toml
$EDITOR ~/.common-crawl-cli.toml         # set athena.output_location at minimum

Config precedence: CLI flags > CC_* environment variables > TOML file > built-in defaults. See config.example.toml for the full annotated schema.

First-time setup

python -m common_crawl_cli init

This creates the ccindex database, defines the external table pointing at s3://commoncrawl/cc-index/table/cc-main/warc/, and runs MSCK REPAIR TABLE to register every existing monthly partition (a few minutes, one-time). Rerun init occasionally to pick up new crawls, or use list-crawls to check.

---

Cost model

Athena charges ~$5 per TB scanned (verify against current AWS pricing for your region). One monthly warc partition is ~300 GB uncompressed metadata; with Parquet columnar storage + predicate pushdown, a partition-filtered domain query typically scans 10–25 MB per partition. Athena also enforces a 10 MB minimum billable scan per query.

Query shape (typical)	Est. scan	Est. cost
domain-count, 1 domain, 1 crawl	~10 MB (min)	~$0.00005
domain-count, 5 domains, 4 crawls	~100 MB	~$0.0005
domain-timeseries, 1 domain, 24 crawls (2 years)	~600 MB	~$0.003
warc-pointers, 1 domain, 4 crawls, no LIMIT	~100 MB	~$0.0005
language-distribution, 1 domain, 12 crawls	~300 MB	~$0.0015
domain-count, 1 domain, --all-crawls (~126)	~3.1 GB	~$0.016
domain-timeseries, 1 domain, --all-crawls	~3.1 GB	~$0.016
lookup-url (CDX backend, any # of crawls)	0	$0.00
lookup-url --backend athena, 1 URL, 1 crawl	~10 MB (min)	~$0.00005
lookup-url --backend athena, 1 URL, all crawls	~1.3 GB	~$0.007
lookup-url --backend athena, 10 URLs, all crawls	~1.3 GB	~$0.007
find-mirrors, 1 URL, all crawls	~6.3 GB	~$0.031
find-mirrors, 1 digest, all crawls	~3.2 GB	~$0.016

Every invocation:

1. Prints an estimated cost before submitting.
2. Aborts if the estimate exceeds --max-cost (default $2, configurable in TOML).
3. Prompts for interactive confirmation above $0.10 unless --yes is passed.
4. Reports actual scan bytes + cost from Athena's execution statistics after completion.

Use --dry-run to print the SQL and estimate without submitting.

---

Commands

Global flags (all commands): --config PATH, --max-cost USD, --yes, --dry-run.

init

Create database + external table + repair partitions. Run once, or after Common Crawl publishes a new crawl you want registered.

list-crawls [--limit N] [--source athena|cdx]

List available crawls, most recent first. --source athena (default) queries ccindex$partitions (reflects what your init registered — costs nothing, but capped at what Athena knows about). --source cdx hits the free public collinfo.json (authoritative list of every crawl ever published — 126 as of August 2026 — no AWS needed).

domain-count --domain D [--domain D ...] --crawl C [--crawl C ...]

URLs per (crawl, registered_domain). The baseline audit query — cheap, and answers "is this domain being harvested, and at what volume?"

domain-timeseries --domain D --crawl C [--crawl C ...]

URL count + distinct subdomain count per crawl for one domain. Growth curve — feeds "37 → 40,000 in 12 months" style time-series for grooming operations.

sample-urls --domain D --crawl C [--crawl C ...] [--limit N] [--language LANG]

URLs for the domain with mime type / language / fetch time. Omit --limit to export every matching URL — Athena's scan cost is identical whether or not a LIMIT is present (partitions are still scanned in full, LIMIT is applied post-scan), so the only trade-off is result-set size on disk. Set --limit N for a lightweight preview. Optional --language filter matches against content_languages (substring).

language-distribution --domain D --crawl C [--crawl C ...]

Distribution of content_languages for the domain per crawl. Diagnostic for multilingual operations — Pravda-family sites publish in 80+ languages.

warc-pointers --domain D --crawl C [--crawl C ...] [--limit N]

Export (url, warc_filename, warc_record_offset, warc_record_length) for every URL of the domain. Downstream, feed these tuples to boto3.S3.get_object(Bucket='commoncrawl', Key=warc_filename, Range=f'bytes={offset}-{offset+length-1}') to fetch just that record's raw bytes — no whole-WARC download.

lookup-url --url URL [--crawl C ...] [--all-crawls] [--since YYYY[-MM[-DD]]] [--until ...] [--backend cdx|athena] [--exact]

Look up which crawls captured a specific URL. --backend cdx (default) is free — hits the public CDX server for each crawl and returns WARC pointers. --backend athena runs the equivalent SQL against ccindex.ccindex (charged; use for consistency when you're already running Athena queries). Use --all-crawls to sweep every crawl without guessing which one holds the URL — CC re-fetches a rotating subset each month, so a given URL usually appears in some crawls and not others.

URL matching. By default lookup-url matches loosely: for Athena it rewrites the query to url_host_registered_domain = HOST AND url_path LIKE PATH_PREFIX% (tolerates scheme, www. prefix, trailing slash, querystring — you'll get one row per crawl-captured variant of the article); for CDX it passes matchType=prefix and lets the server canonicalize. Add --exact if you have the canonical URL from a WARC pointer and want a strict literal match.

Note that CC's stored URL for one page is often https://www.host.tld/path/ even if you typed http://host.tld/path — normalized matching is almost always what you want unless you're joining against a specific known-good record.

Sweeping all crawls: on the audit commands (domain-count, domain-timeseries, sample-urls, language-distribution, warc-pointers) pass --all-crawls instead of an explicit crawl list. The CLI resolves the list from ccindex$partitions at run time and rolls the scan estimate into the cost prompt. For domain-filtered queries this is very cheap (~$0.02 for a full ~126-crawl sweep of one domain).

find-mirrors --url URL | --digest DIGEST [--url-file / --digest-file FILE] [--crawl ... | --all-crawls] [--since ...] [--exact]

Find every URL republishing the same bytes as a source article, via CC's content_digest (SHA-1 of the response payload). Two URLs share it iff their bytes are byte-for-byte identical — the "verbatim republication" case that dominates Pravda-style laundering (same article copied across 150+ mirror sites).

Flow: (1) resolve source URL(s) → distinct content_digest(s); (2) scan crawls for every URL matching those digests. Two Athena queries; skip step 1 by passing --digest DIGEST directly (from a prior lookup-url result). Batchable via --url-file / --digest-file.

Scope limit — exact content only. content_digest catches verbatim republication. It does not catch paragraph-level similarity, translated variants, or lightly-edited copies. For that you need the actual WARC payload:

1. warc-pointers or lookup-url → export (warc_filename, warc_record_offset, warc_record_length).
2. Fetch each record with boto3.S3.get_object(Bucket='commoncrawl', Key=warc_filename, Range=f'bytes={offset}-{offset+length-1}').
3. Parse the WARC record (HTTP response + payload) — warcio handles this.
4. Extract text (BeautifulSoup / trafilatura), then do the similarity work locally (shingling / minhash / embeddings).

That fetch-and-search pipeline is a distinct tool — happy to build it as fetch-warc + text-search if useful; not part of the ccindex-only workflow. find-mirrors is the free lunch: it catches the highest-frequency laundering pattern without leaving Athena.

Batch URLs — --url-file

lookup-url and find-mirrors both accept --url-file FILE (one URL per line, blanks and lines starting with # skipped). Combines with -u URL flags. Athena batches all URLs into one scan — same partition cost as a single URL. CDX loops sequentially (one HTTP GET per URL × crawl).

Example:

# urls.txt
# Pravda-network article variants
france.news-pravda.com/politique/2026-02-03-tisza-toll
slovakia.news-pravda.com/svet/2026-02-04-slovak-tisza

python -m common_crawl_cli find-mirrors \
  --url-file urls.txt --all-crawls --since 2026-01 \
  --output csv --output-file mirrors.csv

--since / --until — trim by publish/crawl date

Every command that takes a crawl list (both explicit --crawl and --all-crawls) accepts --since and --until bounds. Format: YYYY, YYYY-MM, or YYYY-MM-DD. The CLI parses each crawl id (CC-MAIN-YYYY-WW → ISO week Mon–Sun; legacy formats too) and drops partitions whose fetch window is definitively outside the range before submitting anything.

For a URL you know was first published on 2023-01-25, --since 2023-01 eliminates every crawl from 2008 through end of 2022 as a guaranteed miss — cutting the sweep from ~126 partitions to ~40 without touching data. Same pattern for domain-level queries: if a Pravda-family site launched in 2022, --since 2022 cuts the timeseries sweep to only-relevant crawls.

The filter fails-closed on unparseable crawl ids when a bound is set (rare — only affects the two pre-2013 range archives), so if --since matters for correctness you get a "N dropped" line telling you what was excluded.

---

Example — Pravda multi-domain audit

bash examples/pravda_audit.sh

Counts URLs for five Pravda-network domains across the four most recent monthly crawls, writes CSV to pravda_audit.csv. Expected total cost: < $0.01.

Equivalent one-liner:

python -m common_crawl_cli --yes domain-count \
  --domain news-pravda.com \
  --domain france.news-pravda.com \
  --domain slovakia.news-pravda.com \
  --crawl CC-MAIN-2026-30 \
  --crawl CC-MAIN-2026-26 \
  --crawl CC-MAIN-2026-22 \
  --output csv --output-file pravda_audit.csv

Preview the SQL without spending anything:

python -m common_crawl_cli --dry-run domain-count \
  --domain pravda-tv.ru --crawl CC-MAIN-2026-30

Example — single URL, only relevant crawls (free)

Find every Common Crawl capture of a specific article, pruning crawls that predate the article's publish date:

python -m common_crawl_cli lookup-url \
  --url "http://rt.com/russia/570714-russia-ukraine-pentagon-biolabs/" \
  --all-crawls --since 2023-01 \
  --output json --output-file rt_captures.json

The RT piece was published in January 2023, so --since 2023-01 drops every crawl from CC-MAIN-2008-2009 through CC-MAIN-2022-49 as a guaranteed miss — ~86 partitions eliminated before any HTTP request. Sweep drops from ~1 min to ~20 s wall time. Total cost $0.00 either way (CDX backend).

Omit --since and you sweep all crawls — same result, more waiting.

Example — one domain, all crawls (~$0.02)

Full growth curve for a Pravda-family domain across every registered crawl:

python -m common_crawl_cli domain-timeseries \
  --domain pravda-tv.ru --all-crawls \
  --output csv --output-file pravda-tv-timeseries.csv

When to use which backend

Question	Backend	Command
Which crawls captured this specific URL?	CDX (free)	lookup-url --url URL --all-crawls
How many URLs of this domain per crawl?	Athena	domain-count, domain-timeseries
Give me a sample of URLs from this domain	Athena	sample-urls
Give me WARC pointers for every URL of this domain	Athena	warc-pointers
Language breakdown of this domain's corpus	Athena	language-distribution
A handful of URL lookups, no AWS set up	CDX (free)	lookup-url per URL

CDX server caveat. The public CDX server is a shared community resource and returns transient 5xx responses under load. The CLI retries with exponential backoff (1s → 2s → 4s) and skips crawls that stay broken after retries — the sweep continues and logs [warn] CC-MAIN-YYYY-WW: HTTP Error 504 for whatever crawls failed. Re-run the failed crawls individually later, or use the Athena backend (--backend athena) if reliability matters more than cost.

---

Output formats

--output {table,csv,json} selects the format; default is table (Markdown-style, stdout). --output-file PATH writes to a file instead. JSON is a list of {column: value} objects; CSV is header + rows.

Column shape (URL-detail commands)

sample-urls, warc-pointers, lookup-url, and find-mirrors all return the same 16-column canonical projection so downstream code doesn't care which command produced a given CSV:

Column	Notes
crawl	Partition — CC-MAIN-YYYY-WW etc.
url	The URL as CC actually fetched it
url_host_name	Full hostname (e.g. www.news-pravda.com)
url_host_registered_domain	eTLD+1 (e.g. news-pravda.com)
content_digest	SHA-1 of the response payload — feed to find-mirrors --digest
content_mime_type	HTTP Content-Type header as declared
content_mime_detected	Server-side content sniff (often more accurate)
content_charset	Encoding declared or detected
content_languages	Comma-separated ISO codes CC detected
content_truncated	Non-empty if the response was truncated
fetch_time	UTC timestamp of the capture
fetch_status	HTTP status code (200, 301, 404, …)
fetch_redirect	Target URL for redirects, empty otherwise
warc_filename	Path in s3://commoncrawl/
warc_record_offset	Byte offset inside the WARC file
warc_record_length	Byte length of the record

The three "counter" commands — domain-count, domain-timeseries, language-distribution — return aggregate shapes appropriate to their question, not this projection.

---

Troubleshooting

• athena.output_location is required — set it in ~/.common-crawl-cli.toml or export CC_ATHENA_OUTPUT_LOCATION.
• AccessDeniedException — your IAM role needs s3:GetObject on commoncrawl/* (public bucket, but Athena still checks) and read/write on your result bucket.
• Partition not found — a new crawl was published but hasn't been registered. Rerun init (which calls MSCK REPAIR TABLE) or use list-crawls to confirm what's available.
• Query stuck in QUEUED — Athena workgroup capacity limit. Check workgroup metrics in the AWS console.
• HIVE_S3_THROTTLING / SlowDown / HIVE_CANNOT_OPEN_SPLIT — infrastructure event, not a query bug. Common Crawl's S3 bucket is public and shared; S3 returns SlowDown when concurrent request volume spikes against a hot prefix. The CLI auto-retries these classes of failure with exponential backoff (5s → 10s → 20s → 40s, up to 4 attempts). If it exhausts retries, wait a minute and rerun — the throttle window is usually short. Not billed as data scanned for FAILED queries, though the workgroup still logs the execution.
• Estimated cost blocked — pass --max-cost 5.00 for the invocation, or narrow the query. Broad subset=warc queries without a domain filter can scan hundreds of GB.
• lookup-url returned zero rows — CC likely stored the URL under a different scheme, www. prefix, or trailing slash than you typed. lookup-url defaults to normalized matching, which tolerates that; if you passed --exact, drop it. If you still see zero rows, the URL truly isn't captured — Common Crawl decides per-crawl which URLs to fetch based on link-graph reachability and seed lists; publish date doesn't guarantee capture.
• --output-file produced an empty / header-only file — the query succeeded but returned zero rows. The empty file is intentional: it confirms the query ran and returned nothing, so a downstream pipeline can detect the negative case rather than crash on a missing file.

---

Design notes for future maintainers

• Query templates live in common_crawl_cli/queries.py. Every template hard-requires crawl IN (...) and subset = ... filters. Do not add a template that omits these — Athena will happily scan every crawl ever if you let it.
• Cost estimation (cost.py) is deliberately conservative. Post-run, Statistics.DataScannedInBytes from Athena is the ground truth; the estimator only exists to catch obvious mistakes before submission.
• The ccindex schema is versioned in the commoncrawl/cc-index-table repo. If Common Crawl adds columns, update CREATE_TABLE in queries.py and rerun init.
• sample-urls uses LIMIT without ORDER BY RAND() — Athena's LIMIT on partitioned Parquet is effectively a low-cost prefix sample, not a uniform random sample. For statistically meaningful sampling, pull warc-pointers and sample downstream.
• The cdx.py client uses urllib (stdlib) with exponential-backoff retries and per-crawl error isolation. If we need higher throughput on all-crawls sweeps, a concurrent.futures.ThreadPoolExecutor (bounded to 4–8 workers to stay polite) would help — deliberately not added yet.
• boto3 and tabulate are lazy-imported so CDX-only workflows (lookup-url --backend cdx, list-crawls --source cdx) run without AWS deps installed. Only click is required at module load; pip install click alone is enough to use every CDX-backed command.