fix(web-sources): close crawler DoS vector (#399)

[DerrickF]

Summary

Closes the web-crawler DoS vector described in #399. Three related fixes so a flood of crawl requests can no longer freeze the container serving all HTTP traffic.

1. Concurrency guard (highest impact)

POST /assistants/{id}/web-sources/crawl now checks list_active_crawls before any write and returns 409 Conflict if a crawl is already running for that assistant. This closes the amplification vector — a script (or a user double-clicking on a slow connection) can no longer launch unbounded simultaneous crawls.

• The check is DynamoDB-backed, so it works across containers (the process-local _BACKGROUND_CRAWLS set never did — see the issue's Additional Notes).
• list_active_crawls already self-heals stale running rows, so a crawl whose owning process died is auto-failed and does not wrongly block a new one.
• Checked before create_document/create_crawl_job, so a rejected request leaves no orphan rows.

2. Offload CPU-bound parsing

trafilatura.extract() and BeautifulSoup() in crawler.py now run via run_in_executor (thread pool) instead of directly on the shared event loop, so parsing a large page can't stall auth/chat/other requests.

3. Offload blocking DynamoDB

Every boto3 table call in crawl_repository.py (put/get/query/update/delete — 9 call sites) now runs via run_in_executor through a small _run_ddb helper. This matters most for increment_counters, which the crawler hits per page. Chose run_in_executor over aioboto3 to avoid a new dependency and match the existing pattern already used for the S3 put.

Scope note

The issue's two Additional Notes (cross-container crawl count, lazy stale-reaping) are effectively addressed by fix #1 using the DynamoDB-backed guard. The S3 upload (_put_markdown) was already offloaded and is unchanged.

Testing

• New route test: 409 is returned when a crawl is already running, and no document/job/crawler writes happen on rejection.
• Patched the existing happy-path start-crawl tests for the new list_active_crawls call.
• 53 passed across test_routes.py, test_crawler.py, test_crawl_repository.py; ruff clean on all three source files.

Fixes #399 (will close manually after merge — Fixes only auto-closes on merge to the default branch).