[pull] main from firecrawl:main - #366
Merged
Merged
Conversation
…metry (#4289) * docs(sdks): disambiguate the research paper index from the `research` search category Firecrawl serves a research paper index (~43M abstracts, ~90% biomedical — PubMed/bioRxiv/medRxiv — plus arXiv) at /v2/search/research. Separately, `search(categories=["research"])` is only a ~14-domain website filter over ordinary web search. The names collide and agents routinely pick the filter. Neither SDK contained any biomedical vocabulary. This is a documentation/metadata change only. No runtime behavior changes. Python: - v2/methods/research.py: add module + per-function docstrings naming the corpus, warning about the collision, and documenting that these responses are returned raw (camelCase keys, not snake_case-normalized like the rest of the SDK). - v2/methods/aio/research.py: mirror the same docstrings. - v2/client.py, v2/client_async.py: add wrapper docstrings — an agent calling `firecrawl.search_papers(...)` sees the wrapper's docstring, not the module function's. - v2/types.py: `Category` docstring restated the collision as if correct ("Filter results to research papers and academic sites"). Rewritten to say it is a website/domain filter and to point at `search_papers()`. JavaScript: - v2/types.ts: `CategoryOption` had no doc comment; add one, plus a note on the `categories` field of `SearchRequest`. - v2/methods/research.ts, v2/client.ts: both described the surface as "(arXiv papers + GitHub history/readmes)". Rewritten to name the real corpus. - v2/types.ts: `IdMap` claimed "Currently only `arxiv` is populated" while `PaperResult.primaryId` documented `pmid:`/`doi:`. Removed the stale claim rather than inventing an exhaustive namespace list (the API is proxied, so the namespace set is not verifiable in-repo). Both SDKs: - READMEs documented neither paper search nor plain search. Add a "Search" section and a "Research / paper search" section with biomedical examples. - Package metadata (pyproject.toml, setup.py, package.json) had no research vocabulary. Add descriptions, keywords, and Science/Research + Bio-Informatics + Medical Science trove classifiers. - v1 deprecation notices said "/v1/deep-research is deprecated. Use /v2/search instead.", routing agents straight into the website filter. Now point at /v2/search for web research AND search_papers()/research.searchPapers() for literature. (12 occurrences in Python, 3 in JS.) Tests: - Python had zero tests for any research method. Add 39 unit tests mirroring the JS suite (mock the HttpClient, pin method names and query-string construction), sync + async. Deliberately not changed (breaking, needs a product decision): - The Python research return type stays `Dict` rather than typed models; the camelCase-vs-snake_case inconsistency is documented instead. - `related_papers` (Python) vs `similarPapers` (JS) not renamed; cross-referenced in both docstrings instead. * fix(python-sdk): send `origin` from inspect_paper like the other research methods `search_papers`, `read_paper`, `related_papers` and `search_github` all append `origin=python-sdk@<version>` via `_query()`. `inspect_paper` called `_get()` bare, so paper-detail requests arrived with no SDK attribution — the one research call the API could not attribute to the Python SDK. The API accepts it: `paperSchema` in apps/api/src/controllers/v2/research-proxy.ts spreads `commonQuery` (`origin`, `integration`) into its strictObject, and the controller reads it through `requestOrigin(params, req)`. Mirrored in `v2/methods/aio/research.py`. The previous behaviour was pinned by `test_sends_no_query_params`; that test now pins the corrected behaviour (and the id-encoding tests compare the path component rather than the whole URL). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(python-sdk): expose the five research methods on the unified clients The README documents `firecrawl.search_papers(...)` on the client you get from `from firecrawl import Firecrawl`. That class (firecrawl/client.py:210) did not have the method: >>> Firecrawl(api_key="fc-...").search_papers("q") AttributeError: 'Firecrawl' object has no attribute 'search_papers' Unlike the JS SDK — where `class Firecrawl extends FirecrawlClient` inherits everything the v2 client gains — the Python unified client re-exports the v2 surface method by method in `__init__`. `search_papers`, `inspect_paper`, `read_paper`, `related_papers` and `search_github` were added to `v2/client.py` / `v2/client_async.py` only, so they were reachable at `firecrawl.v2.search_papers(...)` (via `V2Proxy.__getattr__`) but not at the top level the docs and the docstrings point at. Add all five to `Firecrawl` and `AsyncFirecrawl` as thin delegations to the v2 implementations, defined as real methods (not `__init__` attribute assignments) so the docstrings — the disambiguation surface an agent reads via `help()` / hover — are visible on the class itself. Docstrings are copied from the v2 wrappers. Tests: pin that both unified clients expose all five, that each one reaches the right `/v2/search/research/...` path with the right query params (v2 HTTP layer mocked), that responses come back verbatim, and that the docstrings still carry the `search(categories=["research"])` disambiguation. 445 -> 456 unit tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(sdks): bump firecrawl-py to 4.35.0 and @mendable/firecrawl-js to 4.32.1 Python 4.34.0 -> 4.35.0 (minor): this branch adds five new public methods to `Firecrawl` / `AsyncFirecrawl`, changes the wire request for `inspect_paper` (now sends `origin`), and rewrites the package metadata (description, keywords, Science/Research + Bio-Informatics + Medical Science classifiers). New public API, backwards compatible. Note on "keeping pyproject.toml and setup.py in sync": neither hardcodes a version. `setup.py` reads `__version__` out of `firecrawl/__init__.py`, and `pyproject.toml` declares `dynamic = ["version"]` so setuptools takes it from setup.py. `firecrawl/__init__.py` is the single source of truth; verified via PEP 517 `prepare_metadata_for_build_wheel` -> `firecrawl_py-4.35.0.dist-info`. It is also what `v2/utils/get_version.py` stamps into the `origin` param. JS 4.32.0 -> 4.32.1 (patch): the JS side of this branch is documentation and metadata only — doc comments, README sections, package description/keywords. The public surface is unchanged; `Firecrawl extends FirecrawlClient` already exposed `.research`, so no new methods were needed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(python-sdk): single source of truth for the research docstrings The five research methods (search_papers, inspect_paper, read_paper, related_papers, search_github) are surfaced at six layers: the sync and async implementation functions, and the delegating methods on FirecrawlClient, AsyncFirecrawlClient, Firecrawl and AsyncFirecrawl. Each layer carried its own literal copy of the prose, so correcting a factual claim — corpus size and composition, the search(categories=["research"]) disambiguation, wire names, Args/Returns — meant a six-way edit that would drift (Cubic P3). Move the text to firecrawl/v2/methods/research_docs.py and apply it with a doc() decorator that sets __doc__ without wrapping the function. Every layer still exposes the complete docstring: help() and pydoc on Firecrawl.search_papers render exactly what they rendered before, byte for byte after cleandoc, at all six layers. No public API, signature, method name or runtime behaviour changes. The two async deltas (the client Arg wording, and the awaited example) are substituted into the shared template rather than forking the prose, so there is one copy of each claim, not six. 568 lines of duplicated prose become 201. Adds two guard tests that fail if a layer forks its own copy again or ships an unresolved template placeholder. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ADHZoLJe28rAnirV1VU2Bg --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )