fix(playwright): poll for late-rendered Cloudflare challenge markup - #4031
Open
B4nan wants to merge 2 commits into
Open
fix(playwright): poll for late-rendered Cloudflare challenge markup#4031B4nan wants to merge 2 commits into
B4nan wants to merge 2 commits into
Conversation
The default challenge detection in handleCloudflareChallenge() ran a single querySelector right after navigation settled. The current Cloudflare template can finish rendering the challenge markup (or arrive via an intermediate redirect hop) only after the load event, so on retries the hook regularly concluded 'no challenge' while the request handler later landed on a fully rendered challenge page (observed in platform run rnKxeF0phWeTnlzdx). When the navigation response signals a likely challenge - a 403 status or Cloudflare's documented cf-mitigated: challenge header - the helper now polls for the markup for up to 5 seconds instead of trusting a single immediate check, failing fast when the page is a hard block. The crawling-context util passes the navigation response through to enable this. With detection reliable, v4 needs no automatic 403 removal from blockedStatusCodes (v3 spliced it out of the session pool because it never reloaded the page, so even a solved challenge kept the original 403 response; v4 reloads and propagates the fresh response instead). Keeping 403 blocked acts as a safety net: an unsolved or undetected challenge is retried with a fresh session rather than feeding the challenge page to the request handler. The camoufox-cloudflare fixture drops its blockedStatusCodes workaround accordingly, and the hook docs now spell out the 403 interplay.
B4nan
added
t-tooling
…llenge options handleCloudflareChallenge() memoized its default isChallengeCallback and isBlockedCallback onto the caller's options object, with the current page captured in the closure. The pre-wrapped handleCloudflareChallengeHook() passes one options object to every call, so after the first attempt the detection kept querying that first (by then closed) page - the evaluate rejected, the catch turned it into 'no challenge', and retries silently ran the request handler on challenge pages. Verified on the Apify platform (run m6QgmOqsSQ5wBvb1p): with the defaults kept local and bound to the page argument, every retry now detects the challenge again.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes two ways
handleCloudflareChallengeHook()lost track of challenges after the first attempt, found while stabilizing the camoufox-cloudflare E2E fixture.The first is a stale page pinned in reused options.
handleCloudflareChallenge()memoized its defaultisChallengeCallback/isBlockedCallbackonto the caller's options object, with the current page captured in the closure. The pre-wrapped hook passes one options object to every call, so after the first attempt the detection kept querying the first (by then closed) page: the evaluate rejected, thecatchturned that into "no challenge", and retries silently ran the request handler on challenge pages. Unit tests never see this because they construct fresh options per call; it took platform runs with debug instrumentation to find (the retry pages had the full challenge markup, yet detection saw nothing). The defaults are now kept local and bound to the page argument. Verified on the platform (runm6QgmOqsSQ5wBvb1p): every retry detects the challenge again and the solver runs each time.The second is single-shot detection. The current Cloudflare template can finish rendering the challenge markup after the
loadevent. When the navigation response signals a likely challenge (a 403 status, or Cloudflare's documentedcf-mitigated: challengeheader), the helper now polls for the markup for up to 5 seconds instead of checking once, and bails out early when the page is a hard block. The crawling-context util passes the response through to make that possible. Pages with other status codes keep the single immediate check and pay no extra cost.This also answers the follow-up question from 99d465d: the hook should not remove 403 from
blockedStatusCodesautomatically the way v3 did. v3 spliced 403 out of the session pool because it never reloaded the page, so even a solved challenge kept the original 403 response. v4 reloads and propagates the fresh response, so a solved challenge passes the blocked-status check on its own. Keeping 403 blocked is a safety net: an unsolved or undetected challenge gets retried with a fresh session instead of reaching the request handler. The E2E fixture drops itsblockedStatusCodesworkaround, and the hook docs now spell out the 403 interplay.The fixture still does not pass, because the click-solver currently cannot clear the challenge at all. Master's nightly fails the same fixture the same way (run from 2026-08-13 06:14 UTC), so that part is environmental and independent of this change.