Serply vs. Built-In Agent Web Search: One Research Question, Measured

• Where they tied: the academic paper
• Where it broke: the primary evidence
• The failure mode is worse than “no results”
• The scoreboard
• The actual lesson, which isn’t about Serply

Most “our API vs. the built-in tool” posts are rigged. The author picks a question their product happens to answer, runs a lazy query through the competitor, and declares victory.

So here’s the setup for this one. We had a real research question, asked in the course of real work, before anyone thought of benchmarking anything:

Why don’t people crowdsource browser agents to share websites? Everyone is automating the same thing over and over.

That question was answered first, using Serply. Only afterwards did we run it again through Claude Code’s built-in WebSearch and WebFetch to see what a purely built-in stack would have produced. Same question, same researcher, no cherry-picking of the queries.

The result was not a clean sweep, and the part where the two tied is the more useful half of the story.

Where they tied: the academic paper

The single best source for this question is ReUseIt: Synthesizing Reusable AI Agent Workflows for Web Automation (ACM IUI ‘26). It’s the paper that actually quantifies why shared agent workflows are hard, and both stacks retrieved it and read it correctly.

WebFetch on the arXiv abstract page returned:

Success Rate Improvement: The approach increased success rates “from 24.2% to 70.1% across fifteen tasks.” User Study: “nine users” participated in the evaluation.

Serply’s POST /v1/request with response_type: markdown returned the same abstract, and the same numbers: 24.2% → 70.1%, fifteen tasks, nine users.

No difference worth having an opinion about. For a public, static, unauthenticated page that wants to be read — an arXiv abstract, a docs page, a blog post — the built-in fetcher is fine, and you should use it. It’s included, there’s no key, and there’s no credit to spend.

The one distinction is shape. WebFetch runs a small fast model over the page and hands back an answer to your prompt. That’s efficient when you want three numbers. It’s lossy when you want the evidence itself, because you get the summary rather than the source, and you cannot audit what the summarizer dropped. Serply hands back the markdown, and you do the reading. Which one you want depends on whether you’re collecting facts or collecting quotes.

So: tie on the paper. Now the part that decided the research.

Where it broke: the primary evidence

The honest answer to “why doesn’t anyone crowdsource this” does not live in a paper. It lives in practitioners complaining, in public, in forum threads. That’s the primary evidence for this class of question — anything about why people don’t do something is a question about motivation, and motivation shows up in discussion, not in documentation.

Serply’s /v1/reddit/comments/{id} on the r/automation thread What are you actually using browser automation for? And what breaks most? returned HTTP 200 in 0.32s, 131 KB, 18 top-level comments with scores and verbatim bodies. That produced the three quotes that carried the entire argument:

“We automate data extraction from invoices and lead forms. Biggest pain: website structure changes break everything monthly. Maintenance time kills ROI fast.”

“Many people end up spending more time patching selectors than improving the actual process.”

“Biggest pain — maintenance across many third-party sites. It scales the breakage, not the value.”

And the single line that answered the incentive question outright, from someone selling access to automations they maintain:

“Authentication and layout inconsistency has been the biggest headache… but keeping many integrations healthy is definitely my competitive advantage.”

That’s the thesis of the whole piece — the maintenance burden is the moat, which is why nobody shares it — and it came from a comment with a score of 2, forty-odd comments deep in a thread. It is not in any article. It is not summarized anywhere. Somebody had to read the thread.

Here is what the built-in stack returned for the same target.

WebFetch on the thread URL:

Error: Claude Code is unable to fetch from www.reddit.com

WebSearch, with the query explicitly naming the source — browser automation what breaks selectors maintenance reddit r/automation — returned ten results and zero from reddit.com. What came back instead: a Yahoo Finance press release about BrowserStack, four dev.to posts, a Justia patent filing, and a Microsoft Power Automate release-plan page.

WebSearch with the domain pinned:

API Error: 400 The following domains are not accessible to our user agent: ['reddit.com']

That last one is the finding. It’s not a phrasing problem, not a ranking problem, and not something a better query fixes. It’s a crawler-level block, documented by Anthropic’s own support page on how site owners block the crawler. Reddit blocks it. The door is closed and no amount of prompting opens it.

The failure mode is worse than “no results”

If the built-in stack had returned nothing, that would be fine. Empty results are honest — you notice, and you go find another route.

That isn’t what happened. The first WebSearch — why don't people crowdsource browser agent automations share per-site workflows — returned eight confident-looking results:

• “11 Best AI Browser Agents in 2026”
• “10 Best Agentic Browsers for AI Automation in 2026”
• “AI Browser Agents: How to Automate Anything on the Web Without Writing Code”
• vendor blogs from Firecrawl, Bright Data, Cloudflare, and an AWS Marketplace listing

Every one of those is a commercial page optimized to rank for “browser agent.” Not one of them is going to tell you that the reason nobody shares browser automations is that the maintenance burden is the business model — because they’re all selling browser automation.

To its credit, the tool said so itself:

“the results don’t directly address why people don’t crowdsource or share per-site workflows”

That’s the right call and a good sign. But an agent under instruction to answer the question, rather than to assess its sources, will summarize those eight pages into a plausible paragraph about “compatibility issues and fragmentation” and hand it over. It will sound fine. It will be sourced entirely from vendors with a stake in the answer, and it will miss the actual mechanism completely.

The second query is the sharper illustration: it returned a real, specific, well-supported finding — teams spend an average of 15 minutes fixing each broken locator, and some organizations lose up to half their QA time to test maintenance. That’s a good statistic. It is also about Selenium test suites in CI, not about people sharing automations. Answering the question from those results means confidently answering a different question. That is more expensive than an empty result set, because nothing in the output signals the substitution.

The scoreboard

	Built-in WebSearch / WebFetch	Serply
arXiv abstract, exact figures	✅ 24.2% → 70.1%, 15 tasks, 9 users	✅ identical
General web search	✅ works, ranked toward vendor SEO	✅ works, titles + descriptions
Reddit thread, verbatim comments	❌ unable to fetch	✅ 200, 0.32s, 18 comments + scores
Reddit via search	❌ 400, domain not accessible	✅ threads surfaced in results
Geo-targeting	❌ US-only	✅ X-Proxy-Location
Setup	✅ none — included	⚠️ API key
Cost	✅ included	⚠️ 1 credit per uncached call, cached free

The built-in tools win on setup and cost, and that is not a small thing. For a question answerable from public documentation, they are the correct tool and reaching for an API is overhead you don’t need.

They lose on one axis only — access — and that axis happened to contain the entire answer.

The actual lesson, which isn’t about Serply

The failure here had nothing to do with search quality. Both stacks ranked plausibly. The difference was that one of them could open the door where the evidence was kept.

That generalizes past this one question, because the forums are where the un-marketed truth lives. Anything you want to know about why a tool is abandoned, which vendor quietly broke a promise, what a workflow actually costs in practice, or why a technique everyone recommends doesn’t work — that information exists only in discussion, written by people with no incentive to rank for it. It is systematically absent from the indexed, SEO-optimized web, which is exactly the half a blocked crawler leaves you with.

So the question to ask about any research stack isn’t “how good are the results.” It’s what can it not reach, and would I notice.

The second clause is the dangerous one. A stack that fails loudly is safe. This one failed loudly on WebFetch — a clean error, impossible to miss — and quietly on WebSearch, where the missing source was replaced with eight vendor pages and a statistic about a different topic. An agent running unattended does not notice that substitution, and neither does the person reading its summary.

Our own conclusion — that crowdsourcing works fine for yt-dlp extractors and uBlock filter lists but fails for browser agents, because the artifact is stateful, auth-bound, legally exposed, and monetized by the people maintaining it — is only supportable because we read the threads. Every load-bearing quote came from a source the built-in stack could not open.

If your agent does research where the answer might be in a forum rather than a docs page, check what it’s locked out of before you trust the summary. If it turns out that matters for your workload, 2,500 free credits is enough to run the same question both ways and diff the sources yourself — which is the only version of this comparison worth believing.