Why Your AI Agent Needs a Scraper, Not Just a Search API

Search results give an agent titles and snippets. Most tasks need the actual page. Here's why request-based scraping belongs next to search in your stack.

Profile picture of Serply
Serply
Diagram showing an AI agent pipeline moving from search results to full page content

Ask an AI agent to research a competitor’s pricing page, summarize a news article, or pull the exact return policy off an e-commerce site, and you’ll quickly run into the same wall: a search API tells you where the answer probably lives. It doesn’t tell you what the answer actually is.

That gap is easy to miss when you’re prototyping. A search call returns a handful of results with titles and two-line descriptions, the agent picks the most relevant-looking one, and the demo looks great. Then someone asks a real question — “what’s the refund window on this specific product” or “what did this article actually say about the merger” — and the agent starts hallucinating, because all it ever had was a snippet. It never read the page.

Search finds the URL. It doesn’t have the content.

This is a structural limitation, not a bug. Search APIs, including Serply’s own Google Search endpoint, are built to answer “what’s out there,” not “what does it say.” A typical result looks like this:

{
  "results": [
    {
      "title": "Show HN: Example Project",
      "link": "https://news.ycombinator.com/item?id=12345678",
      "description": "152 points by user 3 hours ago | 76 comments..."
    }
  ],
  "total": 1840000000,
  "answer": null
}

That’s enough to decide which page is worth reading. It is nowhere near enough to answer a question that depends on the page’s actual content — the full article text, the comment thread, the pricing table, the paragraph buried halfway down a support article. If your agent’s job is to answer questions accurately, at some point it has to go get the real page.

Why agents can’t just fetch the URL themselves

The obvious fix looks simple: take the link from the search result, fetch() it, done. In practice this breaks constantly:

  • Bot defenses. A meaningful fraction of the web sits behind Cloudflare challenges, rate limiters, or outright captchas that a plain HTTP client never gets past.
  • Raw HTML is enormous and mostly noise. A typical page is bloated with <nav>, <script>, <style>, tracking pixels, and boilerplate. Feed that straight into an LLM’s context window and you’re spending tokens — and money — on navigation menus instead of the actual content you asked for.
  • Inconsistent shapes. Every site marks up its content differently. An agent that tries to regex or DOM-parse its way to “the article text” on arbitrary URLs is signing up for endless one-off fixes.

This is exactly the gap Serply’s Request endpoint is built to close: hand it any URL, and it fetches the page with automatic captcha bypass, then hands the content back in whichever shape your agent actually needs.

Search + Request: the pattern that actually works

The pattern that holds up in production is simple: use search to find the right URL, then use a request/scrape call to pull its real content.

1. GET /v1/search/q=your+question   → candidate URLs
2. POST /v1/request { url, response_type: "markdown" } → actual page content
3. Feed the markdown to your LLM

Concretely: search for https://news.ycombinator.com returns a link and a snippet. Passing that same URL to /v1/request with response_type: "markdown" returns the front page’s actual content, already converted to clean markdown — headlines, points, comment counts — with the script tags, ad slots, and layout chrome stripped out. That’s the difference between an agent that guesses from a snippet and one that reasons over the real page.

Using the Request endpoint

The endpoint is a single POST, and it’s worth knowing exactly how the two response types differ, because they’re not the same shape.

POST https://api.serply.io/v1/request

Body:

{
  "url": "https://news.ycombinator.com/",
  "response_type": "full"
}

response_type: "full" returns the raw HTML — but wrapped in JSON, under a data key:

{
  "data": "<html>...</html>"
}

response_type: "markdown" is different in an important way: it returns the converted markdown as the raw response body itself — not wrapped in JSON. The Content-Type header comes back as text/html; charset=utf-8, and if you try to call .json() on it, you’ll get a parse error. Read it as plain text.

JavaScript

// Full HTML — parse as JSON, then pull data.data
const fullResponse = await fetch('https://api.serply.io/v1/request', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'X-Api-Key': 'YOUR_API_KEY'
  },
  body: JSON.stringify({
    url: 'https://news.ycombinator.com/',
    response_type: 'full'
  })
});
const { data: html } = await fullResponse.json();

// Markdown — read as plain text, NOT JSON
const mdResponse = await fetch('https://api.serply.io/v1/request', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'X-Api-Key': 'YOUR_API_KEY'
  },
  body: JSON.stringify({
    url: 'https://news.ycombinator.com/',
    response_type: 'markdown'
  })
});
const markdown = await mdResponse.text();

// This is what you actually hand to your agent's context window:
console.log(markdown);

Python

import requests

headers = {
    "Content-Type": "application/json",
    "X-Api-Key": "YOUR_API_KEY"
}
url = "https://api.serply.io/v1/request"

# Full HTML
full_payload = {"url": "https://news.ycombinator.com/", "response_type": "full"}
full_resp = requests.post(url, headers=headers, json=full_payload)
html = full_resp.json()["data"]

# Markdown — plain text, not JSON
md_payload = {"url": "https://news.ycombinator.com/", "response_type": "markdown"}
md_resp = requests.post(url, headers=headers, json=md_payload)
markdown = md_resp.text

print(markdown)

For most agent use cases, markdown is the one you want. It’s already stripped of the boilerplate that eats context budget, and LLMs handle markdown structure (headings, lists, links) natively — no HTML parsing library required on your end.

When you actually need full

There are legitimate reasons to ask for full instead of markdown:

  • You need to extract something structural that markdown conversion collapses — a specific data- attribute, an embedded JSON blob in a <script> tag, or exact table markup.
  • You’re feeding the HTML into your own extraction pipeline (a DOM parser, an XPath query) rather than an LLM directly.
  • You need to verify the page rendered correctly before trusting the markdown conversion of it.

Otherwise, default to markdown. It’s smaller, cleaner, and it’s what your model actually reads well.

The token-cost angle is real

If you’re paying per token for LLM calls — and you are — raw HTML is one of the most wasteful things you can put in a context window. A page that reads as a few hundred words of actual content can easily ship as tens of thousands of characters of markup. Converting to markdown before it ever reaches your model isn’t just cleaner engineering, it’s a direct cost reduction on every single agent call that touches the open web.

Building the loop

If you’re building a research agent, a shopping assistant, a support bot that needs to check a docs page, or anything that has to answer questions grounded in real web content — not just what a snippet implies — the loop is the same every time: search to find candidates, request to get the truth, then let the model reason over content it actually read.

Both endpoints sit behind the same authentication and the same API key, so wiring them together is a few lines of code, not a new integration. Start with Google Search to find candidate URLs, then pipe anything worth reading through Request before it hits your model. Full details on both, including auth headers and proxy options, are in the Serply docs — or explore them live at api.serply.io.