The Hidden Token Cost of Feeding AI Agents Raw HTML

• What’s actually in that HTML
• A rough sense of the ratio
• The fix: don’t scrape HTML, scrape markdown
• When you actually want the full HTML
• The bigger picture

If you’ve built an AI agent that browses the web, you’ve probably done the obvious thing at some point: fetch a page, grab the HTML, and hand the whole thing to your LLM. It works. It’s also almost certainly wasting a huge chunk of your token budget on content your model will never use.

What’s actually in that HTML

Open the “view source” on almost any modern website and you’ll find that the visible article, product listing, or forum thread is a small fraction of the document. The rest is:

• Inline <script> blocks for analytics, ad tech, and client-side rendering
• <style> blocks and utility-class soup (class="flex items-center gap-2 md:gap-4 lg:gap-6 ...")
• Navigation menus, footers, cookie banners, and newsletter modals repeated on every page
• Deeply nested <div> wrappers that exist purely for layout, not content
• Tracking pixels, hidden metadata tags, and JSON blobs embedded for client-side hydration

None of that is useless to a browser. All of it is dead weight to an LLM that just needs to know what the page says.

A rough sense of the ratio

The exact numbers vary wildly by site, but the pattern is consistent enough to reason about. A typical news article or blog post, once you count every script tag, style block, and layout wrapper, might run 80,000-150,000 characters of raw HTML for a page whose actual readable content — the headline, byline, and body text — is more like 3,000-6,000 characters once it’s been stripped down to clean markdown.

That’s illustrative, not a measurement of any specific page — some sites are leaner, some are far worse (looking at you, single-page apps that ship megabytes of hydration JSON). But a 10-20x gap between “what’s on the wire” and “what the model actually needs” is a normal, everyday occurrence, not an edge case.

Two things follow from that gap:

It costs you money. Tokens are tokens whether they’re signal or noise. If your agent fetches ten pages a minute, the difference between paying for 100,000 characters of markup per fetch versus 4,000 characters of markdown is not a rounding error — it compounds fast across a production workload, and at typical per-million-token LLM pricing (again, illustrative round numbers — check current pricing for whatever model you’re using) that gap alone can be the difference between a research agent that’s cheap to run and one that quietly burns your budget.

It degrades output quality. This part is less obvious but arguably more important. LLMs don’t process a long context uniformly — the more irrelevant material surrounds the content you actually care about, the harder it is for the model to reliably locate and reason about the signal buried in it. This is the same “needle in a haystack” effect you’ll see discussed around long-context benchmarks: recall gets less reliable as the haystack grows, even when the needle is technically “in there somewhere.” Handing an agent 140KB of markup to find one paragraph of actual text isn’t just wasteful, it’s actively working against the model.

The fix: don’t scrape HTML, scrape markdown

The straightforward fix is to convert scraped pages to markdown before they ever reach your LLM’s context window, rather than asking the model to wade through raw markup and figure out what matters. Strip the scripts, strip the styles, strip the layout divs, keep the headings, links, and body text.

This is exactly what Serply’s Request endpoint does. It’s a single API call that fetches a URL and can hand you back either format:

import requests

url = "https://api.serply.io/v1/request"
headers = {
    "Content-Type": "application/json",
    "X-Api-Key": "YOUR_API_KEY"
}

# Raw HTML -- fine if you need the full DOM for some other reason
full = requests.post(url, headers=headers, json={
    "url": "https://example.com/some-article",
    "response_type": "full"
})
html = full.json()["data"]
print(len(html))  # often well into six figures for a real page

# Clean markdown -- what your agent should actually be reading
markdown = requests.post(url, headers=headers, json={
    "url": "https://example.com/some-article",
    "response_type": "markdown"
})
text = markdown.text  # plain markdown, not JSON -- read the body directly
print(len(text))  # a small fraction of the HTML response

Note the two response shapes aren’t symmetric, which trips people up the first time: response_type: "full" comes back as JSON with the HTML tucked into a data field (response.json()["data"]), while response_type: "markdown" comes back as the markdown text itself, sent directly as the response body with a text/html; charset=utf-8 content type — no JSON wrapper to unpack, just read response.text. See the full request/response reference for the exact shapes, and the authentication guide for how the X-Api-Key header works.

When you actually want the full HTML

None of this means “full” mode is useless. If your agent needs to inspect specific DOM attributes, follow structured data that markdown conversion would flatten, or you’re building something like a visual scraper or a link-graph crawler that cares about the raw markup itself, response_type: "full" is the right tool. The point isn’t that HTML is bad — it’s that HTML is the wrong default for the common case of “an LLM needs to read and reason about what’s on this page.”

For everything downstream of that common case — RAG pipelines, research agents, summarizers, fact-checkers, anything that hands page content to a language model as context — markdown should be the default, and raw HTML should be the exception you reach for deliberately.

The bigger picture

This is a small, easy-to-fix problem, but it’s also a symptom of a bigger pattern in agent design: it’s tempting to just hand your model “everything” and let it figure out what matters, because that’s the path of least resistance to get something working. It’s also the most expensive and least reliable way to run an agent at scale. Every unnecessary token you feed a model is a token you’re paying for and a token competing for the model’s attention against the content that actually matters.

Converting HTML to markdown before it hits your context window is one of the cheapest wins available for an agent that touches the open web — it’s a config flag, not a redesign. If your agent is currently piping raw HTML into a prompt, try Serply’s Request endpoint with response_type: "markdown" and see how much smaller (and sharper) your context gets.