How to Tell If Your Agent Is Actually Using Its Search Results

An agent that searches and then answers from memory looks identical to one that searches and reads. Here's how to measure the difference.

Profile picture of Serply
Serply
An evaluation harness checking claims against retrieved sources

The most expensive failure in a search-backed agent is invisible. The agent calls the search tool, gets ten results, and then writes an answer from its training data that happens to be shaped like the results. Traces look healthy. Tool call counts look healthy. The answer is wrong in a way nobody catches until a user notices.

You can’t detect this by reading outputs — a well-written ungrounded answer is more convincing than a hedged grounded one. You need to measure it.

Three things worth measuring

Retrieval quality — did the search return anything that could answer the question? Grounding — is each claim in the answer supported by what was retrieved? Citation accuracy — do the cited URLs actually contain what they’re cited for?

They fail independently. Perfect retrieval with poor grounding is the memory-fallback case. Good grounding with bad citations is a formatting problem. Bad retrieval with confident answers is the worst case, and the most common.

Capturing what the agent saw

You can’t evaluate grounding without recording the retrieved material alongside the answer:

import os
import json
import requests
from urllib.parse import quote_plus
from dataclasses import dataclass, field

API_KEY = os.environ["SERPLY_API_KEY"]


@dataclass
class Trace:
    question: str
    queries: list[str] = field(default_factory=list)
    results: list[dict] = field(default_factory=list)
    pages: dict[str, str] = field(default_factory=dict)
    answer: str = ""


def make_tools(trace: Trace):
    def web_search(query: str, num: int = 10) -> str:
        trace.queries.append(query)
        resp = requests.get(
            f"https://api.serply.io/v1/search/q={quote_plus(query)}&num={num}",
            headers={"X-Api-Key": API_KEY},
            timeout=30,
        )
        resp.raise_for_status()
        results = resp.json().get("results", [])
        trace.results.extend(results)
        return "\n\n".join(
            f"[{r.get('position')}] {r.get('title')}\n{r.get('link')}\n{r.get('description', '')}"
            for r in results
        )

    def read_page(url: str) -> str:
        resp = requests.post(
            "https://api.serply.io/v1/request",
            headers={"X-Api-Key": API_KEY, "Content-Type": "application/json"},
            json={"url": url, "response_type": "markdown"},
            timeout=60,
        )
        resp.raise_for_status()
        text = resp.text[:15000]
        trace.pages[url] = text
        return text

    return web_search, read_page

Recording inside the tool, not around it, is the point — you capture exactly the bytes the model saw, including truncation.

Decomposing the answer into claims

Grounding is per-claim, not per-answer. Split first:

from anthropic import Anthropic

client = Anthropic()

DECOMPOSE = """Split this answer into atomic factual claims.

A claim is one verifiable assertion. Split compound sentences. Drop hedging,
opinions, and meta-commentary like "based on my search."

Return a JSON array of strings. No other output."""


def claims(answer: str) -> list[str]:
    msg = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=2000,
        system=DECOMPOSE,
        messages=[{"role": "user", "content": answer}],
    )
    return json.loads(msg.content[0].text)

Judging each claim against the evidence

JUDGE = """You check whether a claim is supported by source material.

Verdicts:
- "supported": the material directly states this
- "partial": the material implies it but does not state it
- "unsupported": the material does not contain this
- "contradicted": the material says otherwise

Be strict. If the claim contains a specific number, date, or name, that exact
detail must appear in the material. Plausibility is not support.

Return JSON: {"verdict": str, "evidence": str|null}
`evidence` is a direct quote from the material, or null."""


def judge_claim(claim: str, evidence_text: str) -> dict:
    msg = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=800,
        system=JUDGE,
        messages=[{"role": "user", "content":
                   f"CLAIM:\n{claim}\n\nSOURCE MATERIAL:\n{evidence_text[:30000]}"}],
    )
    return json.loads(msg.content[0].text)

“Plausibility is not support” is the sentence that makes this eval work. Without it, a judge model reads a claim that sounds right, finds nothing contradicting it, and marks it supported — reproducing the exact failure you’re trying to detect.

Requiring a verbatim quote for supported is the other half. A judge that must produce the supporting text can’t hand-wave.

The harness

def evaluate(trace: Trace) -> dict:
    evidence = "\n\n".join(
        [f"{r.get('title')}\n{r.get('link')}\n{r.get('description', '')}"
         for r in trace.results]
        + [f"--- {url} ---\n{text}" for url, text in trace.pages.items()]
    )

    verdicts = [
        {"claim": c, **judge_claim(c, evidence)}
        for c in claims(trace.answer)
    ]

    total = len(verdicts) or 1
    counts = {v: 0 for v in ("supported", "partial", "unsupported", "contradicted")}
    for v in verdicts:
        counts[v["verdict"]] = counts.get(v["verdict"], 0) + 1

    return {
        "question": trace.question,
        "n_claims": len(verdicts),
        "grounding_rate": counts["supported"] / total,
        "hallucination_rate": (counts["unsupported"] + counts["contradicted"]) / total,
        "counts": counts,
        "verdicts": verdicts,
        "n_queries": len(trace.queries),
        "n_pages_read": len(trace.pages),
    }

The diagnostic that matters

Split your eval set by whether retrieval succeeded:

def retrieval_succeeded(trace: Trace) -> bool:
    return len(trace.results) >= 3 and len(trace.pages) >= 1


good = [r for t, r in runs if retrieval_succeeded(t)]
bad = [r for t, r in runs if not retrieval_succeeded(t)]

print(f"retrieval OK  → hallucination {mean(r['hallucination_rate'] for r in good):.1%}")
print(f"retrieval bad → hallucination {mean(r['hallucination_rate'] for r in bad):.1%}")

If the second number is high and the answers in that bucket are still confident and fluent, you’ve found the memory-fallback problem. The fix isn’t a better model — it’s making the tool’s empty-result message explicit (“try different phrasing before concluding this doesn’t exist”) and adding a guardrail that rejects answers with zero citations.

A cheap regression check

Full LLM-judged evals are too slow for CI. A crude proxy catches most regressions:

def cheap_check(trace: Trace) -> dict:
    cited = set(re.findall(r"https?://[^\s\)\]]+", trace.answer))
    retrieved = {r.get("link") for r in trace.results} | set(trace.pages)
    return {
        "has_citations": bool(cited),
        "fabricated_urls": len(cited - retrieved),
        "unread_citations": len(cited - set(trace.pages)),
    }

fabricated_urls should be zero, always. A URL in the answer that never appeared in any search result is a hallucination you can detect with a set difference — no judge model required, and it belongs in your test suite today.