A Fact-Checking Agent That Admits When It Can't Tell

• Decomposing first
• Searching for evidence, not confirmation
• Gathering from two surfaces
• Independence matters more than volume
• The verdict, with an escape hatch
• Enforcing it in code

A fact-checking agent that always returns a verdict is worse than no fact-checker, because a confident “false” on something merely unverifiable is itself misinformation. The design goal isn’t maximum coverage. It’s calibration.

Decomposing first

Real-world claims are compound. “The company laid off 500 people last quarter after missing revenue targets” is three assertions, and they can have different truth values.

import os
import json
import requests
from urllib.parse import quote_plus
from anthropic import Anthropic

API_KEY = os.environ["SERPLY_API_KEY"]
client = Anthropic()

DECOMPOSE = """Split this statement into atomic, independently checkable claims.

Each claim must be verifiable on its own — include the subject even if the
original used a pronoun. Drop opinion and framing; keep only assertions of fact.

Return JSON: {"claims": [{"text": str, "type": "statistic"|"event"|"quote"|"attribute"}]}"""


def decompose(statement: str) -> list[dict]:
    msg = client.messages.create(
        model="claude-sonnet-4-5", max_tokens=1500,
        system=DECOMPOSE,
        messages=[{"role": "user", "content": statement}],
    )
    return json.loads(msg.content[0].text)["claims"]

Tagging the claim type pays off later — a statistic needs a primary source, an event needs date-consistent coverage, and they warrant different search strategies.

Searching for evidence, not confirmation

The naive approach searches the claim verbatim, which surfaces pages that already agree with it. Generate queries that could turn up a contradiction too:

QUERIES = """Write 3 search queries to verify this claim.

1. A neutral query for the underlying facts (no wording from the claim itself)
2. A query aimed at the primary or official source
3. A query that would surface a correction or dispute, if one exists

Keyword style, not sentences. Return JSON: {"queries": [str, str, str]}"""


def evidence_queries(claim: str) -> list[str]:
    msg = client.messages.create(
        model="claude-sonnet-4-5", max_tokens=500,
        system=QUERIES,
        messages=[{"role": "user", "content": claim}],
    )
    return json.loads(msg.content[0].text)["queries"]

The third query is the one that changes outcomes. Searching a false claim’s own phrasing reliably finds the sites repeating it; searching for the correction finds the debunk.

Gathering from two surfaces

def web_search(query: str, num: int = 10) -> list[dict]:
    resp = requests.get(
        f"https://api.serply.io/v1/search/q={quote_plus(query)}&num={num}",
        headers={"X-Api-Key": API_KEY, "X-Proxy-Location": "US"},
        timeout=30,
    )
    resp.raise_for_status()
    return resp.json().get("results", [])


def news_search(query: str) -> list[dict]:
    resp = requests.get(
        f"https://api.serply.io/v1/news/q={quote_plus(query)}",
        headers={"X-Api-Key": API_KEY},
        timeout=30,
    )
    resp.raise_for_status()
    return resp.json().get("feed", {}).get("entries", [])

The two endpoints have different response shapes — web results live under results, news articles under feed.entries with title, link, summary, published, and source. For event claims the news endpoint’s publication dates are the reason to bother: a claim about “last quarter” is checkable against when coverage actually appeared.

Independence matters more than volume

Ten sources that all syndicate one wire story is one source. Count domains:

from urllib.parse import urlparse
from collections import defaultdict


def domain(url: str) -> str:
    host = urlparse(url or "").netloc.lower()
    return host[4:] if host.startswith("www.") else host


def independent_sources(items: list[dict], key: str = "link") -> dict[str, list]:
    by_domain = defaultdict(list)
    for i in items:
        d = domain(i.get(key, ""))
        if d:
            by_domain[d].append(i)
    return by_domain

Then read one page per domain rather than five from the same site:

def read(url: str) -> str | None:
    try:
        resp = requests.post(
            "https://api.serply.io/v1/request",
            headers={"X-Api-Key": API_KEY, "Content-Type": "application/json"},
            json={"url": url, "response_type": "markdown"},
            timeout=60,
        )
        resp.raise_for_status()
        return resp.text[:12000]
    except requests.RequestException:
        return None

Markdown mode returns text as the response body, so .text rather than .json().

The verdict, with an escape hatch

VERDICT = """You grade a factual claim against source material.

Verdicts:
- "supported": multiple independent sources state this directly
- "contradicted": sources state something incompatible
- "disputed": credible sources disagree with each other
- "unverifiable": sources don't address it, or only repeat the claim without
  independent confirmation

Rules:
- A specific number, date, or name must appear verbatim in a source to be supported.
- Sources that merely repeat the claim are not confirmation. Look for an original
  or primary source.
- Fewer than 2 independent domains means at most "unverifiable", never "supported".
- "unverifiable" is a good answer. Prefer it over a weakly-supported verdict.

Return JSON:
{"verdict": str, "confidence": "high"|"medium"|"low",
 "evidence": [{"quote": str, "url": str}], "reasoning": str}"""


def grade(claim: str, sources: list[dict]) -> dict:
    material = "\n\n".join(
        f"--- {s['url']} ---\n{s['text'][:6000]}" for s in sources
    )
    msg = client.messages.create(
        model="claude-sonnet-4-5", max_tokens=2000,
        system=VERDICT,
        messages=[{"role": "user", "content": f"CLAIM: {claim}\n\nSOURCES:\n{material}"}],
    )
    return json.loads(msg.content[0].text)

The two-independent-domain floor is a structural constraint rather than an appeal to judgment, which is why it holds. And telling the model explicitly that “unverifiable is a good answer” measurably shifts behavior — without it, models treat every verdict other than a definite one as a failure to try hard enough.

Enforcing it in code

Don’t trust the model to honor its own rules:

def enforce(verdict: dict, sources: list[dict]) -> dict:
    domains = {domain(s["url"]) for s in sources}
    cited = {domain(e.get("url", "")) for e in verdict.get("evidence", [])}

    if verdict["verdict"] == "supported" and len(cited) < 2:
        verdict["verdict"] = "unverifiable"
        verdict["reasoning"] = (
            "Downgraded: fewer than two independent domains cited. "
            + verdict.get("reasoning", "")
        )
    if not cited <= domains:
        verdict["verdict"] = "unverifiable"
        verdict["reasoning"] = "Downgraded: cited a source not in the evidence set."
    return verdict

That second check catches fabricated citations — a URL in the evidence list that never appeared in retrieval. It’s a set difference, it costs nothing, and it should be in every citation-producing pipeline you build.