A Support Agent That Reads the Docs Instead of Remembering Them

Indexing your documentation into a vector store means re-indexing it forever. Site-scoped search reads the current version every time.

Profile picture of Serply
Serply
A support agent retrieving answers from live documentation pages

The standard architecture for a docs support bot is a vector store: crawl the documentation, chunk it, embed it, retrieve, answer. It works, and it comes with a permanent maintenance obligation — every docs deploy makes the index stale, and the failure is silent. The bot confidently explains a flag you removed two releases ago.

There’s a lighter path for public documentation: search it live, scoped to your own site, and read the page. No index, no crawl schedule, no staleness.

Search operators work in the query, and the query goes in the path:

import os
import requests
from urllib.parse import quote_plus

API_KEY = os.environ["SERPLY_API_KEY"]
BASE = "https://api.serply.io/v1"


def site_search(question: str, site: str, num: int = 8) -> list[dict]:
    query = f"site:{site} {question}"
    resp = requests.get(
        f"{BASE}/search/q={quote_plus(query)}&num={num}",
        headers={"X-Api-Key": API_KEY, "X-Proxy-Location": "US"},
        timeout=30,
    )
    resp.raise_for_status()
    return [
        {"title": r.get("title", ""), "url": r.get("link", ""),
         "snippet": r.get("description", ""), "rank": r.get("position")}
        for r in resp.json().get("results", [])
    ]

quote_plus on the whole query including the operator. The colon in site: is safe, but the question that follows it usually isn’t.

One thing to know upfront: this depends on your docs being indexed. Pages behind auth, freshly published pages, and anything a noindex tag covers won’t appear. That constraint is why this is a pattern for public documentation specifically.

Search your docs and the surrounding ecosystem

Users’ questions often aren’t answered in your docs at all — they’re answered in a GitHub issue or a forum thread. Search both, and label the difference:

from concurrent.futures import ThreadPoolExecutor


def gather(question: str, docs_site: str, community_sites: list[str]) -> dict:
    def one(site: str) -> tuple[str, list[dict]]:
        return site, site_search(question, site, num=6)

    sites = [docs_site] + community_sites
    with ThreadPoolExecutor(max_workers=len(sites)) as pool:
        found = dict(pool.map(one, sites))

    return {
        "official": found.get(docs_site, []),
        "community": [
            {**r, "site": s}
            for s in community_sites for r in found.get(s, [])
        ],
    }

Keeping these separate through the whole pipeline matters. Official documentation is authoritative about intended behaviour; a three-year-old forum post is evidence about what someone experienced once. Collapsing them into one ranked list loses that, and the model will cite a stale workaround as though it were policy.

Reading the pages

def read(url: str) -> str | None:
    try:
        resp = requests.post(
            f"{BASE}/request",
            headers={"X-Api-Key": API_KEY, "Content-Type": "application/json"},
            json={"url": url, "response_type": "markdown"},
            timeout=60,
        )
        resp.raise_for_status()
        return resp.text
    except requests.RequestException:
        return None

.text, not .json() — markdown mode returns the content as the response body.

Markdown is the right format for documentation specifically. It preserves heading hierarchy, code blocks, and tables, which is most of what a docs page’s meaning lives in. Plain-text extraction that flattens a parameter table into a run-on paragraph makes the page unusable as evidence.

Trimming to the relevant section

Docs pages are long, and the answer is usually one section. Slice by heading before spending context:

import re


def sections(markdown: str) -> list[dict]:
    parts, current = [], {"heading": "(intro)", "level": 0, "body": []}

    for line in markdown.splitlines():
        m = re.match(r"^(#{1,4})\s+(.*)", line)
        if m:
            if current["body"]:
                parts.append({**current, "body": "\n".join(current["body"])})
            current = {"heading": m.group(2).strip(),
                       "level": len(m.group(1)), "body": []}
        else:
            current["body"].append(line)

    if current["body"]:
        parts.append({**current, "body": "\n".join(current["body"])})
    return parts


def relevant_sections(markdown: str, question: str, keep: int = 4) -> str:
    terms = {w for w in re.findall(r"[a-z0-9_]+", question.lower()) if len(w) > 3}
    scored = []

    for s in sections(markdown):
        blob = f"{s['heading']} {s['body']}".lower()
        hits = sum(1 for t in terms if t in blob)
        heading_hits = sum(1 for t in terms if t in s["heading"].lower())
        scored.append((hits + 3 * heading_hits, s))

    scored.sort(key=lambda x: -x[0])
    chosen = [s for score, s in scored[:keep] if score > 0]
    if not chosen:
        return markdown[:8000]

    return "\n\n".join(f"## {s['heading']}\n{s['body']}" for s in chosen)

Weighting heading matches triple is a small heuristic that works well on documentation. A term appearing in a section title is a much stronger signal than the same term appearing in a paragraph, because docs headings are written to be scanned.

Answering

import json
from anthropic import Anthropic

client = Anthropic()

SYSTEM = """You answer product support questions from documentation.

You have two kinds of sources:
- OFFICIAL DOCS: authoritative. Prefer these for how the product is supposed
  to work.
- COMMUNITY: forum posts, issues, discussions. Useful for workarounds and known
  problems, but may be outdated or describe a different version. Always label
  a community-sourced answer as such.

Rules:
- Answer only from the provided sources. Never fill in a parameter name, flag,
  default value, or endpoint from memory — those change between versions and a
  wrong one costs the user real time.
- Quote exact code, commands, and parameter names as they appear.
- Link the specific documentation page for every step.
- If the docs don't cover it, say so and suggest what to ask support. Do not
  construct a plausible-looking answer.
- If official docs and a community post conflict, say the docs are authoritative
  and note that the community post may describe an older version.

Return JSON:
{"answer": str, "steps": [str], "sources": [{"url": str, "type": "official"|"community"}],
 "confidence": "high"|"medium"|"low", "escalate": bool, "escalate_reason": str|null}"""


def support_answer(question: str, docs_site: str,
                   community_sites: list[str]) -> dict:
    found = gather(question, docs_site, community_sites)

    official_urls = [r["url"] for r in found["official"][:3]]
    community_urls = [r["url"] for r in found["community"][:2]]

    with ThreadPoolExecutor(max_workers=5) as pool:
        pages = list(pool.map(read, official_urls + community_urls))

    blocks = []
    for url, text in zip(official_urls, pages[:len(official_urls)]):
        if text:
            blocks.append(f"=== OFFICIAL DOCS: {url} ===\n"
                          f"{relevant_sections(text, question)}")
    for url, text in zip(community_urls, pages[len(official_urls):]):
        if text:
            blocks.append(f"=== COMMUNITY: {url} ===\n{text[:6000]}")

    if not blocks:
        return {"answer": "I couldn't find documentation covering this.",
                "steps": [], "sources": [], "confidence": "low",
                "escalate": True, "escalate_reason": "no sources retrieved"}

    msg = client.messages.create(
        model="claude-sonnet-4-5", max_tokens=2500, system=SYSTEM,
        messages=[{"role": "user", "content":
                   f"QUESTION: {question}\n\n" + "\n\n".join(blocks)}],
    )
    return json.loads(msg.content[0].text)

The instruction not to invent parameter names is the most important line in the prompt. It’s the specific failure that makes docs bots worse than useless — a hallucinated --verbose-output flag sends someone to debug their own environment for twenty minutes before they discover the flag doesn’t exist.

Escalation as a first-class outcome

def route(result: dict, question: str) -> dict:
    should_escalate = (
        result.get("escalate")
        or result.get("confidence") == "low"
        or not result.get("sources")
        or all(s["type"] == "community" for s in result.get("sources", []))
    )
    return {
        **result,
        "route": "human" if should_escalate else "auto",
        "handoff_note": (
            f"Question: {question}\n"
            f"Docs checked: {[s['url'] for s in result.get('sources', [])]}\n"
            f"Reason: {result.get('escalate_reason') or 'low confidence'}"
        ) if should_escalate else None,
    }

Escalating when every source is community is a rule worth having. An answer assembled entirely from forum posts might be right, but it isn’t documentation, and presenting it with the same authority is how a support bot ends up recommending a workaround that was fixed properly two releases ago.

The handoff note matters as much as the routing. A human picking this up should see what was already searched, so they don’t repeat it.

The trade-off, stated plainly

This is slower and costs more per question than a vector store — two or three HTTP round trips instead of one embedding lookup. What you get for that is an answer from the documentation as it exists right now, with no index to rebuild and no silent staleness.

For a docs site that changes weekly, that trade is usually worth it. For a stable corpus that changes twice a year, a vector store is the better engineering call.