A Haystack Component for Live Web Retrieval

• The component
• Fetching full pages
• Assembling the pipeline
• Hybrid retrieval

Haystack 2.x components are refreshingly plain: a class, a @component decorator, an @component.output_types annotation, and a run method. If your component emits Document objects, every downstream piece — rankers, prompt builders, generators — works with it immediately.

That makes adding live web retrieval to an existing RAG pipeline a matter of one new node, not a rewrite.

The component

import os
import requests
from urllib.parse import quote_plus

from haystack import component, Document


@component
class SerplyWebSearch:
    """Retrieves live web results as Haystack Documents."""

    def __init__(
        self,
        api_key: str | None = None,
        top_k: int = 10,
        proxy_location: str = "US",
    ):
        self.api_key = api_key or os.environ["SERPLY_API_KEY"]
        self.top_k = top_k
        self.proxy_location = proxy_location

    @component.output_types(documents=list[Document])
    def run(self, query: str):
        resp = requests.get(
            f"https://api.serply.io/v1/search/q={quote_plus(query)}&num={self.top_k}",
            headers={
                "X-Api-Key": self.api_key,
                "X-Proxy-Location": self.proxy_location,
            },
            timeout=30,
        )
        resp.raise_for_status()
        results = resp.json().get("results", [])

        documents = [
            Document(
                content=f"{r.get('title', '')}\n\n{r.get('description', '')}",
                meta={
                    "url": r.get("link", ""),
                    "title": r.get("title", ""),
                    "position": r.get("position"),
                    "source": "google",
                },
                score=1.0 / (r.get("position") or 1),
            )
            for r in results
        ]
        return {"documents": documents}

Serply’s search endpoint puts the query string in the path — /v1/search/q=... — and returns organic hits under results, each carrying title, link, description, and position.

The score here is reciprocal rank, not a similarity value. Haystack rankers that expect embedding scores in a fixed range will misbehave if you mix these documents directly with vector-store output; run a proper ranker over the merged set instead of trusting the raw numbers.

Fetching full pages

Snippets rarely answer a real question. A second component turns URLs into full documents:

from concurrent.futures import ThreadPoolExecutor


@component
class SerplyPageFetcher:
    """Fetches full page text for documents that carry a URL."""

    def __init__(self, api_key: str | None = None, top_k: int = 4, max_chars: int = 12000):
        self.api_key = api_key or os.environ["SERPLY_API_KEY"]
        self.top_k = top_k
        self.max_chars = max_chars

    def _fetch(self, url: str) -> str | None:
        try:
            resp = requests.post(
                "https://api.serply.io/v1/request",
                headers={"X-Api-Key": self.api_key, "Content-Type": "application/json"},
                json={"url": url, "response_type": "markdown"},
                timeout=60,
            )
            resp.raise_for_status()
            return resp.text
        except requests.RequestException:
            return None

    @component.output_types(documents=list[Document])
    def run(self, documents: list[Document]):
        targets = documents[: self.top_k]
        urls = [d.meta.get("url", "") for d in targets]

        with ThreadPoolExecutor(max_workers=self.top_k) as pool:
            texts = list(pool.map(self._fetch, urls))

        out = []
        for doc, text in zip(targets, texts):
            if text:
                out.append(Document(
                    content=text[: self.max_chars],
                    meta={**doc.meta, "fetched": True},
                    score=doc.score,
                ))
            else:
                out.append(doc)
        return {"documents": out + documents[self.top_k :]}

Markdown mode returns the converted text as the response body rather than JSON, so .text is right. Use "response_type": "full" only if you actually need markup — it returns JSON with the HTML under data, and feeding raw HTML to an LLM wastes an enormous number of tokens on tags.

Falling back to the snippet document when a fetch fails, rather than dropping it, keeps the pipeline’s document count stable. Silent shrinkage downstream is a miserable thing to debug.

Assembling the pipeline

from haystack import Pipeline
from haystack.components.builders import PromptBuilder
from haystack.components.generators import OpenAIGenerator

template = """
Answer the question using only the sources below. Cite each claim with its URL.
If the sources do not answer it, say so plainly.

{% for doc in documents %}
--- {{ doc.meta.url }} ---
{{ doc.content }}
{% endfor %}

Question: {{ query }}
Answer:
"""

pipe = Pipeline()
pipe.add_component("search", SerplyWebSearch(top_k=10))
pipe.add_component("fetch", SerplyPageFetcher(top_k=4))
pipe.add_component("prompt", PromptBuilder(template=template, required_variables=["query"]))
pipe.add_component("llm", OpenAIGenerator(model="gpt-4o"))

pipe.connect("search.documents", "fetch.documents")
pipe.connect("fetch.documents", "prompt.documents")
pipe.connect("prompt", "llm")

result = pipe.run({
    "search": {"query": "SERP API geo-targeting support"},
    "prompt": {"query": "Which SERP APIs let you choose a proxy country?"},
})
print(result["llm"]["replies"][0])

Hybrid retrieval

The version worth actually shipping merges web results with your own corpus:

from haystack.components.joiners import DocumentJoiner

pipe.add_component("local", your_embedding_retriever)
pipe.add_component("join", DocumentJoiner(join_mode="reciprocal_rank_fusion"))

pipe.connect("local.documents", "join.documents")
pipe.connect("fetch.documents", "join.documents")
pipe.connect("join.documents", "prompt.documents")

Reciprocal rank fusion is the right join mode here precisely because the two sources produce incomparable scores. It only looks at ordering, which sidesteps the whole problem of normalizing a cosine similarity against a SERP position.