A Product Feedback Agent That Mines Reddit for Complaints

Your users file tickets. Everyone else complains on Reddit and never tells you. Here's an agent that reads the second group and reports what's actually breaking.

Profile picture of Serply
Serply
Reddit threads being sorted into recurring product complaint themes

The users who file a support ticket are the ones who still care enough to bother. The much larger group hits a problem, says something about it on Reddit, and quietly switches to a competitor.

That second group is where the useful signal is, and nobody on your team has time to read r/yourcategory every day.

This is an agent that does. It’s about 150 lines, it runs on a schedule, and the design work is mostly in not alerting you about things that don’t matter.

What it needs to do

Four jobs, in order:

  1. Find posts mentioning your product across the subreddits where your category is discussed
  2. Throw out the ones that aren’t complaints
  3. Read the ones that survive, properly
  4. Report only themes that are new or getting worse

Job four is the one that determines whether anyone still reads the output in week three.

Finding candidates

Two discovery paths, and you want both. Subreddit listings catch discussion in communities you already know about. Search catches mentions in places you didn’t think to monitor.

import os
import urllib.parse

import httpx

HEADERS = {"X-Api-Key": os.environ["SERPLY_API_KEY"]}
BASE = "https://api.serply.io/v1"

SUBREDDITS = ["selfhosted", "devops", "sysadmin", "webdev"]
PRODUCT_TERMS = ["yourproduct", "your product"]


def listing(subreddit: str, limit: int = 50) -> list[dict]:
    r = httpx.get(
        f"{BASE}/reddit/subreddit/{subreddit}",
        params={"limit": limit, "sort": "new"},
        headers=HEADERS,
        timeout=15.0,
    )
    r.raise_for_status()
    return [c["data"] for c in r.json()["data"]["children"] if c.get("kind") == "t3"]


def mentions(post: dict) -> bool:
    haystack = f"{post['title']} {post.get('selftext', '')}".lower()
    return any(term in haystack for term in PRODUCT_TERMS)


def discover() -> list[dict]:
    found = {}
    for sub in SUBREDDITS:
        for post in listing(sub):
            if mentions(post):
                found[post["id"]] = post
    return list(found.values())

sort=new rather than hot is deliberate here. You want complaints while they’re fresh, not once they’ve been upvoted to the front page — by then you’re reading about it in your mentions anyway.

Deduplicating by post id matters because cross-posts are common and you don’t want to process the same thread twice.

For the wider net, a site-scoped search picks up subreddits that aren’t on your list:

def search_reddit(term: str, num: int = 20) -> list[dict]:
    q = urllib.parse.quote_plus(f"site:reddit.com {term}")
    r = httpx.get(f"{BASE}/search/q={q}&num={num}", headers=HEADERS, timeout=15.0)
    r.raise_for_status()
    return r.json()["results"]

Run that weekly rather than hourly. New communities don’t appear that fast, and it’s how you discover that your product is being discussed in a subreddit nobody internally had heard of.

Triage before you spend

Most mentions are not complaints. “Anyone using X?” is a question. “X just shipped Y” is news. Reading full comment trees for all of them is how a cheap agent becomes an expensive one.

Classify on the title and post body alone — you already have both from the listing, so this costs nothing extra:

CLASSIFY = """Classify this Reddit post about our product.

Answer with exactly one word:
COMPLAINT  - reports a problem, bug, frustration, or reason they left
QUESTION   - asking for help or opinions
PRAISE     - positive
NEWS       - announcement or link, no opinion
UNRELATED  - different product with a similar name

Title: {title}
Body: {body}"""


def classify(post: dict) -> str:
    msg = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=8,
        messages=[{
            "role": "user",
            "content": CLASSIFY.format(
                title=post["title"],
                body=post.get("selftext", "")[:1500],
            ),
        }],
    )
    return msg.content[0].text.strip().upper()

UNRELATED earns its place in that list. Any product name short enough to be memorable is also a word someone uses to mean something else, and without this category your agent will confidently report that customers are upset about a video game.

Keep COMPLAINT, and keep QUESTION too — an unanswered question about your product is a documentation gap, which is feedback of a different kind.

Reading the survivors

Now the expensive call, on the small set that made it through. The comment thread is where the detail lives: the post says “it broke,” and the eighty comments underneath say how.

def thread(post_id: str) -> tuple[dict, list]:
    r = httpx.get(
        f"{BASE}/reddit/comments/{post_id}",
        params={"sort": "confidence"},
        headers=HEADERS,
        timeout=20.0,
    )
    r.raise_for_status()
    payload = r.json()
    post = payload["data"][0]["data"]["children"][0]["data"]
    comments = payload["data"][1]["data"]["children"]
    return post, comments


def render(comments: list, max_depth: int = 2, min_score: int = 3) -> str:
    lines = []

    def walk(children, depth=0):
        for child in children:
            if child.get("kind") != "t1" or depth > max_depth:
                continue
            c = child["data"]
            if depth > 0 and c.get("score", 0) < min_score:
                continue
            body = " ".join(c["body"].split())
            if len(body) >= 15:
                lines.append(f"{'  ' * depth}[{c.get('score', 0):+d}] {body}")
            replies = c.get("replies")
            if isinstance(replies, dict):
                walk(replies["data"]["children"], depth + 1)

    walk(comments)
    return "\n".join(lines)

The kind != "t1" guard skips Reddit’s more markers for collapsed branches, and the isinstance check handles Reddit returning "" instead of an object when a comment has no replies. Both will crash a naive parser on the first real thread. There’s more on getting large threads down to a workable size if your subreddits run big.

Keeping the score inline is what lets the next step distinguish “one person had this problem” from “forty people confirmed it.”

Extracting themes, not summaries

Here’s where most versions of this go wrong. A per-thread summary produces a pile of prose nobody reads. What a product team needs is the same complaint appearing in five separate threads.

So extract structured issues and cluster them:

EXTRACT = """Extract distinct product issues from this Reddit thread.

For each issue return:
- issue: one specific sentence, no hedging
- severity: blocker | major | minor
- confirmations: how many commenters independently reported it
- quote: the most representative comment, verbatim

Only include issues about our product. Ignore general discussion.
Return JSON: {{"issues": [...]}}

{thread}"""

Then group across threads by similarity, and rank by how many separate threads an issue appears in rather than by upvotes. Cross-thread recurrence is the strongest signal you have — one loud thread might be one unlucky user, but the same complaint surfacing independently in four communities is a real defect.

def rank(all_issues: list[dict]) -> list[dict]:
    for issue in all_issues:
        issue["weight"] = (
            issue["distinct_threads"] * 3
            + issue["confirmations"]
            + (5 if issue["severity"] == "blocker" else 0)
        )
    return sorted(all_issues, key=lambda i: i["weight"], reverse=True)

Only report what changed

Run this daily and report everything, and by day four it’s the same list and everyone’s filtering it to a folder. The agent needs state.

import json
from pathlib import Path

STATE = Path("seen_issues.json")


def load() -> dict:
    return json.loads(STATE.read_text()) if STATE.exists() else {}


def diff(issues: list[dict], seen: dict) -> dict:
    new, escalating = [], []
    for issue in issues:
        key = issue["issue"][:80].lower()
        prior = seen.get(key)
        if prior is None:
            new.append(issue)
        elif issue["weight"] > prior["weight"] * 1.5:
            escalating.append(issue)
        seen[key] = {"weight": issue["weight"]}
    return {"new": new, "escalating": escalating, "seen": seen}

Report the new and the escalating. Silence when there’s nothing new is a feature — it’s what makes the alerts credible when they do arrive. Same discipline as a news-based brand monitor: the value is in what it declines to send you.

Say where it came from

The output should let a human check the work in one click:

NEW — 3 threads, 14 confirmations
Uploads over 100MB fail silently on the free tier with no error message.

  "spent two hours thinking my file was corrupted, turns out it just
   doesn't tell you it hit a limit" (+92, r/selfhosted)

  Threads: reddit.com/r/selfhosted/comments/1vfemi1
           reddit.com/r/devops/comments/1vg2x8p
           reddit.com/r/webdev/comments/1vh0k22

Always include the links. Reddit reports are the kind of finding someone will want to verify personally before acting, and an unverifiable complaint is easy to dismiss. It’s also a hedge against the model overstating a theme — the reader can check in seconds.

One caution worth building in: this agent reads a self-selected group of people who chose to post. That’s a real signal about real problems, but it isn’t a representative sample of your users, and it’s worth checking author history before treating a single vocal account as a trend.

Scheduling and cost

Hourly is more than enough; Reddit discussion doesn’t move faster than that, and responses are cached for ten minutes with cached responses free.

Per run: four listing calls plus maybe three comment fetches on whatever survived triage. Call it seven credits an hour, around 5,000 a month — roughly $5 at the $0.75/1,000 volume rate, or comfortably inside the 2,500 free credits if you run it a few times a day instead.

Which is a very cheap way to find out what your users are saying when they don’t think you’re listening.

Related Posts