Fitting a 3,000-Comment Reddit Thread Into a Context Window

A big thread is a deep tree, not a list. Flattening it naively wastes most of your budget on replies nobody upvoted. Here's the pruning that keeps the signal.

Profile picture of Serply
Serply
A deep comment tree being pruned down to its highest-scoring branches

The first time you point an agent at a genuinely popular Reddit thread, you find out that “read this thread” is not a small request. A front-page post can carry several thousand comments across a tree eight or nine levels deep. Serialized naively, that’s well past 100,000 tokens of mostly “this”, “lol”, and ”^ this guy gets it”.

Truncating at the top doesn’t help either, because the useful answer is frequently four levels down inside the third-highest comment.

This post is about the middle path: how to turn a deep comment tree into a few thousand tokens that actually contain the thread’s conclusions.

What you’re actually parsing

Reddit’s comment response isn’t a flat list, and it isn’t shaped like the other listing endpoints either. The comments endpoint returns an envelope with two listings — the post, then the comment tree:

{
  "cached": true,
  "data": [
    { "kind": "Listing", "data": { "children": [ { "kind": "t3", "data": { "title": "..." } } ] } },
    { "kind": "Listing", "data": { "children": [ { "kind": "t1", "data": { "body": "..." } } ] } }
  ]
}

So the post is data[0].data.children[0].data and the thread is data[1].data.children. Each child is a t1 comment, and replies hang off it recursively. Reddit’s raw payload passes through untouched, so the fields you know from Reddit’s own JSON are all present.

The recursion is where people get bitten. A comment’s replies is either another listing object or an empty string — Reddit uses "" rather than null when there are no children, which will happily blow up a naive replies["data"]["children"] access. There’s also a more marker (kind: "more") standing in for collapsed branches, which has no body at all.

A parser that survives contact:

def walk(children: list, depth: int = 0):
    """Yield (depth, comment_dict) for every real comment in the tree."""
    for child in children:
        if child.get("kind") != "t1":
            continue  # skips "more" markers and anything unexpected
        data = child["data"]
        yield depth, data

        replies = data.get("replies")
        if isinstance(replies, dict):
            yield from walk(replies["data"]["children"], depth + 1)

Guarding on kind != "t1" and isinstance(replies, dict) handles both edge cases without a special case for either.

The naive approach and why it fails

The obvious move is to flatten everything and truncate:

text = "\n".join(c["body"] for _, c in walk(thread))[:50_000]

This is worse than it looks. Reddit’s default confidence sort puts good top-level comments first, so truncation preserves breadth — you get the first N top-level comments and their entire subtrees, including every joke reply, and lose everything after. On a thread where the correct answer is a well-upvoted comment sitting at position 12, you’ve spent 50,000 tokens on eleven comment chains and missed it.

You also lose the structure. Flattened, the model can’t tell that a given comment is a rebuttal of the one above it. That distinction is most of why Reddit is worth reading — the disagreement is the signal.

Prune by score, keep the shape

The fix has two parts: filter aggressively on score, and preserve depth as indentation so the model can still see who’s replying to whom.

def render(children: list, min_score: int = 5, max_depth: int = 3) -> str:
    lines = []
    for depth, c in walk(children):
        if depth > max_depth:
            continue
        # Always keep top-level comments; apply the score floor below that.
        if depth > 0 and c.get("score", 0) < min_score:
            continue
        body = " ".join(c["body"].split())
        if len(body) < 15:
            continue  # "this", "same", "lol"
        lines.append(f"{'  ' * depth}[{c.get('score', 0):+d}] {body}")
    return "\n".join(lines)

Three cuts, each doing distinct work:

Depth cap. Below level three you’re almost always in a tangent. The exceptions exist but aren’t worth the tokens.

Score floor on replies only. Top-level comments stay regardless, because a zero-score top-level comment might be a recent correction that hasn’t been voted on yet. Replies below the floor are noise by community judgment.

Minimum length. Kills the agreement chatter, which is a startling fraction of any large thread by comment count and contributes nothing.

Keeping the score visible in the rendered text matters more than it seems. It gives the model the crowd’s weighting inline, so a [+840] claim and a [+2] contradiction don’t arrive looking equally authoritative.

On a typical 3,000-comment thread this lands somewhere around 4,000–8,000 tokens — a 90%-plus reduction, with the conclusions intact.

Budget it properly

Fixed filters are fragile across threads of different sizes. A score floor of 5 is far too permissive on a thread with 20,000 upvotes and too strict on a niche subreddit post that peaked at 40. Scale the floor to the thread instead:

def adaptive_render(payload: dict, token_budget: int = 6000) -> str:
    post = payload["data"][0]["data"]["children"][0]["data"]
    children = payload["data"][1]["data"]["children"]

    header = f"# {post['title']}\n{post.get('selftext', '')[:1000]}\n\n"

    for min_score in (5, 15, 50, 150, 500):
        body = render(children, min_score=min_score)
        # ~4 chars per token is close enough for a budget check
        if len(header) + len(body) < token_budget * 4:
            return header + body

    return header + render(children, min_score=500, max_depth=1)

Escalating the floor until it fits is more robust than guessing, and it degrades in the right direction: on an enormous thread you end up with only the comments the community strongly endorsed, which is exactly what you’d want to read if you only had a minute.

Ask for less in the first place

The cheapest token is the one you never fetch. Two request-level parameters do real work before any of the above runs.

sort=confidence (the default for comment threads) surfaces well-supported comments rather than merely early ones. sort=top is the alternative when you want raw popularity. For a listing, sort=top with t=month narrows to what a subreddit actually cared about recently:

GET /v1/reddit/subreddit/kubernetes?limit=25&sort=top&t=month
GET /v1/reddit/comments/1vfemi1?sort=confidence
GET /v1/reddit/post/1vfemi1

That last one is the zero-comment case: when the thing worth reading is the post itself — a long write-up, a changelog, a detailed question — post/{id} returns the post alone, so none of your budget goes to a comment tree you were going to discard anyway.

And limit caps the listing at the source — 1 to 100, defaulting to 25. Pulling 100 posts to filter down to 5 in your own code costs the same one credit as pulling 25, but it costs you the parsing time and the temptation to pass them all along.

Pick threads before you read them

The real win is upstream: read fewer threads, more deliberately. Same principle as bounding an agent’s tool calls — the listing endpoint gives you titles, scores, and comment counts cheaply, so triage there and spend the expensive comment fetches on the two or three threads that survive.

def pick_threads(posts: list[dict], n: int = 3) -> list[dict]:
    scored = [
        p for p in posts
        if p["num_comments"] >= 15 and p["score"] >= 25
    ]
    scored.sort(key=lambda p: p["num_comments"], reverse=True)
    return scored[:n]

Sorting by num_comments rather than score is deliberate. A high score means people liked the post; a high comment count means people had something to say about it, and discussion is what you came for. A 5,000-upvote screenshot with 30 comments contains almost nothing. A 300-upvote question with 400 comments is the thread you want.

Three threads at ~6,000 tokens each is 18,000 tokens of genuine community discussion — a fraction of what one unpruned thread would have cost, and a much better answer.

Free retries while you tune

Thresholds like these need a few rounds of tuning against real threads, and that’s normally the expensive part. Reddit responses are cached for ten minutes, and cached responses cost no credits, so iterating on your pruning logic against the same thread is free after the first fetch. Only your first call to a given thread bills.

If you want to try it against your own subreddits, the free tier includes 2,500 credits, which goes a long way when your tuning loop isn’t burning any of them.

Related Posts