How to Scrape Reddit Threads

I wanted to archive an entire r/Python discussion: the question someone posted, the body of that post, and every reply underneath it, kept in one file with the structure intact. That is a Reddit thread, and it is a different job from grabbing a list of comments. A thread is the whole conversation as one unit, the post plus its full reply tree, and capturing it means treating those two halves as one thing.

This guide on how to scrape Reddit threads covers the whole-thread workflow. I will show why the obvious shortcut returns a 403, the official PRAW route that pulls the post body and the comment tree together, the managed API that returns both in a single request, and how to export a complete thread without flattening the structure that makes it a conversation.

What is a Reddit thread, exactly?

A Reddit thread is a single post and the entire tree of comments that hangs off it, treated as one conversation. The post is the root: it has a title, an author, a score, and, for a text submission, a body. Underneath sit the top-level comments, each of which can have replies, and each of those replies can have its own replies, several layers deep.

The distinction that matters for scraping is “thread” versus “comment”. A comment is a single node. A thread is the root post plus the whole node tree below it, captured as one object. So a thread scraper has to do something a comment scraper does not always bother with: grab the post body and metadata from the root and the full reply tree, then keep them joined so the discussion reads in order.

That is why two fields do the heavy lifting later. Each comment carries a parent_id that points at whatever it replies to (a comment or the post itself) and a depth that records how far down the tree it sits. Hold on to both and you can rebuild the conversation from a flat file. Drop them and you have a bag of comments with no shape.

Why does the public .json view return a 403?

The quick way to read a thread looks like it should be free: every Reddit URL serves a JSON version if you append .json, so https://www.reddit.com/r/<sub>/comments/<id>.json ought to hand back the post and its comments. From a datacenter IP it does not. It returns an HTTP 403 HTML page instead, served by snooserv (Reddit’s web edge), before any thread data is sent.

I re-ran that request from a cloud server in June 2026 and got the same 403, and the User-Agent did not change the result. Reddit refuses the connection on the IP reputation and request fingerprint, so a browser User-Agent string on a datacenter IP makes no difference. The same block, and what actually moves it, is what I walk through in my guide on how to scrape Reddit without getting blocked.

So the naive thread fetch fails at the first step. A working thread scraper has to authenticate through a route Reddit answers, or come from an IP Reddit trusts. The official API handles the first, and PRAW wraps it cleanly.

How do you scrape a whole Reddit thread with PRAW?

You scrape a whole Reddit thread with PRAW by loading the submission by its id, reading the post body and metadata off the submission object, then walking its comment forest for the replies. PRAW (the Python Reddit API Wrapper) handles the OAuth token and the comment-tree objects, so the two halves of the thread come back in a few lines.

First, register a script app at https://www.reddit.com/prefs/apps to get a client_id and client_secret. A read-only instance needs those plus a descriptive user agent. The post id is the short code in any thread URL, the part after /comments/:

import praw

reddit = praw.Reddit(
    client_id="YOUR_ID",
    client_secret="YOUR_SECRET",
    user_agent="reddit-thread-scraper/1.0 by u/yourname",
)

submission = reddit.submission(id="1uf89x0")   # the post id from /comments/<id>/

# The root of the thread: the post itself.
print("TITLE :", submission.title)
print("AUTHOR:", submission.author)
print("SCORE :", submission.score, "| comments:", submission.num_comments)
print("BODY  :", submission.selftext[:200])    # empty string for a link post

That is the part a comment scraper skips. The post body is not in the comment tree, it lives on the submission as submission.selftext, and for a text post that is the opening message of the whole thread. A link post has an empty selftext and the destination URL in submission.url instead. The PRAW submission documentation lists the rest of the root fields worth keeping: title, score, num_comments, created_utc, permalink, and locked.

With the post captured, the comment tree is the other half. submission.comments is a CommentForest, and iterating it gives you the top-level replies, each with its own nested replies:

def walk(comments, depth=0):
    for c in comments:
        print("  " * depth, c.score, c.author, "-", (c.body or "")[:60])
        walk(c.replies, depth + 1)        # recurse into nested replies

walk(submission.comments)

Walking the forest recursively, rather than flattening it, is what preserves the thread shape: a reply prints under the comment it answers, at its own depth. The fields you pull per comment are c.body, c.score, c.author, c.created_utc, c.id, and c.parent_id. That parent_id is the link back up the tree, the thing that lets you reconstruct the conversation from a saved file.

One catch carries over from comment scraping. submission.comments only contains the replies Reddit sent on the first load. Deep or numerous branches arrive as MoreComments placeholders, and you expand them with submission.comments.replace_more(limit=None) before you walk the tree. For whole-thread capture the rule is simple: replace_more(limit=None) fills in every collapsed branch so the tree you walk is the complete one, and each placeholder it replaces is a separate API request against the rate limit, which is what makes a very large thread slow.

# Expand collapsed branches so the thread is whole, then walk it.
submission.comments.replace_more(limit=None)
walk(submission.comments)

Reddit’s free tier allows 100 queries per minute per client, averaged over a 10-minute window, and as of 2025 every app needs pre-approval before it can use the Data API, including personal projects. PRAW does not lift that ceiling, because it is the official API underneath. So a thread with thousands of replies is slow and burns rate budget on the expansion, and you maintain the app registration and the token refresh. When that is the bottleneck, the question is how to get the same post-plus-tree without the credentials.

How do you scrape a Reddit thread without the API?

You scrape a Reddit thread without the official API by sending the post to a managed scraper API that returns the root post and the full comment tree as one JSON object, which sidesteps both the OAuth registration and the 403 on the public .json route. In my June 2026 testing, a single call to ChocoData’s Reddit endpoint returned the post body, its metadata, and the comments already expanded and nested, no app to register.

ChocoData is a separate third-party scraper API, not affiliated with this site. The request is a plain GET with the subreddit, the post id, a sort, and your api key:

curl "https://api.chocodata.com/api/v1/reddit/post?subreddit=Python&post_id=1uf89x0&sort=top&api_key=YOUR_CHOCO_API_KEY"

The Python version returns a post object and a comments array in the same response, so you get the whole thread in one shot:

import requests

resp = requests.get(
    "https://api.chocodata.com/api/v1/reddit/post",
    params={
        "subreddit": "Python",
        "post_id": "1uf89x0",
        "sort": "top",                 # top | new | controversial
        "api_key": "YOUR_CHOCO_API_KEY",
    },
    timeout=30,
)
data = resp.json()

post = data["post"]
print("TITLE:", post["title"], "| score:", post["score"])
print("BODY :", (post["body"] or "")[:200])
print(data["comments_returned"], "comments in the tree")

You pass post_id with the subreddit, or a full url instead of either. When I ran that against an r/Python thread (post id 1uf89x0) in June 2026, the post object came back with the title “Monorepo, testing and deployment”, a 1,978-character body, and num_comments of 6, alongside a comments array of those 6 replies, the top one at a score of 19. The post fields are id, title, author, score, upvote_ratio, num_comments, created, body, permalink, external_url, domain, is_locked, and is_removed, so the root of the thread arrives fully described.

The comments come back already nested: every comment carries id, parent_id, depth, score, author, body, created, permalink, and a replies list of the same shape, so the tree is intact without you calling replace_more or chasing placeholders. A _meta block reports the sort, the pages_fetched, and a truncated flag, which was false in my run, meaning the whole thread came back. The proxy rotation, anti-bot handling, and tree expansion all happen server-side, and the call is a multi-hop scrape, so an occasional transient 502 is worth one retry. You can get an API key on the ChocoData sign-up page and drop it into the snippet.

This route trades the free official tier for a managed one, so it earns its keep once your volume or the per-branch expansion cost outweighs your time. For a one-off archive of a single thread, PRAW within the free tier is fine. For pulling complete threads across many posts on a schedule, offloading the expansion and the blocking is usually worth it, and the best Reddit scrapers in 2026 roundup compares the managed options head to head.

How do you export a complete thread to JSON or CSV?

You export a complete thread by saving the post and its comment tree together, and the format depends on whether you want to keep the nesting. JSON keeps the tree as-is, which is the honest representation of a conversation. CSV needs the tree flattened into rows, so you carry parent_id and depth on every row to record the structure you are giving up visually.

For an archive, JSON is the cleaner choice, since the post sits at the root and the replies stay nested underneath exactly as they came back:

import json

thread = {"post": data["post"], "comments": data["comments"]}
with open("thread_1uf89x0.json", "w", encoding="utf-8") as f:
    json.dump(thread, f, ensure_ascii=False, indent=2)

For analysis in pandas or a spreadsheet, flatten the tree to rows. A recursive walk turns the nested replies into a flat table while keeping the parent_id and depth that let you rebuild it later:

import pandas as pd

def flatten(comments, rows):
    for c in comments:
        rows.append({
            "id": c["id"],
            "parent_id": c["parent_id"],     # who this reply answers
            "depth": c["depth"],             # how deep in the thread
            "score": c["score"],
            "author": c["author"]["username"],
            "body": c["body"],
            "created": c["created"],
        })
        flatten(c["replies"], rows)          # recurse into nested replies
    return rows

df = pd.DataFrame(flatten(data["comments"], []))
df.to_csv("thread_comments.csv", index=False)
print(df[["score", "author", "depth"]].head())

The same flatten step works on the PRAW output, except you map each comment object’s attributes into the row instead of dict keys. Either way, store the post row separately or as depth of -1 so the root of the thread is not lost among the replies. Keeping parent_id and depth is what makes the export a thread rather than a pile of comments: with them you can reconstruct the discussion, run a conversation-level analysis, or rebuild a readable transcript. From there it loads into any sentiment or topic pipeline like any other text dataset.

Is it legal to scrape Reddit threads?

Scraping publicly visible Reddit threads is generally treated as lawful in the United States, on the same footing as scraping any other public page. The Ninth Circuit’s ruling in hiQ v. LinkedIn held that scraping public data likely does not violate the Computer Fraud and Abuse Act, and the EFF’s summary of the hiQ decision walks through the reasoning that a public site has “erected no gates” to bypass.

That is the baseline, not the whole picture. Reddit’s User Agreement prohibits automated access without permission, so the terms are a contract question separate from the CFAA one. A whole thread also bundles many authors’ words, and that text can carry personal data with its own obligations under regimes like the GDPR once you store it, while logged-in or private threads are a different matter entirely.

I walk through robots.txt, the User Agreement, and the case law in is scraping Reddit legal, which is the place to settle the question before you archive at scale.

Capturing one thread is a single endpoint in a larger toolkit. For the full workflow across posts, subreddits, and search results, the complete guide to scraping Reddit ties the endpoints together.