How to Scrape Reddit Subreddits

I wanted a rolling dataset of every new post in r/Python, with scores and comment counts, refreshed on a schedule. That is the job people mean by “scrape a subreddit,” and it is a different job from scraping a thread. You are not after the discussion inside one post. You are after the listing: the feed of posts a community surfaces under hot, new, top, or rising, paged through and dumped to a table.

This guide on how to scrape Reddit subreddits covers what that listing actually contains, why the obvious public route returns a 403, the official PRAW way I tested in June 2026, the managed-API way, and how to monitor many communities at once without babysitting proxies.

What does scraping a subreddit return?

Scraping a subreddit returns a listing of posts from that community’s feed, not the comments inside them. Each entry in the listing is a post with its metadata: the title, the score (net upvotes), the num_comments count, the author, the permalink, the created timestamp, and for link posts the outbound domain and external_url. What you do not get is the conversation. The replies under each post are a separate scrape against that post’s thread.

That distinction drives every choice that follows. A subreddit feed is paginated, so pulling it means walking pages with a cursor, not expanding a tree. The feed also has a flavor. The same community returns different posts depending on the sort you ask for, and the sorts are the real knobs:

For monitoring a community as it moves, new is the feed you poll. For building a “best of” dataset, top with a time_filter is the one. The mechanics of fetching them are the same; only the sort string changes.

Why does the public r/subreddit.json route fail?

The public .json route fails because Reddit blocks the request at the IP level before it serves any posts. Append .json to a subreddit URL, request it from a datacenter IP, and you get an HTTP 403 HTML page rather than a listing. I re-ran https://www.reddit.com/r/python/hot.json from a cloud server in June 2026 and got the same 403 served by snooserv (Reddit’s web edge) that I document in my guide on how to scrape Reddit without getting blocked.

The trap is that the route looks like it should work. Years of tutorials tell you to append .json to any Reddit URL for free structured data, and from a residential browser it still does. From a server it does not, and the User-Agent is not the deciding factor. A datacenter IP gets the block whether you send no User-Agent, a full Chrome string, or a descriptive one, because the refusal happens on IP reputation and TLS fingerprint, not the header.

So a subreddit scraper that runs anywhere other than your home machine has to pick a door Reddit actually answers: the authenticated official API, your own residential proxy pool, or a managed API that owns the blocking problem. The official route is the cleanest starting point, and PRAW makes it short.

How do you scrape a subreddit with Python and PRAW?

You scrape a subreddit with Python by authenticating through the official API and iterating the community’s submission listing, and PRAW (the Python Reddit API Wrapper) is the standard way to do it. PRAW handles the OAuth token and the pagination, so the listing code stays to a few lines.

First, register a script app at https://www.reddit.com/prefs/apps to get a client_id and client_secret. A read-only instance needs those plus a descriptive user agent, then reddit.subreddit("name") is your entry point into the feed:

import praw

reddit = praw.Reddit(
    client_id="YOUR_ID",
    client_secret="YOUR_SECRET",
    user_agent="reddit-subreddit-scraper/1.0 by u/yourname",
)

# The hot feed, top 25 posts.
for post in reddit.subreddit("python").hot(limit=25):
    print(post.score, post.num_comments, "-", post.title)

That prints each post’s score, its num_comments, and its title. The fields worth collecting per post are post.title, post.score, post.num_comments, post.author, post.permalink, post.created_utc, post.url (the outbound link for link posts), and post.id. Switching feeds is a method swap, not a rewrite. The PRAW subreddit documentation lists each listing method:

sub = reddit.subreddit("python")

newest = sub.new(limit=50)                       # newest posts first
top_week = sub.top(time_filter="week", limit=50) # best of the last 7 days
rising = sub.rising(limit=25)                     # gaining velocity now

The time_filter on .top() accepts hour, day, week, month, year, and all, which is how you scope a “top posts” pull to a window. PRAW pages through the feed transparently as you iterate, requesting the next batch under the hood until it hits your limit or the feed runs out.

How do you paginate past the first page of a subreddit?

You paginate a subreddit by letting PRAW walk the listing for you, but the feed has a hard ceiling: Reddit will not page a single subreddit feed past roughly 1,000 posts. Iterating with a high limit, or limit=None, keeps fetching batches until that wall, and asking for more does not get you more.

count = 0
for post in reddit.subreddit("python").new(limit=None):
    count += 1
print(count, "posts (Reddit stops near ~1,000 per feed)")

That ceiling is the single fact that reshapes a subreddit project: you cannot pull a community’s entire history out of one feed, so collecting deeply means working around it rather than through it, and the honest workarounds are three. Poll new on a schedule and accumulate posts over time, so your dataset grows past 1,000 even though any single pull cannot. Vary the top time_filter across week, month, year, and all to surface different posts from each window. Or scrape the subreddit’s search by keyword and date, which I cover in how to scrape Reddit search results, to reach posts the listing endpoints will not page back to.

There is also a rate ceiling sitting on top of the post ceiling. Reddit’s free tier allows 100 queries per minute per client, averaged over a 10-minute window, and the OAuth token expires after an hour, so a long-running monitor has to refresh it. PRAW does not lift either limit, because it is the official API underneath. When the per-feed cap, the rate budget, and the app registration start to bite together, the question becomes how to get the same listing without any of them.

How do you scrape a subreddit without the official API?

You scrape a subreddit without the official API by sending the community name to a managed scraper API that returns the post listing as JSON, which sidesteps both the OAuth registration and the 403 you hit on the public .json route. In my June 2026 testing, a single call to ChocoData’s subreddit endpoint returned a community’s posts with real scores, no app to register and no proxy pool to rotate.

ChocoData is an independent scraper-API provider, not affiliated with this site. The request is a plain GET with the subreddit, a sort, and your api key as a query parameter:

curl "https://api.chocodata.com/api/v1/reddit/subreddit?subreddit=Python&sort=top&t=week&limit=25&api_key=YOUR_CHOCO_API_KEY"

The Python version returns a JSON object with a posts array, ready to load into pandas and write to CSV:

import requests
import pandas as pd

resp = requests.get(
    "https://api.chocodata.com/api/v1/reddit/subreddit",
    params={
        "subreddit": "Python",
        "sort": "top",
        "t": "week",
        "limit": 25,
        "api_key": "YOUR_CHOCO_API_KEY",
    },
    timeout=30,
)
data = resp.json()

print(data["total_results"], "posts |", "next cursor:", data["after_cursor"])
posts = data["posts"]
df = pd.DataFrame(posts)[["title", "score", "num_comments", "upvote_ratio", "permalink"]]
df.to_csv("python_top_week.csv", index=False)
print(df.head())

When I ran that against r/Python with sort=top&t=week in June 2026, it returned five posts with real numbers: the top one sat at a score of 258 with 86 comments and an upvote_ratio of 0.90, sourced from Reddit’s Shreddit surface. The response shape is stable. The top level carries subreddit, sort, t, after_cursor, total_results, and posts, and each post in the array has id, title, score, num_comments, upvote_ratio, awards, author, author_id, subreddit, permalink, external_url, domain, and created. The sort parameter accepts hot, new, top, rising, and controversial, and t accepts hour through all for the windowed sorts, mirroring PRAW’s time_filter.

Pagination is the after parameter. Each response hands back an after_cursor, and you feed it into the next request to walk the feed:

def scrape_all(subreddit, sort="new", key="YOUR_CHOCO_API_KEY", max_pages=10):
    rows, after, pages = [], None, 0
    while pages < max_pages:
        params = {"subreddit": subreddit, "sort": sort, "limit": 75, "api_key": key}
        if after:
            params["after"] = after
        data = requests.get(
            "https://api.chocodata.com/api/v1/reddit/subreddit",
            params=params, timeout=30,
        ).json()
        rows.extend(data["posts"])
        after = data["after_cursor"]
        pages += 1
        if not after:           # cursor is null when the feed is exhausted
            break
    return rows

posts = scrape_all("Python", sort="new")
print(len(posts), "posts collected")

The cursor comes back null when the feed is exhausted, which is your stop signal, and the same ~1,000-post ceiling Reddit enforces still applies. You get an API key on the ChocoData sign-up page and drop it into the snippets above. The proxy rotation, the anti-bot handling, and the Shreddit parsing happen server side, so you trade the free official tier for a managed one that costs a few credits per call.

For a single one-off pull, PRAW within the free tier is fine. For listings across many communities on a schedule, offloading the blocking usually wins, and the best Reddit scrapers in 2026 roundup compares the managed options head to head.

How do you monitor many subreddits on a schedule?

You monitor many subreddits by running the listing scrape on a loop, sorting each feed by new, and keeping a record of the post IDs you have already seen so each run only stores the deltas. The pattern is the same whether the fetch is PRAW or a managed API. What makes it a monitor rather than a one-off is the dedupe and the cadence.

import requests, time

WATCH = ["Python", "MachineLearning", "datascience"]
seen = set()

def poll(subreddit, key="YOUR_CHOCO_API_KEY"):
    data = requests.get(
        "https://api.chocodata.com/api/v1/reddit/subreddit",
        params={"subreddit": subreddit, "sort": "new", "limit": 50, "api_key": key},
        timeout=30,
    ).json()
    fresh = [p for p in data["posts"] if p["id"] not in seen]
    for p in fresh:
        seen.add(p["id"])
    return fresh

while True:
    for name in WATCH:
        for p in poll(name):
            print(name, "|", p["score"], "|", p["title"][:60])
    time.sleep(300)   # every 5 minutes

Two cautions decide whether this stays healthy: poll on a sane interval, since every few minutes is plenty for most communities and hammering new every few seconds buys you nothing because posts do not arrive that fast. Cache the IDs you have seen so you fetch each feed once and only write the new posts, which keeps both your storage and your call count down. On the official API the same loop has to stay inside the 100-queries-per-minute budget across every subreddit you watch, so a long watch list pushes you toward the managed route where the rate handling is not your problem. If your interest is the discussion rather than the post list, the next step from a monitored feed is scraping each new post’s comment thread as it appears.

Is it legal to scrape subreddits?

Scraping publicly visible subreddit listings is generally treated as lawful in the United States, on the same footing as scraping other public pages. The Ninth Circuit’s ruling in hiQ v. LinkedIn held that scraping public data likely does not violate the Computer Fraud and Abuse Act, and the EFF’s summary of the hiQ decision walks through the reasoning that a public site has “erected no gates” to bypass.

That is the baseline, not the whole answer. Reddit’s User Agreement prohibits automated access without permission, so the terms are a contract question separate from the CFAA one. Post titles and bodies can also contain personal data, which carries obligations under regimes like the GDPR if you store it, and private or restricted subreddits are off the table entirely.

I walk through robots.txt, the User Agreement, and the case law in is scraping Reddit legal, which is the place to settle the question before you collect at scale.

Pulling a subreddit’s post listing is one piece of the larger job. For the full workflow across posts, comments, threads, and search, the complete guide to scraping Reddit ties the endpoints together.