Reddit’s API is effectively dead for archival. Third-party apps are gone. Reddit has threatened to cut off access to the Pushshift dataset multiple times. But 3.28TB of Reddit history exists as a torrent right now, and I built a tool to turn it into something you can browse on your own hardware.
The key point: This doesn’t touch Reddit’s servers. Ever. Download the Pushshift dataset, run my tool locally, get a fully browsable archive. Works on an air-gapped machine. Works on a Raspberry Pi serving your LAN. Works on a USB drive you hand to someone.
What it does: Takes compressed data dumps from Reddit (.zst), Voat (SQL), and Ruqqus (.7z) and generates static HTML. No JavaScript, no external requests, no tracking. Open index.html and browse. Want search? Run the optional Docker stack with PostgreSQL – still entirely on your machine.
API & AI Integration: Full REST API with 30+ endpoints – posts, comments, users, subreddits, full-text search, aggregations. Also ships with an MCP server (29 tools) so you can query your archive directly from AI tools.
Self-hosting options:
- USB drive / local folder (just open the HTML files)
- Home server on your LAN
- Tor hidden service (2 commands, no port forwarding needed)
- VPS with HTTPS
- GitHub Pages for small archives
Why this matters: Once you have the data, you own it. No API keys, no rate limits, no ToS changes can take it away.
Scale: Tens of millions of posts per instance. PostgreSQL backend keeps memory constant regardless of dataset size. For the full 2.38B post dataset, run multiple instances by topic.
How I built it: Python, PostgreSQL, Jinja2 templates, Docker. Used Claude Code throughout as an experiment in AI-assisted development. Learned that the workflow is “trust but verify” – it accelerates the boring parts but you still own the architecture.
Live demo: https://online-archives.github.io/redd-archiver-example/ GitHub: https://github.com/19-84/redd-archiver (Public Domain)
Pushshift torrent: https://academictorrents.com/details/1614740ac8c94505e4ecb9d88be8bed7b6afddd4
Would love to see you learn an entire foreign language just so you are able to communicate with the world without being laughed at by people as hostile as yourself.
I mean I can’t see what the comment was and I’m assuming it must’ve been downright hateful, but that person almost certainly has learned a foreign language just to communicate with the world and in fact had to learn another foreign language in school because their name is Estonian for “gypsy” and learning two foreign languages (usually English and Russian, sometimes German or something else for the second foreign language) is required. Likely they speak 2.5 languages as is common here (my German is so bad I count it as half a language - native speakers speak too fast for me, but I can kinda get my point across if needed), but could be more.
Just pointing out that even when trying to be accepting of others, subtle anglo-defaultism can show up in your comment, not that I necessarily agree with whatever the comment was.
I can’t even learn my own language!
They said it wasn’t their “first” lanugage. Which leads me to believe that they do speak English. If that’s the case, then they indeed are kind of lazy. There have already been studies in the impact of AI when used for communication and the results are not positive.
This isn’t something I’d personally point out and criticize, just something I wouldn’t do personally. Take the time to express your own ideas in your own words. The long term cost is higher than the short term gains.
Hey I drove to the library, picked up all these things you needed, got dinner here ya go, free!
You drove? man that’s lazy…
He used AI to clean up translation and save time after he spent a fuck ton of time curating and delivering us a helpful product. Calling him out as lazy is an awful take.
there are the so called activists that complain alot then there are the activists that deliver projects and code… enough said
“Activists”? What are you even talking about?
Regardless, I specifically said that what you did wasn’t wrong or anything likw that. I simply think that it’s going to do you more harm than good in the long run. You’re free to do whatever you want though, obviously.
Another piece of advice. When someone simply shares an opinion, don’t get instantly butthurt over nothing. Otherwise this might as well be reddit.
First, that’s an awful analogy.
Second, you’re assuming (for some unknown reason) that they “cleaned up” the “translation” using ai. You have literally no idea exactly how they wrote the post. It’s kinda weird to make up a random scenario but ok.
Third, no, it’s not an awful take. You can code something that requires a ton of effort but write awful documentation. One thing does not make the other impossible.
Fourth, I already explained that there have already been studies that concluded that using AI to write stuff for you has a negative impact on your communication skills. This is not an opinion or me being ingrateful or whatever. I was just sharing information.
If that documentation was awful, I’d REALLY like to see your take on NixOS :)
I have A1 and A2 level in a couple of non-first languages, technically I can speak those, realistically I don’t and will not be able to communicate something more complex than ‘here, take a look’
So I don’t agree with your absolutistic stance
There’s nothing “absolutistic” about my “stance”. If you’re rusty using a language, you won’t get better if someone else does the homework for you. Make an effort, make mistakes, write in a way that sounds weird, who cares. But practice. If you only take the easy way out, that’ll be your only option in the future.
Although, like I already said, that’s MY way of thinking about it. If you want to use ai to write your stuff, you do you. It doesn’t negate the fact that, whle it’s not “wrong”, it’s the lazy (or minimum effort) option. Don’t know why it bothers you so much.