My high school and college years were transformed by Google Reader (RIP). It was a golden age: I could follow the intellectual currents of the web and it truly opened the world for me. RSS made the internet feel manageable and cozy: like a periodical tailored just for you, delivered on your doorstep each morning.

We do not live in those times anymore.

Which is a shame because there’s so much going on! It’s not that information is actually scarce: it’s that we’re so flooded with content that finding the good stuff is difficult. These days content online feels more like furtively hoarding scarce nuggets for a long winter.

I had been using Inoreader, which has been serviceable. But nowadays, there’s almost no barrier to trying to roll your own software. You can create bespoke software to address exactly what you want to accomplish in the form that’s most convenient to you. And what’s more important than shaping the information that you consume?

I think I can do better; and now with Claude as my implementer, I have!

The Problem

For instance, the first thing to get rid of is the addictive patterns. Whenever you open the RSS reader, you get some variable reinforcement: maybe there’s a great new article, maybe not, but either way you want to keep checking. And keep checking it. Eventually you end up more excited to check the thing than to read anything! Certainly not good for me.

But now everything is under my control. I just instituted a cutoff: each day I can only read items that arrived by 6 am that morning. This way I can have my groggy morning RSS scan but then not be tempted the rest of the day!

But even with that fixed, you miss some of the benefits of detailed attention on the reading side. The original sin of most RSS readers is that content arrives in chronological order. This gets even worse with the addition of “saved pages” (RIP ReadItLater), which just arrive in chronological order (and not even grouped by source). This leads to a disjointed experience: you’re reading about Bayesian statistics and then rucking and then LLMs and then enjoying some SMBC. It’s not conducive to maintaining focus: just flitting about. It is not an unpleasant state: it is intoxicating and much better than exploring the mire that is non-curated discovery on the internet.

As a reader, I care about topic/context. For instance, there are a number of newsletters I follow specifically to be aware of events happening in San Francisco. I don’t want to let them sit unread too long; otherwise I might miss a fun event. But other feeds are more like a TV season: Ben Recht is liveblogging his forecasting class and, instead of dutifully watching every week, I just want to binge it all at the end. And this cuts across sources: lately, there’s been a lot of discussion about the future of mathematics and AI and I want to get the full context by reading diverse perspectives: so I wait a little while for enough pieces to accumulate.

You can attempt to solve this by adding tags, but that’s just crazy-making. You can try to just read individual blogs at a time, but often you want to incorporate multiple sources. And of course you can try to do the clustering on the fly, but then you just keep churning through the same content again and again, and start to feel bad about the ever lengthening queue.

We need to hand this off to the machine.

The Solution: Clustered Topics

Here’s how my reader works.

(Almost) everything comes into a forage queue: this is the same setup as your usual RSS reader. I look from feed to feed and add to my “saved” set for later reading.

Upon being saved, I start the clustering. Note that my classification is only run once I save an item: otherwise I’m going to be facing an order of magnitude higher costs. Each saved item’s title, source, and snippet is sent to a prompt, which seeks to categorize it within my existing clusters based on their descriptions. For instance, I have:

  • The code review process: how to give or receive review, review culture, and how it’s changing in the AI age. This is meant to inspire some professional work on modernizing code review.
  • Game-playing search algorithms: monte-carlo tree search, game AI, and related tree-search techniques. These should lead to a couple blog posts (stay tuned).
  • Events: invitations/ads for lectures, plays, concerts, shows, or meet-ups. These are not meant to be digested all at once: I want these flagged so I don’t lose them in the mix.

And, of course, I can also place it in no cluster (roughly 50% of my posts are here so far).

I can either make clusters myself or automate the process by sending all of the titles and snippets to an LLM and asking for some suggested clusters, which was helpful when I started (and less so now). If I make a new cluster, I can reclassify all of the uncategorized posts to sweep matches into my new cluster.

The Next Frontier

Of course, if I can automate my judgment for curating clusters, where else can I automate too?

One problem that I’ve had is high-volume feeds like Hacker News or lobste.rs. And they’re addicting: there’s enough good content that pops up that I pavlovianly check their website on my phone. Back in grad school, I would check arXiv, but nowadays there’s just not enough time. But the whole point of RSS is that I don’t have to do that; however, they’re so big that they dominate my entire feed!

So I need a way of only surfacing the relevant articles for me. Now, running an LLM here is going to be more expensive than just running it on the saved pages: these feeds are significantly larger than my saved inputs (mostly from larger arXiv feeds). Which, while not expensive in absolute terms (500 items/day * $0.0013 per item = $20 a month), is still far more than I would care to pay for this feature.

But deciding if something is relevant to my interests (and being okay with some false positives and false negatives) doesn’t require an LLM: I can instead use more lightweight NLP approaches. I just train a classifier over an embedding to predict whether I’ll keep or discard the items. Then I have a separate tab where I can look at these “fire hydrant” feeds and skim from the top for things that are more likely to be relevant to me. This has been working quite well: it’s led me to way more useful arXiv papers than I’d been finding before!