ownread beta

The Downread Feed Fetcher

Downread is an RSS reader. When someone subscribes to a feed in Downread, or just reads it, our servers download that feed regularly so new posts show up in their reader. The program doing that is the Downread feed fetcher. This page explains what it does, why it requests what it requests, and how to reach us if it causes you trouble.

The fetcher is not a crawler. It doesn't follow links, doesn't index websites, and only fetches feeds that a person added to Downread.

How it identifies itself

Every request the fetcher sends carries this user agent:

Downread/1.0 (+https://downread.com/feed-fetcher.html)

When we fetch a feed on its regular schedule, the user agent also tells you how many Downread users subscribe to it, the way Feedbin and Overcast do:

Downread/1.0 (+https://downread.com/feed-fetcher.html; 42 subscribers; feed-id=f01a2b3c4d)

The word is always subscribers , even for one, so your statistics can simply match (\d+) subscribers. For a public feed, the number includes one automatic subscriber for RSS Directory, which lists public feeds through Downread. For a private feed, it's the real number. People who read a feed in Downread without subscribing aren't counted.

feed-id is our ID of the feed. If your feed is reachable under several addresses, it tells you which requests belong to the same Downread feed, so you don't count its subscribers twice. Only public feeds have it. Private feeds leave it out, because their address can contain a personal token.

Everything else carries the user agent without a count: requests made while someone adds a feed, icons, images, WebSub subscriptions, and a short HEAD request that checks whether the www. and non-www. addresses of a site serve the same feed. Retries through proxies (see When a request is refused) send the same user agent as the first attempt. A proxy changes the address a request comes from, never what it says it is.

The one exception is YouTube, which shows non-browsers a consent page instead of the channel. Requests to YouTube use a browser user agent.

Requests are also signed with Web Bot Auth (HTTP Message Signatures), so you can verify they really come from us. Our public key is at https://downread.com/.well-known/http-message-signatures-directory.

We don't publish a list of IP addresses. Public feeds are fetched from our server in the US. Private feeds of users in the EU are fetched from our server in France. Retries go through proxies with changing addresses.

One request for all readers

Every feed is fetched once, no matter how many people follow it. A feed with a thousand Downread readers or subscribers costs you the same as a feed with one. That's also true for private feeds (feeds with a personal token in the URL): everyone who has the same URL shares a single copy.

How often we fetch

There's no fixed interval. The fetcher looks at how often a feed publishes and whether anyone has actually read it lately:

Some rules always apply:

Requests we try to avoid

What else we request

Besides the feed itself, we occasionally fetch:

Images inside posts are not downloaded by us. They load straight from your server when someone reads the post, so those requests come from readers' browsers.

What Downread does with your feed

Posts appear as your feed delivers them: full text if your feed has full text, a summary if it only has summaries. Every post links back to the original on your site. We don't add ads, and we don't change what your content says. We do remove scripts and other unsafe markup, and we adjust the styling so posts look their best in the reader. For some popular feeds we add hand-crafted style adjustments made for that feed, only so its posts look perfect in our reader. When a new public feed is added, an AI model sorts it into a category based on its title, description and recent posts. We don't train any AI models on your content.

Our catalog lists public feeds so other readers can find them. If you'd like to control how your feed is presented there and in RSS Directory, our public feed directory, you can claim the listing on RSS Directory. You can also hide it from the catalog completely.

robots.txt

The fetcher doesn't read robots.txt. It's a feed reader, not a crawler: it requests a feed because a person reads or subscribed to it, the same way a desktop feed reader would. robots.txt is designed to guide crawlers that discover pages on their own, which the fetcher never does. A published feed exists so that feed readers can read it.

When a request is refused

Many sites block automated requests with generic rules: web application firewalls, bot protection, or blocks for whole data-center IP ranges. Usually those rules aren't meant for feed readers, and feeds suffer from them only by accident. So when your server refuses a request with 403, 429 or a typical CDN error (502 to 504, 520 to 526), we try again from other addresses:

  1. Through proxies on Cloudflare Workers and our own proxy server.
  2. As a last resort, only after another 403 or 429 , through a paid residential proxy service. This step is limited to one attempt per feed per day.

The retries send the same user agent and headers as the first request. Feeds that only work through the residential proxy are fetched at most once a day, and Downread shows readers that the feed updates with a delay.

If you block our user agent on purpose, the retries are refused as well. We'd still check the feed on its normal schedule, retries included. If you want us to stop, answer with 410 Gone or tell us (see Asking us to stop).

When a feed fails

When a feed returns errors, we keep trying on its normal schedule, since most errors are temporary. 404 Not Found counts as temporary too: we only give up after a long series of failures.

410 Gone is different. It says the feed was removed for good, so we stop fetching it right away.

When someone tries to add a feed that can't be fetched at all, we don't create it in the first place, and we remember that the address failed.

Asking us to stop

The quickest way to make Downread stop fetching your feed is to answer our requests with 410 Gone. We stop at the next attempt, and nobody needs to read an email.

You can also email with the feed's address, or use the Report button on the feed in Downread and choose "I'm the owner". We'll then block the feed: fetching stops, and it disappears from Downread, its catalog and search.

The same address works for anything else: problems with the fetcher, questions about this page, or a request we've handled wrong.