> Markdown version of https://downread.com/feed-fetcher.html. Downread is a free RSS reader for people: https://downread.com/

# The Downread Feed Fetcher

Downread is an RSS reader. When someone subscribes to a feed in Downread, or just reads it, our servers download
that feed regularly so new posts show up in their reader. The program doing that is the Downread
feed fetcher. This page explains what it does, why it requests what it requests, and how to reach
us if it causes you trouble.

The fetcher is not a crawler. It doesn't follow links, doesn't index websites, and only fetches
feeds that a person added to Downread.

## How it identifies itself

Every request the fetcher sends carries this user agent:

`Downread/1.0 (+https://downread.com/feed-fetcher.html)`

When we fetch a feed on its regular schedule, the user agent also tells you how many Downread users
subscribe to it, the way Feedbin and Overcast do:

`Downread/1.0 (+https://downread.com/feed-fetcher.html; 42 subscribers; feed-id=f01a2b3c4d)`

The word is always `subscribers`, even for one, so your statistics can simply match
`(\d+) subscribers`. For a public feed, the number includes one automatic subscriber for
[RSS Directory](https://rssdirectory.org/), which lists public feeds through Downread. For a private
feed, it's the real number. People who read a feed in Downread without subscribing aren't counted.

`feed-id` is our ID of the feed. If your feed is reachable under several addresses, it tells you
which requests belong to the same Downread feed, so you don't count its subscribers twice. Only
public feeds have it. Private feeds leave it out, because their address can contain a personal
token.

Everything else carries the user agent without a count: requests made while someone adds a
feed, icons, images, WebSub subscriptions, and a short `HEAD` request that checks whether the
`www.` and non-`www.` addresses of a site serve the same feed. Retries through proxies (see *When
a request is refused*) send the same user agent as the first attempt. A proxy changes the address
a request comes from, never what it says it is.

The one exception is YouTube, which shows non-browsers a consent page instead of the channel.
Requests to YouTube use a browser user agent.

Requests are also signed with [Web Bot Auth](https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/)
(HTTP Message Signatures), so you can verify they really come from us. Our public key is at
`https://downread.com/.well-known/http-message-signatures-directory`.

We don't publish a list of IP addresses. Public feeds are fetched from our server in the US.
Private feeds of users in the EU are fetched from our server in France. Retries go through proxies
with changing addresses.

## One request for all readers

Every feed is fetched once, no matter how many people follow it. A feed with a thousand Downread
readers or subscribers costs you the same as a feed with one. That's also true for private feeds (feeds with
a personal token in the URL): everyone who has the same URL shares a single copy.

## How often we fetch

There's no fixed interval. The fetcher looks at how often a feed publishes and whether anyone has
actually read it lately:

- Busy feeds that many people read: every 15 to 60 minutes.
- Feeds that post less often, or that fewer people read: every few hours.
- Feeds that nobody has opened recently: every few hours up to once every two weeks, depending
  on how long ago they last published.

Some rules always apply:

- **Never more than every 10 minutes**, except when your WebSub hub tells us about a new post.
- **`Retry-After` is honored** on `429 Too Many Requests`. Without it, we wait an hour.
- **`Cache-Control: max-age`** counts as a minimum interval, up to one hour.
- **`skipHours` and `skipDays`** in RSS feeds are honored.

## Requests we try to avoid

- **Conditional requests.** We send `If-None-Match` and `If-Modified-Since` whenever your server gave
  us an `ETag` or `Last-Modified`, so an unchanged feed costs a `304` and no body.
- **Compression.** We accept gzip, deflate and Brotli.
- **WebSub (PubSubHubbub).** If your feed names a hub, we subscribe to it. While the subscription
  is active, we fetch about a minute after each notification and otherwise poll only every few days,
  just in case a notification gets lost. If notifications stop arriving for new posts, we poll
  more often again for a while.
- **Redirects.** When a public feed redirects to a new address, we store the new address and stop
  requesting the old one. Today that happens for temporary redirects too.

## What else we request

Besides the feed itself, we occasionally fetch:

- **Icons.** When a feed is added, we load the site's homepage to find its icon (`<link rel="icon">`,
  `/favicon.ico`, or an `og:image` as a last resort), plus the feed's own `<image>`. We repeat this
  on about one fetch in a hundred, so changed icons show up eventually. Icons are scaled down and
  cached on our own servers, so your site doesn't serve them to our readers.
- **Feed discovery.** If someone enters a web page instead of a feed URL, we read that page to find
  the feed it links to. Sometimes that includes a few candidate pages or the site's root URL. This
  explains requests for addresses that don't exist on your site: we're trying out what someone
  typed or what a page links to.

Images inside posts are not downloaded by us. They load straight from your server when someone
reads the post, so those requests come from readers' browsers.

## What Downread does with your feed

Posts appear as your feed delivers them: full text if your feed has full text, a summary if it only
has summaries. Every post links back to the original on your site. We don't add ads, and we don't
change what your content says. We do remove scripts and other unsafe markup, and we adjust the
styling so posts look their best in the reader. For some popular feeds we add hand-crafted style
adjustments made for that feed, only so its posts look perfect in our reader. When a new public feed is
added, an AI model sorts it into a category based on its title, description and recent posts. We
don't train any AI models on your content.

Our catalog lists public feeds so other readers can find them. If you'd like to control how your
feed is presented there and in [RSS Directory](https://rssdirectory.org/), our public feed
directory, you can claim the listing on RSS Directory. You can also hide it from the catalog
completely.

## robots.txt

The fetcher doesn't read robots.txt. It's a feed reader, not a crawler: it requests a feed because
a person reads or subscribed to it, the same way a desktop feed reader would. robots.txt is designed to
guide crawlers that discover pages on their own, which the fetcher never does. A published feed
exists so that feed readers can read it.

## When a request is refused

Many sites block automated requests with generic rules: web application firewalls, bot protection,
or blocks for whole data-center IP ranges. Usually those rules aren't meant for feed readers, and
feeds suffer from them only by accident. So when your server refuses a request with `403`, `429` or a
typical CDN error (`502` to `504`, `520` to `526`), we try again from other addresses:

1. Through proxies on Cloudflare Workers and our own proxy server.
2. As a last resort, only after another `403` or `429`, through a paid residential proxy service.
   This step is limited to one attempt per feed per day.

The retries send the same user agent and headers as the first request. Feeds that only work through
the residential proxy are fetched at most once a day, and Downread shows readers that the feed
updates with a delay.

If you block our user agent on purpose, the retries are refused as well. We'd still check the feed
on its normal schedule, retries included. If you want us to stop, answer with `410 Gone` or tell us
(see *Asking us to stop*).

## When a feed fails

When a feed returns errors, we keep trying on its normal schedule, since most errors are
temporary. `404 Not Found` counts as temporary too: we only give up after a long series of failures.

`410 Gone` is different. It says the feed was removed for good, so we stop fetching it right away.

When someone tries to add a feed that can't be fetched at all, we don't create it in the first
place, and we remember that the address failed.

## Asking us to stop

The quickest way to make Downread stop fetching your feed is to answer our requests with
`410 Gone`. We stop at the next attempt, and nobody needs to read an email.

You can also email **(email address on the web page)** with the feed's address, or use the Report button on the
feed in Downread and choose "I'm the owner". We'll then block the feed: fetching stops, and it
disappears from Downread, its catalog and search.

The same address works for anything else: problems with the fetcher, questions about this page, or
a request we've handled wrong.