The Crosswire fetcher
If you found this page in your access logs, you are the operator of one of the 907 news outlets Crosswire lists, and something here requested your home page. This page explains exactly what it asks for, how often, what happens to the response — and the single line of robots.txt that stops it. If anything below does not match what you are seeing, write to [email protected] and we will fix it.
What Crosswire is
Crosswire puts the live front pages of news outlets side by side, so a reader can see how the same morning is presented in different countries and across the political spectrum. Each pane is your own home page, fetched server-side and shown in a frame with your outlet's name above it; every link in it opens on your site. There is no advertising on Crosswire, no paywall around publisher content, no archive, and no extraction of your text into a database. The methodology covers how outlets are chosen and labelled.
Only one URL per outlet is ever requested: the home page listed in the public dataset. Crosswire does not follow links, does not fetch article pages, and does not download your images, stylesheets or scripts server-side — the reader's browser loads those from you directly.
The two agents you may see
Reader traffic. Someone opened a comparison that includes your outlet, and this is that one page, fetched at that moment. It is on demand and one-at-a-time, not a crawl. So that it renders the way a visitor's browser would, the request is shaped like Chrome, with our own identity appended:
Mozilla/5.0 (…) Chrome/124.0.0.0 Safari/537.36 Crosswire/1.0 (+https://crosswire.online/bot)The weekly sweep. Once a week an automated job requests every outlet's home page once, to find the ones that have stopped rendering so we can stop putting them in front of readers. This is the only systematic crawling Crosswire does, and it declares itself plainly:
Crosswire-Audit/1.0 (+https://crosswire.online/bot)Both requests also carry an x-crosswire-contact header pointing back at this page, so a request from us is identifiable in your logs even if the User-Agent has been rewritten by a proxy in between.
How to block it
One rule in your robots.txt stops everything Crosswire sends. Readers who open a comparison containing your outlet will see a card explaining that the publisher asked not to be previewed, with a link to your site.
User-agent: Crosswire
Disallow: /If your objection is only to the scheduled crawl and you are happy for readers to keep opening your front page on demand, block the sweep on its own — the token is matched by prefix, so the rule above covers this one but not the reverse:
User-agent: Crosswire-Audit
Disallow: /Prefer not to touch robots.txt, or want the outlet delisted entirely? Email [email protected] from an address at your domain and we will remove it from the dataset. No reason needed, and no argument from us.
How robots.txt is applied, and where it isn't
The scheduled sweep obeys robots.txt in full, as any crawler must: a blanket User-agent: * disallow stops it, an unreachable robots.txt stops it, and a Crawl-delay is respected. Outlets it may not crawl are skipped without a single request reaching your servers.
Reader traffic is treated differently, and it is worth being direct about why. A person clicking to see today's front page is not a crawl: it is one page, fetched once, because a human asked for it that second — the same class of request as their browser making it, or a reader-mode view, or a translation proxy, none of which consult robots.txt either. So a blanket anti-crawler rule does not cancel a reader's request. A rule naming Crosswire does, every time — that is not a wide net we happened to fall into, it is you telling this project specifically to stop, and there is no reading of that where ignoring it would be defensible.
What we do to keep the load off you
Conditional requests. Every fetch stores your ETag and Last-Modified and sends them back on the next one. When your server answers 304 it sends no body at all, and we serve the copy we already had.
Edge caching. A successfully fetched front page is cached for three minutes at each Cloudflare location and held for two hours as a fallback, so a hundred readers opening the same comparison do not become a hundred requests to you — and a brief outage on your side shows readers your last front page rather than an error.
Backoff. An HTTP 429, or a 503 carrying Retry-After, stops requests to your host entirely for as long as you asked — and if you did not name an interval, for a minute, then two, then four, up to an hour. While your site is resting we do not call at all; readers get the cached copy. The clock resets the first time you answer normally again.
One thing we would rather you heard from us
When a site refuses the reader request outright, Crosswire retries it twice more, presenting the request the way a search crawler and then a social link-preview fetcher would — the tokens most bot-walls are configured to let through so that headlines unfurl on other platforms. It is a workaround for a wall, and we would rather be straightforward about it than have you discover it in a log.
Those retries only ever happen after a refusal, only ever for the one home-page URL a reader asked for, never during the scheduled sweep, and never for a publisher whose robots.txt names us. If you would rather they did not happen at all, the robots.txt rule above is honoured before any of it starts — the request is not made.
Questions publishers ask
How often does Crosswire request my front page?+
Reader traffic is on demand and unpredictable by nature, but each successful fetch is cached at Cloudflare's edge for three minutes and revalidated with a conditional request for fifteen, so a burst of readers becomes a small number of origin hits. The scheduled sweep requests your home page once a week, twice at most if the first attempt fails.
Do you crawl anything other than the home page?+
No. Crosswire only ever requests the single home-page URL listed for an outlet in its public dataset. It does not follow links, does not fetch article pages, and does not download assets server-side — images, stylesheets and scripts are loaded by the reader's own browser directly from your servers, exactly as if they had typed your address.
Do you store or republish my content?+
The only copy kept is a two-hour cache at the edge, used so a reader sees your front page rather than an error card if your site is briefly unreachable. Nothing is archived, no text is extracted into a database, and no content is republished anywhere on Crosswire. Every pane is labelled with the outlet's name and links out to your site.
Is there advertising or payment around my page?+
No. Crosswire carries no advertising anywhere, sells no access to publisher content, and has no commercial relationship with any outlet it displays. Ad slots on your own front page render normally inside the pane.
I blocked Crosswire but readers can still open my site from your pages.+
That is intended. Blocking the fetcher stops Crosswire from requesting your pages; it does not remove your outlet from the directory, which is a list of links and factual metadata. Every profile page continues to link to you. If you want the listing itself removed, email us and we will remove it.
How quickly does a robots.txt change take effect?+
Within a day. A fetched robots.txt is cached for 24 hours per Cloudflare location, and a change that removes access is honoured as soon as that copy expires. If you need it to take effect immediately, email us and we will remove the outlet from the dataset by hand.
Anything else — a rule that isn't being honoured, a listing to correct, a request to be removed? Email [email protected].