The Front Desk Review · Crawler policy

FrontDeskReviewBot

If you found this page from your server logs: this is our crawler, and this page tells you exactly what it does and how to make it stop. One email is enough — we don't ask for a reason and we don't argue.

To be excluded: email [email protected] from an address at your domain, or add a Disallow for FrontDeskReviewBot to your robots.txt. Either works. Robots.txt takes effect on our next run; an email is actioned by hand and we remove the whole host, including anything already collected from it.

What it is

We publish sourced, dated prices and specifications so buyers can compare them. This crawler re-checks those figures against the pages they came from and records the date it looked. It also finds and reads the pages new figures come from (makers' and sellers' pages, published catalogues, reviewers' feeds and regulators' notices), reads a product page a visitor gives our import tool, reads once a week the claim and the pricing pages of a vendor that has asked us to and the testers' pages its claim names, fetches the product photos we show, and asks a few public services for data our tools use: exchange rates, search-demand signals and a registry listing. Whoever reads a page for what we publish — a scheduled job, one of us, or an AI assistant working for us — reads it through this crawler.

It identifies itself on every request as:

Agent/FrontDeskReviewBot FrontDeskReviewBot/1.0 (+https://frontdeskreview.com/bot/)

How it behaves

  • It honours robots.txt. Checked per host before fetching, for our own token and for *. A Disallow that covers a page means we don't request that page, even when another page redirects to it or shows it in a frame. When we render a page, each file it loads — a script, a style sheet, an image, the data its scripts fetch — is asked about the same way, at every address a redirect sends it to. If your robots.txt cannot be reached at all — no connection, a timeout or a server error — we request nothing from your site on that run, neither the page nor its files; a robots.txt that is missing or refused (a 4xx answer) counts as no rules, as the robots.txt standard, RFC 9309, says. Two kinds of request still carry this name without asking robots.txt: our calls to Amazon's product API, which run under Amazon's terms for that API, and the rare requests a page sends from a web worker or shared worker inside our browser, which do not pass through our check.
  • It takes one page at a time, rarely. Our sources are spread across thousands of domains, so a given host typically sees a request or two per day. We are not a load problem.
  • It reads, it doesn't act. No forms, no logins, no carts, no accounts on your site. Only pages that are public to anyone with the link.
  • It takes the price and the specs — the facts on the page — and links back to your page as the source, with the date we captured it.
  • It does not work around a "no". No rotating agents, no pretending to be a browser, no defeating anti-bot measures. If you block us, we're blocked and we stay blocked.
  • A browser or an archive is not a way around it either. When we render a page in a browser, the browser is shown what our own request for that page was served, and the same goes for a page inside one of its frames; the files those pages load are fetched for them by our crawler, as above, and the WebSockets a page opens are not connected, except one opened from a web worker, the gap described above. We may ask the Internet Archive to keep a copy of a page we cite, which its crawler then fetches under its own name, and we read the archive's older copies of pages we cite to date past prices, and ask it for the day it first captured the page of a product whose maker has asked us to list it. We do none of this for a page that refuses our request or that your robots.txt closes to us.
  • It signs what it can. A name in a header is only a claim, and anyone can copy one. Every page our price checks request from our own server carries an Ed25519 signature over the host being called, so you can check that it really came from us — see below. Some of our requests carry our name without a signature: those of our import tool and our review-feed reader, those made from anywhere but our own server, the files a page we render loads, our requests to web archives, and our calls to Amazon's product API.

Until 2026-10-22

We did not always keep to the rules above.

  • Until 2026-09-01 our scheduled price checks sent a copied browser user agent and did not read robots.txt.
  • Until 2026-09-25 other scripts of ours sent a browser user agent too, and our import tool's page reader did until 2026-10-22. Several did not read robots.txt until 2026-10-22, and several more asked it about the first page of a redirect but not about the pages it led to. When we rendered a page in a browser, the files it loaded and the pages in its frames were requested without asking robots.txt.
  • When a plain request was refused, our checks sometimes tried the page again in a browser, and pages that had refused us were still opened in a browser by our re-reading tool, sent to be read again from another network, and captured through a web archive.
  • People and AI assistants working for us also read pages with tools of their own, under names other than ours, pages for our records among them. Some of those reads sent a standard browser's user agent and headers, including to pages that had refused our automated reads, and some went through a third-party reading service. Several records still say so in their provenance.

We changed this on 2026-10-22. Since then every read of your pages for what we publish — a price, a plan, a specification, a source we cite — goes through this crawler, whether a scheduled job makes it, one of us runs it by hand, or an AI assistant working for us needs the page. It carries the name above, robots.txt is asked about the page, about every page a redirect leads to and about every file a page we render loads (apart from a web worker's requests, as said above), a browser of ours shows only pages our own requests were served, and a page that refuses it is not read by any other means — not in a browser, not through an archive or a reading service, not from another network — until it lets us in. Reading documentation or news for background, never cited as a source for anything we publish, is outside this. The corrections log records the change.

How to verify it is really us

Signed requests carry Signature, Signature-Input and Signature-Agent headers, following RFC 9421 and the Web Bot Auth profile. The public key is published here:

/.well-known/http-message-signatures-directory

The signature covers the authority of the request, so one captured from your logs cannot be replayed against anyone else, and it expires a minute after it is made. A request that carries our signature headers and does not verify against that key is not us — and we would want to know.

What we publish from your page

A price, spec values, and a link to you, each stamped with the day we read it. For a product we may also show one photo of it, stored on our own server; the product's own page credits it to its source. We don't copy your page, your description or your layout, and we don't reproduce your text at length. If you believe we have published something of yours that goes beyond that, tell us at [email protected] and we will remove it — the removal happens first, any discussion after.

Our accuracy standard · Corrections log · Open data · How we make money