The Front Desk Review · Crawler policy
FrontDeskReviewBot
If you found this page from your server logs: this is our crawler, and this page tells you exactly what it does and how to make it stop. One email is enough — we don't ask for a reason and we don't argue.
To be excluded: email [email protected]
from an address at your domain, or add a Disallow for FrontDeskReviewBot to your
robots.txt. Either works. Robots.txt takes effect on our next run; an email is actioned by hand and
we remove the whole host, including anything already collected from it.
What it is
We publish sourced, dated prices and specifications so buyers can compare them. To keep those figures honest we re-check them against the page they came from, and record the date we looked. That re-check is all this crawler does.
It identifies itself on every request as:
FrontDeskReviewBot/1.0 (+https://frontdeskreview.com/bot/)How it behaves
- It honours robots.txt. Checked per host before fetching, for our own token and for
*. ADisallowthat covers the path means we don't request it. - It takes one page at a time, rarely. Our sources are spread across thousands of domains, so a given host typically sees a request or two per day. We are not a load problem.
- It reads, it doesn't act. No forms, no logins, no carts, no accounts. Only pages that are public to anyone with the link.
- It takes the price and the specs — the facts on the page — and links back to your page as the source, with the date we captured it.
- It does not work around a "no". No rotating agents, no pretending to be a browser, no defeating anti-bot measures. If you block us, we're blocked and we stay blocked.
- It signs its requests. A name in a header is only a claim, and anyone can copy one. Every request we make carries an Ed25519 signature over the host we are calling, so you can check that it really came from us — see below.
How to verify it is really us
Our requests carry Signature, Signature-Input and Signature-Agent
headers, following RFC 9421
and the Web Bot Auth profile. The public key is published here:
/.well-known/http-message-signatures-directory
The signature covers the authority of the request, so one captured from your logs cannot be replayed against anyone else, and it expires a minute after it is made. If a request claims to be us and does not verify against that key, it is not us — and we would want to know.
What we publish from your page
A price, spec values, and a link to you, each stamped with the day we read it. We don't copy your page, your description, your photography or your layout, and we don't reproduce your content at length. If you believe we have published something of yours that goes beyond that, tell us at [email protected] and we will remove it — the removal happens first, any discussion after.
Our accuracy standard · Corrections log · Open data · How we make money