The Front Desk Review · Crawler policy
FrontDeskReviewBot
If you found this page from your server logs: this is our crawler, and this page tells you exactly what it does and how to make it stop. One email is enough — we don't ask for a reason and we don't argue.
To be excluded: email contact@frontdeskreview.com
from an address at your domain, or add a Disallow for FrontDeskReviewBot to your
robots.txt. Either works. Robots.txt takes effect on our next run; an email is actioned by hand and
we remove the whole host, including anything already collected from it.
What it is
We publish sourced, dated prices and specifications so buyers can compare them. To keep those figures honest we re-check them against the page they came from, and record the date we looked. That re-check is all this crawler does.
It identifies itself on every request as:
FrontDeskReviewBot/1.0 (+https://frontdeskreview.com/bot/)How it behaves
- It honours robots.txt. Checked per host before fetching, for our own token and for
*. ADisallowthat covers the path means we don't request it. - It takes one page at a time, rarely. Our sources are spread across thousands of domains, so a given host typically sees a request or two per day. We are not a load problem.
- It reads, it doesn't act. No forms, no logins, no carts, no accounts. Only pages that are public to anyone with the link.
- It takes the price and the specs — the facts on the page — and links back to your page as the source, with the date we captured it.
- It does not work around a "no". No rotating agents, no pretending to be a browser, no defeating anti-bot measures. If you block us, we're blocked and we stay blocked.
What we publish from your page
A price, spec values, and a link to you, each stamped with the day we read it. We don't copy your page, your description, your photography or your layout, and we don't reproduce your content at length. If you believe we have published something of yours that goes beyond that, tell us at contact@frontdeskreview.com and we will remove it — the removal happens first, any discussion after.
Our accuracy standard · Corrections log · Open data · How we make money