How we scrape
Last updated 5 August 2026. Written for the retailers we check.
Shoe Tracker reads publicly listed prices and stock from running-shoe retailers and shows them side by side, with a link out to the store. We think this is good for you — it is free traffic to whoever is cheapest that day — but it is your website, so here is exactly what we do to it and how to make us stop.
What we do
- We read public product pages only: the same pages any shopper sees,
plus the JSON endpoints your platform serves them from and your
sitemap.xml. - We never create accounts, log in, add to a cart, start a checkout, submit a form, or touch anything behind authentication.
- We do not attempt to defeat a block. If you rate-limit us we back off; if you deny us, we stop.
- We keep prices, stock and product details. We copy your product photos' URLs to show a thumbnail; we do not re-host them.
- We link to your product page for the sale. Every purchase is yours.
How hard we hit you
Not hard, and it is enforced in code rather than promised:
- One request at a time per store, with at least 1 second between page requests (0.4 s for a JSON data endpoint, which is cheaper to serve than a rendered page), plus a random extra delay on top. We never open parallel connections to one store.
- Everything is cached for the day. A given URL is fetched at most once per day no matter how many times our own code asks for it.
- We back off when told to. A 429, a 403 or a server error is
retried a few times with exponential backoff, honouring your
Retry-Afterheader, and then abandoned. - The daily run is a few hundred requests spread over minutes — the traffic of one browsing customer, once a day.
robots.txt
Our catalog crawl — the broad pass that walks your sitemap to learn what you
carry — reads and obeys your robots.txt. A path you disallow is not
crawled, and a Crawl-delay you set is honoured if it is longer than
the interval above. If your robots.txt is unreachable we treat that
as "not permitted" and skip the crawl rather than guessing.
We should be straightforward about the limit of that: we send an ordinary browser User-Agent, so a rule aimed at a named bot will not match us. The reliable way to be excluded is to ask, below — we act on it the same day.
Opting out
If you would rather we did not include your store, email hello@running-shoe-tracker.com from an address at your domain and say so. No justification needed and no argument back — we will remove your store from the site, stop all requests to it, and confirm when it is done.
The same address is the fastest way to reach us about anything else: a page we are parsing wrong, a price displayed incorrectly, a photo you want taken down, or a rate that is still too high for your infrastructure. We would much rather hear from you than be blocked.
Who we currently check
The full list is public, on Stores we check.