How we scrape
Last updated 6 September 2026. Written for the retailers we read.
Running Shoe Tracker reads publicly listed prices and stock from running-shoe retailers and shows them side by side, with a link out to the store. We think this is good for you — it is free traffic to whoever is cheapest that day — but it is your website, so here is exactly what we do to it and how to make us stop.
What we do
- We read public product pages only: the same pages any shopper sees,
plus the JSON endpoints your platform serves them from and your
sitemap.xml. - We never create accounts, log in, add to a cart, start a checkout, submit a form, or touch anything behind authentication.
- We do not attempt to defeat a block. If you rate-limit us we back off; if you deny us, we stop.
- We keep prices, stock and product details. We copy your product photos' URLs to show a thumbnail; we do not re-host them.
- We link to your product page for the sale. Every purchase is yours.
How hard we hit you
Not hard, and it is enforced in code rather than promised:
- One request at a time per store, with at least 60 seconds between requests — every request, page or data endpoint, one floor for all of them — plus a random extra delay on top. We never open parallel connections to one store.
- Everything is cached for the day. A given URL is fetched at most once per day no matter how many times our own code asks for it.
- We do not retry a refusal. A 429, a 403 or a server error ends that request; we move on to the next one and try again no sooner than the next run. There is no retry loop to push back against you.
- At one request a minute we are slower than a single person browsing your site, and our run is long rather than heavy — the pace is the point.
robots.txt
Our catalog crawl — the broad pass that walks your sitemap to learn what you
carry — reads and obeys your robots.txt. A path you disallow is not
crawled, and a Crawl-delay you set is honoured if it is longer than
the interval above. If your robots.txt is unreachable we treat that
as "not permitted" and skip the crawl rather than guessing.
There is one exception and we would rather write it down than have you find
it. A page sometimes gets what it shows from a separate host the same
store runs — its stock, or the list of products a shopper browses through
— and that host may publish no robots.txt we can read: no
file, or an error, every time we ask. Where the store's own site has already
permitted us, we read those endpoints: they are the same public pages, fetched
the way the pages themselves fetch them. If such a host does publish a file and
it disallows us, we stop, per endpoint. The opt-out below covers it either
way.
We should be straightforward about the limit of that: we send an ordinary browser User-Agent, so a rule aimed at a named bot will not match us. The reliable way to be excluded is to ask, below — we act on it the same day.
Opting out
If you would rather we did not include your store, email hello@running-shoe-tracker.com from an address at your domain and say so. No justification needed and no argument back — we will remove your store from the site, stop all requests to it, and confirm when it is done.
The same address is the fastest way to reach us about anything else: a page we are parsing wrong, a price displayed incorrectly, a photo you want taken down, or a rate that is still too high for your infrastructure. We would much rather hear from you than be blocked.