About our crawler
If you saw linux-statistics-bot in your server logs, this page explains what it does.
What it fetches
The bot collects a small number of public pages and data files to build the statistics on this site: distribution popularity, desktop market share, repository activity, structured facts about each distribution and article page views. It reads only public URLs and never logs in, submits forms or follows links beyond the specific pages it needs.
How it behaves
- One request per source per run; runs are daily, weekly or monthly depending on how often the source changes.
- Fetches a fixed, hand-maintained list of URLs; it does not discover or follow links. A
403or429stops the run at once; it never retries those or changes its User-Agent. - Retries server errors a couple of times with backoff, then gives up and keeps the last good data.
- Fetches from a single, fixed IP address; happy to share it on request.
Blocking it
Return 403 to the linux-statistics-bot User-Agent and it stops immediately. Or email us and we'll take your site off the
fetch list by hand; the list is short and curated, so this is quick.
Contact and source
- Email: hello@linuxstatistics.com
- Source code (the fetcher is
apps/worker/src/http.ts): GitHub