About & methodology

Website Insights publishes free, weekly reports on the world's most popular websites. Everything is built from public data, and the code and data are open source.

Popularity rank

Ranks come from the Tranco list, a research ranking designed to be hard to manipulate. It combines several sources (Chrome UX Report, Cloudflare Radar, Cisco Umbrella, Majestic and Farsight) and averages them over 30 days. We take the top 10,000 domains of the latest daily list each week and keep a weekly history. Tranco ranks registrable domains, so mail.google.com counts as google.com.

Citation: Victor Le Pochat, Tom Van Goethem, Samaneh Tajalizadehkhoob, Maciej Korczyński and Wouter Joosen. "Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation." NDSS 2019.

Domain registration

Registration, expiry, registrar, name server and DNSSEC data come from each registry's public RDAP service, found through the IANA bootstrap file. Some country registries (for example .de, .cn and .jp) don't offer RDAP, so those sites show no registration date. Records are refreshed every two months.

Technology detection

Our scanner downloads each homepage once a week and matches about 120 technology signatures against the HTML, script and stylesheet URLs, meta tags and HTTP response headers. It doesn't run JavaScript, so tools that are injected later, loaded after a cookie-consent click, or used only on inner pages can be missed, and client-rendered pages may look thin to the SEO checks. The same signatures power the browser extension, which sees the fully rendered page.

Our scanner (WebsiteInsightsBot)

The scanner identifies itself with this user agent:

Mozilla/5.0 (compatible; WebsiteInsightsBot/1.0; +https://jliushi.github.io/website-insights/about/#bot)

Each week it requests /robots.txt, the homepage, and, when robots.txt lists no sitemap, a HEAD request for /sitemap.xml. That is at most four requests per site per week. It obeys robots.txt: if the rules for WebsiteInsightsBot (or for *, when there is no group for us) disallow /, the homepage is not fetched. To opt out, add:

User-agent: WebsiteInsightsBot
Disallow: /

Limitations

• Rank reflects popularity relative to other sites, not an exact visitor count.
• The scanner runs from a US data centre; sites that vary by country, block data-centre traffic or show a challenge page may report less.
• Response times are measured once, from one location, and are indicative only.

Contact

Corrections, removal requests and suggestions: open an issue on GitHub.