Methodology
Live checks, run on every visit
Every time someone opens a domain report, StatsMyWeb runs a set of live checks against that domain. Nothing is cached: what you see is what the domain and the public sources answered at that moment. The checks run for any domain, whether or not StatsMyWeb holds any other data about it.
| Check | What is shown | Where it comes from |
|---|---|---|
| DNS records | A, AAAA, MX and NS records | DNS-over-HTTPS |
| HTTPS reachability | Whether the domain answers, and with what status code | One GET request to https://{domain}/, following redirects |
| Response headers | Server, Content-Type, Cache-Control, Content-Encoding and six security headers | The response to that same request |
| Redirect chain | Each hop from the starting address to the final one | That same request |
| WHOIS data | Registrar, registration, expiration and last-changed dates, status codes, nameservers | RDAP, through the rdap.org bootstrap |
| SSL certificate | Common name, issuer, valid from, valid until | Certificate Transparency logs, through crt.sh |
| Meta tags | Title, description, canonical, robots, generator, Open Graph and Twitter card tags | The fetched page |
| robots.txt | The file as the domain serves it | The domain |
| Technology | Software the site appears to use | Signals in headers, cookies and HTML |
| Preview | A live view of the site, or a screenshot | The site, or a screenshot rendered on demand |
What an empty section means
Each check runs on its own. A slow or broken source empties its own section of the report and leaves the rest of the page in place. An empty section can therefore mean that the data does not exist, or that its source was slow or failing at that moment. Reloading the page runs every check again.
What the response headers show
The headers come from the response to the same GET request used for the reachability check. Besides four general headers, the report looks for six security headers: Content-Security-Policy, Strict-Transport-Security, X-Frame-Options, X-Content-Type-Options, Referrer-Policy and Permissions-Policy.
Each one is shown as sent or absent. StatsMyWeb does not grade them or add them up into a score, because whether a site needs a given header depends on what the site does. The guide to HTTP security headers explains what each one controls and how to read them yourself.
Where the WHOIS data comes from
RDAP, the Registration Data Access Protocol, is the IETF’s JSON-based successor to WHOIS. StatsMyWeb queries it through the rdap.org bootstrap, which points each request to the RDAP service responsible for the domain. The report shows the registrar, the registration, expiration and last-changed dates, the status codes and the nameservers. Not every top-level domain offers RDAP yet, so for some domains this section may stay empty.
Why the SSL certificate can differ from the one the server presents
The certificate shown is not read from a TLS connection to the server. It comes from the Certificate Transparency logs, the public records in which certificate authorities log the certificates they issue, queried through crt.sh. StatsMyWeb shows the most recently issued certificate that is currently valid and was logged for that exact domain. If there is none, it falls back to a certificate that covers the domain alongside other names.
A server is free to keep presenting an older certificate that is still valid, or a different one altogether. To see the certificate a server presents right now, open a TLS connection yourself:
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
| openssl x509 -noout -subject -issuer -datesHow technology detection works
Technology detection is best-effort. It looks for a small set of well-known signals in the response headers, the cookies and the HTML of the fetched page. A site that removes or hides those signals shows fewer technologies, or none. The list may be incomplete, and a technology missing from it does not mean the site does not use it.
How the preview is shown
When a site allows other pages to frame it, the report shows it in a sandboxed live frame. A site can forbid framing with X-Frame-Options or with the frame-ancestors directive of Content-Security-Policy. In that case the report shows a screenshot, rendered when you ask for it.
Connectivity rank
Where it exists, a domain report also shows a connectivity rank: how connected the domain is to the rest of the web. It is computed from Common Crawl’s open hyperlink graph, a periodic, independently published snapshot of how pages across the web link to each other, reduced to links between domains.
The figure is the domain’s position by harmonic centrality, out of every domain in that snapshot. Harmonic centrality rewards a domain that many other domains can reach in few links. A lower position number means a more connected domain. It says nothing about the quality or accuracy of a site’s content, and nothing about its traffic. The guide to the connectivity rank works through how it is computed and how to read it.
The current site runs on the cc-main-2026-jul-aug-sep snapshot: 133,241,980 domains.
Web Graph Explorer
The Web Graph Explorer at /explore/{domain}/ shows a domain’s neighbours in the Common Crawl hyperlink graph: the domains that link to it (incoming) and the domains it links to (outgoing).
Graph data exists for a curated set of about 10,000 seed domains. Any other domain gains it once someone explores it, usually within about a day. Neighbour lists are capped. When a list is cut, it is marked as “N of total”, so you can see how much was left out.
Related domains
The domain report also lists related domains: its neighbours in the graph, and the domains ranked just above and just below it by connectivity rank. Both come from the Common Crawl data, not from the live checks, so they appear only where that data exists.
Which pages search engines can index
A domain report is open to search engines only when the domain has a connectivity rank. A Web Graph Explorer page is open to them only when its graph data exists. Every other page still works for anyone who opens it; it is kept out of search results, because a page made only of live checks shows facts anyone could show the same way, and earns no place in search results on its own.
What is not covered yet
StatsMyWeb does not yet have connectivity data for every domain that exists. The underlying graph is large, and importing all of it is a bigger undertaking than a first version needs. A domain without connectivity data still gets a full domain report with its live checks. As more of the graph is imported, more domains gain a rank.
StatsMyWeb does not run its own web crawler. The live checks request a domain’s pages only when someone opens its report. The report shows a page’s meta tags and a preview, not its text, and StatsMyWeb is not affiliated with any domain it shows data for.