StatsMyWeb
Main menu

Domain report

crawlpilot.tech

crawlpilot.tech ranks in the top 0.29% of domains by connectivity (Common Crawl, cc-main-2026-jul-aug-sep), with 1 distinct host observed under it.

Explore crawlpilot.tech in the Web Graph

Connectivity

#380,938harmonic-centrality rank
Top percentile
0.29%
Hosts observed
1

From the Common Crawl cc-main-2026-jul-aug-sep hyperlink graph, 133,241,980 domains total. What this rank means

HTTPS reachability

200HTTP status

https://crawlpilot.tech/ answered with HTTP 200.

DNS records

A
  • 64.29.17.1
  • 216.198.79.1
AAAA
No records found.
MX
No records found.
NS
  • ns1.vercel-dns.com.
  • ns2.vercel-dns.com.

Live preview

This is a live, sandboxed embed of crawlpilot.tech — not a copy. What it shows can change any time the site itself does.

Domain registration (WHOIS / RDAP)

RDAP lookup failed (HTTP 429).

How to read WHOIS and RDAP data

SSL certificate

Certificate Transparency lookup failed (HTTP 429).

How to check when a certificate expires

HTTP & security headers

Response

Server
cloudflare
Content-Type
text/html; charset=utf-8
Cache-Control
public, max-age=0, must-revalidate
Content-Encoding
Not sent

Security

Content-Security-Policy
Not sent
Strict-Transport-Security
max-age=63072000
X-Frame-Options
Not sent
X-Content-Type-Options
Not sent
Referrer-Policy
Not sent
Permissions-Policy
Not sent

What each security header does

Redirect chain

  1. 1.http://crawlpilot.tech/HTTP 308

    → https://crawlpilot.tech/

  2. 2.https://crawlpilot.tech/HTTP 200

Meta tags

Title
Crawl Pilot | The Content Web Scraper
Description
The privacy-first, open-source infrastructure for AI-powered web scraping and data extraction. Turn any website into structured data instantly with CrawlPilot.
Canonical
https://crawlpilot.tech
Robots
Not set
Generator
Not set
Open Graph title
CrawlPilot | AI-Powered Web Scraping & Data Extraction Infrastructure
Open Graph description
Turn the web into structured data with the privacy-first, open-source AI scraper.
Open Graph image
Not set
Open Graph type
website
Twitter card
summary_large_image
Twitter title
Crawl Pilot | Content Intelligence & Web Scraping
Twitter image
Not set

robots.txt

Disallow rules (User-agent: *)
5
Disallow rules (other agents)
0

Sitemap declared:

  • https://crawlpilot.tech/sitemap.xml

Technology detected

Best-effort, based on a small set of well-known signals — this may be incomplete.

  • Cloudflarecdn

    Server: cloudflare

  • Next.jsframework

    Found "_next/static/" in the page's HTML

  • Google Analyticsanalytics

    Found "googletagmanager.com/gtag/js" in the page's HTML

Related domains

Similar connectivity

Domains ranked just above and below it in the same snapshot.