One Click.
Your Entire Site in Monitoring.

Every other AI visibility tool monitors only the pages that AI platforms already cite. That means your product pages, your support docs, your landing pages — hundreds of pages that matter to your business — are invisible to your monitoring until an AI decides to mention them. Site Crawl fixes that. One click, and CiteMetrix crawls your site three levels deep, captures a structured snapshot of every page it finds, and adds them all to your Content Change Tracker monitoring. No CSV exports. No third-party tools. No waiting.

500Max pages per crawl

3Link levels deep

1/secCrawl rate, respecting your robots.txt

0Competitors that offer automated site crawling

The Coverage Gap Every Platform Ignores

You’re monitoring 12 pages. You have 400.

Here’s the uncomfortable math. The average CiteMetrix user monitors 8–15 pages per domain — the ones AI platforms have already cited. Their actual website has 200, 500, sometimes thousands of pages. Product pages, service descriptions, location pages, blog posts, documentation, case studies. Every one of those pages is a potential source for an AI citation. None of them are being watched.

This matters because AI platforms don’t cite the same pages forever. They discover new ones, drop old ones, shift which page they reference for a given topic. A product page that’s never been cited today could be the one Perplexity cites tomorrow — and if you don’t have a baseline snapshot, you won’t know what content was on it when that happened.

Until now, the only way to get those pages into CiteMetrix was to export a Screaming Frog CSV and import it — a manual process that requires a separate tool, a desktop crawl, and the discipline to re-import every time your site changes. Most users never did it.

Site Crawl makes full-site coverage a one-click operation. It runs inside CiteMetrix, crawls your domain from the homepage out, follows internal links three levels deep, and captures a structured snapshot of every discovered page. Your monitoring goes from 12 pages to 400 pages in about fifteen minutes.

Full-site coverage with Site Crawl

Open Content Change Tracker in CiteMetrix
Click “Crawl Site”
Go do something else
Come back to full-site snapshots in monitoring
Schedule regular crawls to keep coverage current
Done

How Site Crawl Works

🔍

Smart Link Discovery

Starting from your homepage, Site Crawl follows every internal link up to three levels deep. It discovers product pages, blog posts, category pages, documentation — every page linked from your site’s natural navigation. Pages behind login walls, external domains, and noindex pages are automatically excluded.

🤖

Respectful Crawling

Site Crawl identifies itself honestly as CiteMetrix, respects your robots.txt directives and any Crawl-delay you’ve set, and maintains a steady one-request-per-second pace. Your site’s visitors won’t notice it’s running, and your server logs will show exactly what it did.

📊

Full Structured Snapshots

Every discovered page gets the same structured snapshot as your existing monitored pages: title, meta description, headings, word count, links, schema types, canonical URL, and content hash. From the moment the crawl finishes, every page has a baseline for change detection.

⏱

Fast Turnaround

A typical crawl completes in 10–20 minutes, moving through your site at a steady one-page-per-second pace so it never disrupts your server. Watch live progress in the Jobs panel, or navigate away and check back later — it keeps running in the background either way.

What a Crawl Produces

✦ Sample Crawl Results — E-Commerce Site (247 pages discovered)

Domain: example-store.com — Crawl completed Sep 12, 2026

CRAWL SUMMARY:
  Pages discovered:    247
  Pages fully analyzed: 247
  New to monitoring:   231 (previously tracking 16 pages)
  Crawl depth reached: 3 levels
  Time elapsed:        11 minutes

TOP DISCOVERIES BY TYPE:
  Product pages:       89 pages   (none previously monitored)
  Collection pages:    24 pages   (3 previously monitored)
  Blog posts:          67 pages   (8 previously monitored)
  Support/FAQ pages:   31 pages   (2 previously monitored)
  Landing pages:       19 pages   (3 previously monitored)
  Other:               17 pages

SNAPSHOT STATUS:
  All 247 pages now have baseline snapshots.
  Next scan will detect changes against these baselines.

Results appear in the Jobs panel as the crawl runs. Navigate away and come back — it runs in the background.

Site Crawl by Plan

	Solo	Starter	Professional	Enterprise	Agency
Site Crawl	—	✓	✓	✓	✓
Crawls per domain / month	—	2	4	8	4–12
Max pages per crawl	—	250	500	500	500+
Screaming Frog import	—	✓	✓	✓	✓
Page snapshots	✓	✓	✓	✓	✓
Change detection	✓	✓	✓	✓	✓

Agency plans: Small doesn’t get Site Crawl access. Medium gets 4 crawls/mo, Large gets 8, Global gets 12 — all with 500+ page limits. See the agency pricing page for the full breakdown.

What to Know Before You Crawl

Bot protection on some e-commerce platforms

Platforms with aggressive bot protection (notably Shopify) may throttle or block sustained automated requests, even when the crawler respects robots.txt. If this happens, your results will show which pages were successfully analyzed and which returned errors. CiteMetrix identifies itself honestly and never spoofs a browser — some platforms simply don’t allow automated access. Your successfully crawled pages still get full snapshots.

Crawl depth is three levels

Site Crawl follows internal links three levels from your homepage. Pages buried deeper than three clicks from the home page won’t be discovered automatically. If you need deeper pages, you can still use Screaming Frog import to add them, or add individual page URLs to your monitoring manually.

One crawl at a time per domain

To keep server load predictable for both you and us, only one crawl runs per domain at a time, with a minimum one-hour gap between crawls of the same domain. If you need more frequent coverage, your regular scan cycle continues capturing snapshots of all monitored pages independently.

Common Questions

How long does a crawl take?

Depends on your site size and your server’s response time. At one request per second with a 500-page cap, a full crawl typically completes in 10–20 minutes. You’ll see progress in the Jobs panel in real time, and you can navigate to other parts of CiteMetrix while it runs.

Will it slow down my website?

No. Site Crawl makes one request per second — well below what any production web server notices. If your robots.txt specifies a slower Crawl-delay, CiteMetrix respects it. The crawl also runs headless Chromium rendering (the same technology search engines use), which requests pages exactly the way a real visitor would.

What about pages behind a login?

Site Crawl only follows publicly accessible links. Pages that require authentication, return a redirect to a login page, or are blocked by robots.txt are excluded automatically. If you need to monitor authenticated pages, add them individually through the dashboard.

Can I crawl a subdomain or a specific section of my site?

Currently, Site Crawl starts from the domain’s homepage and follows internal links. It will naturally discover subdomain pages if they’re linked from the main site. A future update will allow specifying a start URL other than the homepage. For now, you can import specific sections via Screaming Frog CSV.

I’m on a Solo plan. How do I get Site Crawl?

Upgrade to Starter or above. Starter gets 2 crawls per domain per month (up to 250 pages each). Professional gets 4 crawls per domain per month (up to 500 pages each). Enterprise gets 8 crawls per domain per month (up to 500 pages each). You can upgrade directly from the Plans page in your CiteMetrix account.

Stop Monitoring a Fraction of Your Site.
Start Monitoring All of It.

Site Crawl is available on Starter plans and above. One click, full-site coverage, structured snapshots of every page.

See Plans and Pricing

One Click.Your Entire Site in Monitoring.