Purpose
See when a user reads you through ChatGPT
Each crawler is classed by purpose: model training, indexing, or an on-demand fetch to answer the question a user just asked.
Get Found, the first rung
See who reads your site when nobody clicks
70% of discovery queries now get an AI answer. When the reader is an engine rather than a browser, your logs say which one, for what purpose, and whether a citation followed.
Key takeaway
When a user asks ChatGPT or Perplexity, the engine comes to read your page, and no click reaches you. That visit does not exist in GA4. It exists in your logs, with the engine, the purpose and the time, and TrustData ties it to the citation that followed, or did not.
Get Found
Four questions your logs answer and GA4 cannot
An engine answering a user reads your page on their behalf, with no browser and no JavaScript. GA4 does not see it. Server-side first-party tracking records it.
Per engine
Find out which engine actually read the page
Each known crawler maps to the engine it feeds: OpenAI, Claude, Gemini, Perplexity, Mistral, DeepSeek, Meta AI and the rest.
Crawl to citation
Know when to stop waiting for a citation
Per page and per engine, the gap between first crawl and first citation over 90 days, so you know whether a change has landed.
Blocked pages
Find the pages your robots.txt turns away
Your money pages blocked in robots.txt, listed one by one: a blocked page is never cited, whatever the rest of the site does.
The four outcomes of a crawl
Crawls and citations are matched page by page and engine by engine. Each case calls for a different action.
Crawled and cited
The delay between the two is measured, and its distribution per engine shows how fast your sector gets picked up.
Crawled, not cited
The engine read the page and has not cited it in 90 days: the problem is the content, and the page audit is next.
Cited with no crawl record
The engine cites a page it never read: the citation comes from its training data, and editing the page will not change it.
Insufficient history
Under 14 days of history, the row says so instead of showing a zero that is not one.
What you do with it
Crawl data is only useful if it changes a decision. These are the three it changes most often.
Fix robots.txt with evidence
Most blocks are inherited, an anti-scraping rule or a plugin default. You see which crawler is refused on which page.
Separate an access problem from a content problem
Never crawled is an access problem, crawled without a citation a content problem. The two are not fixed in the same place.
Count AI reads for every visit it sent
The ratio of crawls to referred visits, per engine, points at the one that reads you heavily and sends you nobody.
Against the usual ways of checking this
Most teams either read raw logs by hand or assume it is fine.
AI crawler visits identified server-side
TrustData
Google Analytics 4
Other tools
Log tools, unclassifiedCrawler mapped to the engine it feeds
TrustData
Google Analytics 4
Other tools
Crawl purpose: training, index or on-demand fetch
TrustData
Google Analytics 4
Other tools
robots.txt checked page by page
TrustData
Google Analytics 4
Other tools
By handCrawl-to-citation delay
TrustData
Google Analytics 4
Other tools
| Feature | TrustData | Google Analytics 4 | Other tools |
|---|---|---|---|
| AI crawler visits identified server-side | Log tools, unclassified | ||
| Crawler mapped to the engine it feeds | |||
| Crawl purpose: training, index or on-demand fetch | |||
| robots.txt checked page by page | By hand | ||
| Crawl-to-citation delay |
FAQ
Direct answers.
Because AI crawlers do not execute JavaScript, and GA4 only measures what a browser runs. TrustData detects them server-side in the first-party tracking layer, which inspects the user agent of every request before any script loads. That is why crawler activity is missing from most dashboards rather than showing as zero: the measurement never had a chance to happen.
Because the decisions are different. A training crawler is building a corpus that will be baked into a future model. A search-index crawler is maintaining an index that answers questions today. An on-demand fetch happens because a user asked something right now and the engine went to read your page for that answer. Blocking training while allowing retrieval is a defensible position on your content rights. Blocking all three because a wildcard rule caught them is what usually happens, and it costs you citations for no considered reason.
The citation generally comes from the model's training data rather than from live retrieval. That is worth knowing because it behaves differently: editing the page will not update what the model says, and the citation can vanish with a model refresh you do not control. It is also why probes run in two modes, with live web search and without. A brand strong without search and weak with it exists in the model but is not being retrieved; the reverse means the opposite.
Crawls and citations are matched per page and per engine across 90 days of history, keeping the pages where only one of the two exists. Where both exist, the delay between the first crawl and the first citation is measured. Where only one exists, the row is classified: crawled without a citation, or cited without a crawl. Under 14 days of history nothing is concluded, and the row says so.
On the retrieval side, directly and completely: an engine that cannot fetch a page cannot cite it, which is why the Get Found rung caps at weak when half or more of your audited money pages block AI bots. On the training side it is a genuine trade rather than a mistake, and some brands make it deliberately. TrustData's job is to make sure it is a decision. The common case we see is neither: a rule written years ago for a different problem, still quietly refusing the crawlers that matter now.
14-day free trial
Extend your analytics to what AI reads
14-day free trial. Crawler monitoring is part of AI Visibility, included in every plan.