Key takeaway
When a citation arrives with no matching crawl, it came from training data, not retrieval. Rewriting the page will not move it. Blocking training while allowing retrieval is a coherent position; blocking all three with one inherited wildcard is what usually happens.
Detection happens server-side in first-party tracking, so this is observed traffic rather than a third-party estimate.
Name the engine behind every visit. Each known AI crawler user-agent maps to what it feeds, OpenAI, Claude, Gemini, Perplexity, Mistral, DeepSeek, Meta AI and the rest, and a visit you cannot attribute is labelled untagged rather than quietly dropped.
Decide what to block on purpose rather than by accident. Each bot carries a purpose, training corpus, search index, on-demand fetch or mixed, and blocking training while allowing retrieval is a coherent strategy that almost nobody is actually running.
See which money pages your own robots.txt refuses. Half or more of them blocking AI bots caps the Get Found rung at weak however heavily the rest of the site is crawled, because a blocked page cannot be cited.
Stop guessing how long to wait. Per page and per engine you get the gap between first crawl and first citation over 90 days, so you know whether a publishing change has had time to land.
Crawl records and citation records are joined on a full outer join, so the cases where one side is missing survive rather than disappearing. Each is a different problem.
Both sides present, so the latency between them is measurable. This is the working case, and the distribution of that latency per engine is what tells you how fast your category moves.
The engine fetched the page and has not cited it in 90 days. Retrieval is not the problem. This is an extractability or authority problem, which is rungs 3 and 4, and the page audit is where you go next.
The engine cites the page but never fetched it inside your history. That usually means the citation comes from training data rather than live retrieval, which is a very different asset: it does not update when you edit the page.
Fewer than 14 days since the first event, so no conclusion is drawn. It reads as insufficient history rather than as a zero, because a page crawled yesterday has not failed to be cited.
First seen, last seen and visit count per bot per page over 90 days, so a crawler that stopped coming is visible as a change rather than as an absence you have to notice.
Crawl volume by engine over time, so you can see one engine going quiet while the others hold steady, which is usually a robots or a rate-limit problem rather than a content one.
Crawl data is only useful if it changes a decision. These are the three it changes most often.
Turn the robots argument into a list. Most AI bot blocks are inherited rather than chosen, a wildcard written for scrapers or a plugin default, and seeing which named bots are refused on which pages ends the debate.
Find out which of three different weeks of work you are in. Crawled but not cited is a content problem, never crawled is an access problem, and cited with no crawl record is a training-data asset you cannot edit your way into or out of.
Know when a change has genuinely failed. The observed crawl-to-citation latency for your own pages and engines tells you how long absence has to last before it means anything, measured rather than assumed.
Most teams either read raw logs by hand or assume it is fine.
AI crawler visits identified server-side
TrustData
GA4
Other tools
Log tools, unclassifiedBot mapped to the engine it feeds
TrustData
GA4
Other tools
Training against retrieval against on-demand
TrustData
GA4
Other tools
robots.txt checked against your money pages
TrustData
GA4
Other tools
ManualCrawl-to-citation latency
TrustData
GA4
Other tools
| Feature | TrustData | Google Analytics 4 | Other tools |
|---|---|---|---|
| AI crawler visits identified server-side | Log tools, unclassified | ||
| Bot mapped to the engine it feeds | |||
| Training against retrieval against on-demand | |||
| robots.txt checked against your money pages | Manual | ||
| Crawl-to-citation latency |
FAQ
Detection happens server-side in the first-party tracking layer, which inspects the user agent of every request before any JavaScript runs. AI crawlers do not execute JavaScript, so a client-side analytics tool cannot see them at all. That is why crawler activity is missing from most dashboards rather than showing as zero: the measurement never had a chance to happen.
Because the decisions are different. A training crawler is building a corpus that will be baked into a future model. A search-index crawler is maintaining an index that answers questions today. An on-demand fetch happens because a user asked something right now and the engine went to read your page for that answer. Blocking training while allowing retrieval is a defensible position on your content rights. Blocking all three because a wildcard rule caught them is what usually happens, and it costs you citations for no considered reason.
It generally means the citation comes from the model's training data rather than from live retrieval. That is worth knowing because it behaves differently: editing the page will not update what the model says, and the citation can vanish with a model refresh you do not control. It is also why probes run in two modes, with live web search and without. A brand strong without search and weak with it exists in the model but is not being retrieved; the reverse means the opposite.
Crawl events and citation records are joined per page and per engine across 90 days of history, on a full outer join so that pages with only one side present are kept rather than dropped. Where both sides exist, the gap between the first crawl and the first citation is the latency. Where only one exists, the row is classified: crawled but not cited, or cited with no crawl record. Under 14 days since the first event nothing is concluded, and the row reads insufficient history.
On the retrieval side, directly and completely: an engine that cannot fetch a page cannot cite it, which is why the Get Found rung caps at weak when half or more of your audited money pages block AI bots. On the training side it is a genuine trade rather than a mistake, and some brands make it deliberately. TrustData's job is to make sure it is a decision. The common case we see is neither: a rule written years ago for a different problem, still quietly refusing the crawlers that matter now.
14-day free trial
14-day free trial. Crawler monitoring is part of AI Visibility, included in every plan.