Use CDM's preregistered log-study protocol to verify AI crawler identity, requests, response outcomes and robots behavior without equating access with citation.
Short answer: answer: CDM cannot yet say which AI crawlers reach marketing sites because no consented server-log dataset has been supplied. This article preregisters the Verified Crawler Observation Protocol, which records claimed user agent, verified operator, declared purpose, requested resource and response outcome separately. A request proves access at a moment; it does not prove indexing, model training, retrieval, citation or ranking.
Lists of crawler names are not observations. Copying a user-agent string from a vendor page and calling it “traffic” confuses what a bot may identify as with what participating servers actually received.
CDM’s position is that verified access logs should replace assumptions about discoverability, while crawler identity and downstream citation remain separate research questions. No result, site count or request share is claimed here. The record below is synthetic, and findings must be published only after consented collection, privacy review and reproducible analysis.
The Verified Crawler Observation Protocol
The protocol applies a five-link chain to each request: Capture → Claim → Verify → Classify → Interpret. Breaks remain visible. A “GPTBot” text match with an unverified source is a claimed identity, not confirmed OpenAI traffic; a verified 200 response is not evidence that ChatGPT cited the page.
Official documentation already shows why purpose matters. OpenAI distinguishes OAI-SearchBot for search, GPTBot for potential model training and ChatGPT-User for user-initiated actions. Anthropic documents ClaudeBot, Claude-SearchBot and Claude-User for different uses. Cloudflare’s 2026 taxonomy similarly separates Search, Agent and Training behavior. The study freezes the vendor reference table on collection start and preserves changes during the window.
Questions and observation unit
The primary questions are: which verified crawler families request participating marketing sites; what paths and resource types do they request; and what response status, robots state and latency do they encounter? Secondary questions compare declared purpose, revisit intervals, page freshness and site configuration.
The observation unit is one HTTP request at the origin or trusted edge. Aggregations by crawler, host, URL class and day are derived later. The study does not treat visits as people, and it will not infer model behavior from an operator label.
Full log schema
Participating site owners configure an export containing only approved fields:
| Group | Required fields |
|---|---|
| study context | site pseudonym, collection window, time zone, edge/origin source, consent version |
| request | event ID, UTC timestamp, host pseudonym, normalized path class, method, protocol, status, bytes, latency |
| agent claim | full user-agent stored in restricted tier, parsed product token, claimed family, claimed purpose |
| network verification | truncated or keyed source-IP value, ASN if permitted, vendor IP-list match, reverse-DNS result, verification method and timestamp |
| controls | robots rule applicable at request time, allow/disallow result, robots retrieval event, WAF action, rate-limit action |
| content context | HTML/image/script/feed/robots/sitemap/other, template class, published/updated age band, canonical status |
| quality | clock check, duplicate flag, cache/origin visibility, parser version, missingness and exclusion reason |
Query strings, cookies, authorization headers, request bodies, referrers, account IDs and full IP addresses are excluded from the research file. Paths are mapped to safe classes such as article, category, homepage, asset, sitemap and robots. A restricted transient table may be necessary for verification, but it must be access-controlled and deleted on the registered schedule.
Recruit sites and collect a consecutive window
Recruit verified site owners or authorized operators, not scraped domains. Include content-led marketing sites that can export reliable edge or origin logs and document robots/WAF configuration. Seek variation in geography, hosting stack, site size and AI-bot policy. For every enrolled site, collect a consecutive 60-day window; do not select only days with known bot activity.
The design target is 40 sites. It is not an achieved sample. Public comparative results require at least 15 independent sites, three hosting stacks and 45 valid observation days per included site, with no site contributing more than 20% of weighted request counts. Otherwise CDM will publish a pilot methods report without market-wide prevalence claims.
The site owner consents to log use and confirms authority to provide it. The protocol still needs a privacy assessment because IP addresses and request paths can relate to individuals or reveal confidential URLs. Inform relevant operators, minimize fields, document lawful basis where required, and exclude private/authenticated areas. A data-protection or ethics reviewer decides whether further notice or approval is necessary in each jurisdiction.
Verify identity without trusting the label
Verification follows operator-supported evidence captured at request time:
- Parse the user-agent token against the frozen reference table.
- Match source address to an operator’s published IP ranges when available.
- Use forward-confirmed reverse DNS when the operator documents that method.
- Record trusted edge-provider verification signals and their product scope.
- Label the event verified, inconsistent, unverified or unverifiable.
Google explicitly warns that user-agent headers are often spoofed and recommends reverse DNS or published ranges. Cloudflare describes published IP lists, reverse DNS and cryptographic Web Bot Auth as verification routes in its verified-bot system. The study will not collapse those methods into equal certainty; it will publish the rule set and parser version.
Provider documentation changes. Preserve the reference JSON or page hash, retrieval date and effective interval. Do not retroactively apply a new range to old requests without a registered sensitivity analysis.
Classify requests and controls
Map each verified family to the purpose its operator declares at the time: Search, Training, User-directed Agent, or Other. This is a declared-purpose classification, not direct evidence of downstream use. Record robots.txt state for that token at the moment of request and whether the server allowed, blocked, challenged or failed it.
Normalize URLs before aggregation: remove queries and fragments, map redirects, identify canonical and alternate URLs, and classify resources. Preserve status categories—2xx success, 3xx redirect, 4xx client/control response and 5xx server failure. Report robots and sitemap fetching separately because those requests can dominate small sites without representing article access.
Failure test
Before accepting a site, inject synthetic log lines representing spoofed agents, changing IP ranges, redirects, blocked paths and duplicated edge/origin events. The pipeline must flag the spoof, retain verification uncertainty, deduplicate known mirrored events and reconcile control totals. Synthetic lines use reserved site and event prefixes and can never enter results.
Prespecified analysis
Primary outputs are the proportion of participating sites reached by each verified crawler family, requests per site-day, requested path-class distribution and response/control outcomes. Give site-weighted medians and interquartile ranges; also show raw event counts with their concentration. Report unverified claims separately rather than discarding them invisibly.
Estimate revisit intervals only for repeated normalized resources. Compare declared purposes and robots states descriptively. Do not publish a global “AI traffic share” unless ordinary human/bot traffic was collected under the same coverage and privacy rules. Do not test whether crawling causes citation: that would require a separate, controlled observation of answer systems and queries.
Freeze code before the final data lock. Report exclusions, missing edge coverage, cache visibility, configuration changes and downtime. Any exploratory analysis—such as geographic egress or status-code transitions—must be labelled exploratory. The hypothesis that AI-search crawlers disproportionately request current articles is a preregistered hypothesis, not a result.
Illustrative record—not a finding
At 2026-10-14 09:15:00 UTC, synthetic site SITE-X3 records an HTML GET to path class /article/, status 200 and 184 ms latency. The user agent claims OAI-SearchBot. The source matches the operator IP list cached that morning; the applicable robots rule allows the token; no WAF action occurs. Classification: verified operator, Search declared purpose, successful article request.
This record is invented to test the schema. It does not show that OAI-SearchBot reached CDM, that it visits marketing sites commonly, or that the page appeared in a ChatGPT answer.
Publication and stopping rules
Pause ingestion if logs expose credentials, full query strings, authenticated paths, unstable identity keys or unexpected personal data. Quarantine affected files, notify the site custodian and obtain privacy/security review. Stop at the registered date; do not extend because a desired crawler has not appeared.
Publish only after minimum coverage, verification-quality reporting, reproducible code review, disclosure-risk testing and technical SEO/research review. Release the frozen protocol, data dictionary, vendor-reference snapshot, synthetic test pack and aggregated disclosure-controlled results. If site diversity or log completeness fails, say so and publish no prevalence headline.
Related guides
Frequently asked questions
Which AI crawler visits marketing websites most often?
CDM does not know yet. No consented, verified multi-site log dataset has been analyzed, so naming a leader would fabricate a result. The protocol will compare site-weighted reach and request rates after separating verified traffic from user-agent claims. It will report both how many sites were reached and how concentrated requests were.
The caveat is that participating sites may not represent the web, and edge controls can prevent requests from reaching origin logs. Any future answer must state the sample, observation window, log location and verification rule beside the number.
Is a user-agent string enough to identify an AI crawler?
No. User-agent text is easy to copy, so it establishes a claim, not identity. Verify against operator-published IP ranges, documented reverse-DNS procedures, cryptographic signals or a trusted provider’s verified-bot classification, and record which method was used. Apply the check to the request timestamp, not only to today’s vendor list.
The caveat is that verification support and ranges change; absence from a list does not prove maliciousness. Preserve uncertainty and the reference version instead of relabelling every unmatched request as fake.
Does a crawler request mean my page will be cited by an AI answer?
No. A log entry proves only that a server or edge handled a request with particular attributes. Indexing, retrieval, ranking, answer generation and citation happen downstream and are not visible in that request. Even a verified search crawler returning 200 does not guarantee inclusion. A blocked request can diagnose access, but it still reveals nothing about earlier stored copies.
The caveat is that repeated successful access can be a useful technical diagnostic. Pair it with a separate, reproducible answer-monitoring study if you want to investigate citations, and avoid causal claims.
Should the study store full IP addresses and URLs?
Not in the analysis dataset. Use transient, restricted source data only when necessary for verification, transform IPs to keyed or truncated values, and map URLs to safe path classes before analysis. Exclude queries, private paths, cookies and bodies entirely. Keep the transformation key outside the working dataset and delete it on schedule.
The caveat is that pseudonymized fields may still permit re-identification when combined with timestamps and public pages. A privacy review, retention schedule and disclosure test must govern every release rather than assuming hashing equals anonymity.
How will robots.txt behavior be measured?
Capture the applicable file and rule state over time, map each product token to its rule, and record whether the server allowed, blocked, challenged or failed the request. Preserve configuration-change timestamps because one current robots file cannot explain an earlier visit.
The caveat is that user-triggered agents and operator policies may treat robots directives differently, and logs do not prove an operator’s internal interpretation. Report the observed control path alongside the vendor’s documented purpose and avoid declaring compliance from one event.
When can CDM publish the log-study results?
Only after consented collection reaches the registered coverage minimum, the verification pipeline passes synthetic failure tests, the dataset is locked, and privacy, technical and methods reviewers approve the analysis and release. The code and frozen vendor-reference table should be released with safe aggregates. A shortfall can support a labelled pilot or methods note, not a market benchmark.
The caveat is that numeric thresholds do not cure selection bias or missing edge traffic. The report must disclose recruitment, hosting mix, exclusions, concentration and observability limits so readers can judge its reach.
Next decision: Technical Foundations for AI Discovery: Crawlability, Snippets, Internal Links and Structured Data
Related reading: Structured Data That Matches Visible Content: A Practical Governance Guide · Original Research as Authority Infrastructure: From Question to Citable Asset · How to Build an Evidence-First Content Page That AI Systems Can Quote Safely
Sources and research notes
- OpenAI: Overview of OpenAI Crawlers — checked 26 September 2026; official distinctions among OAI-SearchBot, GPTBot and ChatGPT-User, plus published IP resources.
- Anthropic: Web Crawlers and Site Controls — checked 26 September 2026; official distinctions among ClaudeBot, Claude-SearchBot and Claude-User and robots guidance.
- Google: Overview of Crawlers and Fetchers — checked 26 September 2026; official crawler categories and technical properties.
- Google: Googlebot and Verification — checked 26 September 2026; warns that user agents can be spoofed and documents verification routes.
- Cloudflare: Verified Bots — checked 26 September 2026; verification methods and 2026 behavior taxonomy covering Search, Agent and Training.
- ICO Anonymisation Guidance — checked 26 September 2026; identifiability, pseudonymisation and governance considerations for log release.
Limitations: CDM has not received or analyzed server logs and claims no crawler frequency, site reach, indexing or citation result. Proposed targets are protocol choices, not achieved samples. Edge visibility, spoofing, dynamic IP ranges, configuration changes, caching and self-selected sites will limit inference. Replace this protocol with findings only after consented, ethical collection and review while retaining the preregistration record.
This article is editorial guidance. Apply the principles in proportion to your market, evidence, and responsibilities.
