Build a verified bot ledger that separates spoofed user-agents, successful requests and content delivery from indexing, retrieval and citation claims.
Short answer: answer: Server logs can show that a request reached your edge or origin, what identity it claimed, which URL it requested, the response and bytes returned. Classify a named AI bot only after validating the source with the operator’s current IP or verification guidance; otherwise label it unverified. A successful verified request is evidence of delivery, never proof of indexing, model training, retrieval, answer generation or citation.
The tempting dashboard is “AI bot visits this month.” That number often counts any user-agent containing GPT, AI or bot, including spoofed traffic, security scanners and user-triggered fetches. It may also miss legitimate requests terminated at a CDN because those never reach the origin log.
Logs are valuable precisely when their claim is kept narrow. Our position is: a small, verified footprint ledger is more useful than a large bot-traffic chart, because false identity turns crawl measurement into theatre.
The Verified Bot Footprint Ledger
CDM’s Verified Bot Footprint Ledger moves every request through five states: Captured, Claimed, Verified, Delivered and Interpreted. Preserve the raw event, then add derived classifications. Never overwrite the source record with a friendly bot name.
| State | Question | Defensible output |
|---|---|---|
| Captured | Where did the request terminate? | edge, WAF, load balancer or origin event |
| Claimed | What did the user-agent say? | exact string and parsed token |
| Verified | Does the network source match current operator guidance? | verified, failed or not checked |
| Delivered | What response and representation were returned? | status, bytes, content type, cache and fingerprint |
| Interpreted | What does this event establish? | request, denial or delivery—nothing downstream |
OpenAI publishes address files for OAI-SearchBot, GPTBot and ChatGPT-User. Google recommends reverse and forward DNS checks or matching its published IP ranges, and Bing offers a Verify Bingbot method while warning that user-agents are easy to spoof. Apply the method documented for the specific operator; do not assume one vendor’s method authenticates another.
Capture the right log surfaces
Start at the public edge. A CDN cache hit can serve a crawler without generating an origin request, while a WAF block can prevent the request from reaching application logs. Collect CDN access, security-event and origin logs with synchronized UTC timestamps. Preserve the edge request ID across upstream calls where the platform permits it.
At minimum, capture time, host, path, query handling, method, status, response bytes, content type, user-agent, source IP, edge/WAF action, cache result, request ID and origin status. Limit or hash query strings when they can contain personal data or tokens. Retention and access should follow the organization’s privacy and security rules.
Avoid interpreting response status alone. A 200 may contain a challenge page or empty application shell. For critical pages, store a privacy-safe body signature or monitor for a known public sentence. Do not log confidential response bodies merely to improve crawler analytics.
Normalize claims without granting identity
Keep user_agent_raw immutable. Add claimed_agent using explicit patterns and a parser-version field. Versioning matters because crawler strings change. OpenAI’s current documentation, for example, says version numbers in sample user-agents may change.
A useful event schema is:
``json { "event_time": "2026-09-26T12:04:17Z", "request_id": "edge-5d91", "host": "www.brand.example", "path": "/research/claim-ledger/", "status": 200, "bytes": 38219, "cache": "HIT", "user_agent_raw": "...OAI-SearchBot/1.4...", "claimed_agent": "OAI-SearchBot", "source_ip": "redacted-in-report", "identity_result": "verified", "verification_basis": "openai-range-snapshot-2026-09-26", "delivery_result": "expected-public-content", "interpretation": "verified-request-delivered" } ``
The example is a schema, not a real CDM log. Keep the sensitive address in controlled raw storage and expose only a result and verification snapshot to most analysts.
Verify identity against dated evidence
Download or query the operator’s current published data through a controlled job, validate its format, record retrieval time and retain a hash. Compare source addresses using a proper IP-address library, not string prefixes. IPv6 and CIDR boundaries make home-grown comparisons error-prone.
For Googlebot, the official process can use reverse DNS followed by forward confirmation, or published IP ranges. For Bingbot, use Bing’s verification mechanism. For OpenAI crawlers, compare with the published address data linked from the crawler documentation. Recheck results when ranges update; a once-verified address is not a permanent certificate.
Use three outcomes: verified, failed and not_checked. “Not checked” is not “probably legitimate.” Requests that claim a bot but fail verification may be spoofing, outdated data, proxying or a processing error; investigate before blocking or attributing.
Measure delivery and recurrence
Create one daily row per verified agent, URL and response class. Count requests, distinct paths, successful content deliveries, blocks, rate limits and server errors. Add first-seen and last-seen timestamps. Separate robots.txt requests from content requests, and separate automatic crawlers from user-initiated fetchers where the vendor makes that distinction.
An illustrative query is:
``sql SELECT event_date, claimed_agent, path, COUNT(*) AS requests, SUM(CASE WHEN identity_result='verified' AND status=200 THEN 1 ELSE 0 END) AS verified_200, SUM(CASE WHEN status IN (403,429) THEN 1 ELSE 0 END) AS denied FROM crawler_events WHERE event_date BETWEEN DATE '2026-09-01' AND DATE '2026-09-26' GROUP BY event_date, claimed_agent, path; ``
This query still does not prove the returned body contained the expected evidence. Join to a body-fingerprint check or content monitor for priority pages. Label cache hits clearly so teams do not misdiagnose missing origin rows.
Keep the interpretation ladder strict
A verified log event supports only the lower rungs of this ladder:
- Claimed: a request carried a named user-agent.
- Verified: the source matched the operator’s current method.
- Delivered: the stack returned the expected public representation.
- Acquired or indexed: requires engine-side evidence; a server cannot observe storage.
- Retrieved: requires query, citation or internal trace evidence.
- Generated or cited: requires answer-level observation or first-party reporting.
Do not fill upper rungs from lower ones. A crawler may fetch a page and discard it. A system may cite a URL discovered through another search provider without a recent request visible in your retention window. Training is a separate purpose and generally cannot be inferred from output resemblance.
Reproducible test protocol and expected results
Pick three public URLs, one expected-block URL and robots.txt. Record the body fingerprint and expected policy. Send approved simulated requests to validate logging and field parsing. Then observe real traffic for a defined window without generating it. Run identity verification on every claimed named bot.
Expected classifications are:
- simulated request, expected body: pipeline test only;
- claimed bot, identity failed: unverified automation;
- verified bot, 200, expected fingerprint: verified delivery;
- verified bot, 403 or challenge: access failure at the recorded layer;
- verified robots request only: robots access, not content access;
- no request observed: no evidence within the window, not proof of blocking.
Audit a random sample against raw logs and rerun after parser or CDN changes. If counts differ materially between edge and origin, explain cache, blocks or sampling before publishing totals.
Related guides
Frequently asked questions
Can a user-agent string identify an AI crawler?
It can identify what the request claims to be, not who sent it. User-agent strings are plain request headers and are easy to copy. Parse and preserve the string, then verify the source using the named operator’s current published method.
Until that check passes, report “claimed OAI-SearchBot” or “unverified bot,” not “OpenAI visit.” Some user-triggered agents and third-party retrieval services have different purposes and verification options. Even verified identity establishes a request, not subsequent indexing, training or use in an answer.
Why do CDN and origin bot counts disagree?
They measure different points in the request path. A CDN may satisfy a cache hit without contacting the origin, block a request at the WAF, retry an upstream failure or combine fields differently. Origin logs therefore undercount edge-served requests, while CDN logs may lack application-level content checks.
Correlate request IDs, cache status, edge action and origin status before reconciling. Clock skew and log sampling can add discrepancies. Choose the public edge as the main request count, then use origin records to diagnose upstream delivery rather than forcing both totals to match.
Does a verified 200 request mean the bot read the article?
It means the verified requester received a 200 response and, if checked, the expected bytes or content fingerprint. “Read” can imply parsing, retention or understanding that server logs cannot observe. Label the event “expected representation delivered.” Confirm content type, response length and decisive-text fingerprint because a soft error or challenge can also return 200.
The requester may discard the response, parse only part of it or use it for a purpose unrelated to search. Indexing, retrieval and citation require separate evidence.
Can logs show whether content was used for model training?
No. Logs can show a verified request by a crawler whose documented purpose includes collecting content that may be used for training, such as GPTBot. They cannot show whether the response entered a dataset, survived filtering, influenced a model or appears in an output.
Preserve the purpose label and robots decision, but state the limit. An answer that resembles site text is also not reliable proof of training because the fact may exist elsewhere or be retrieved at answer time. Training claims require information from the model provider, not inference from a web log.
How long should crawler logs be retained?
Retain enough history to compare normal cadence, deployments and incidents, while complying with privacy, security and contractual limits. Ninety days is a useful operational starting point for aggregated verified footprints, but raw IP-level retention may need to be shorter or more restricted.
Store verification snapshots or hashes so an old classification can be audited after published ranges change. Document sampling and deletion. The correct period depends on risk, traffic volume and applicable law; log analytics is not a reason to retain personal data or credentials indefinitely.
What should an AI crawler dashboard display?
Display verified successful deliveries, verified denials or errors, unverified claims, top public paths, robots-versus-content requests, edge cache status and the verification coverage rate. Include the analysis window, log surfaces, sampling and a permanent note that requests do not prove citation. Avoid a single “AI visibility” score.
If first-party citation reporting is available, place it in a separate panel with its own definitions and coverage. Keeping request and citation panels apart prevents viewers from interpreting correlated movement as a causal funnel.
Next decision: How Should You Separate OAI-SearchBot, GPTBot and Googlebot Controls?
Related reading: How Do You Test AI Crawler Access Across Robots.txt, CDN and Firewall Rules? · How Do You Correct Citation Drift After a Brand Fact Changes? · How to Measure Brand Visibility in AI Answers Without Inventing Certainty
Sources and research notes
- OpenAI: Overview of OpenAI crawlers — primary user-agent purposes and links to published address data for OAI-SearchBot, GPTBot and ChatGPT-User. Checked 26 September 2026.
- Google Search Central: Verify Googlebot and other Google crawlers — primary reverse/forward DNS and published-range verification guidance. Checked 26 September 2026.
- Bing Webmaster Tools: Which crawlers does Bing use? — primary Bingbot strings and warning that user-agents can be spoofed. Checked 26 September 2026.
- Bing Webmaster Tools: How to report an issue with Bingbot — first-party recommendation to verify both user-agent and address and preserve log samples. Checked 26 September 2026.
- Google Search Central: Googlebot — primary details on user-agent changes, rendering and verification. Checked 26 September 2026.
- Limitations: the schema and SQL are vendor-neutral CDM examples and require adaptation to the site’s privacy policy and log platform. Published IP data and user-agents change. Server-side evidence cannot reveal downstream storage, training, retrieval or citation.
This article is editorial guidance. Apply the principles in proportion to your market, evidence, and responsibilities.
