How Should Brand Content Be Tested Differently for Public AI Search and Private RAG?

Use a dual-environment test to measure public crawl and citation separately from private RAG retrieval, grounding, refusal and scope preservation.

Short answer: answer: Test the same brand facts in two independent environments. For public AI search, verify external access, search eligibility, observed discovery and answer citation; for private RAG, verify ingestion, chunk retrieval, source-grounded generation, scope preservation and refusal. Pass only when at least 90% of critical test questions retrieve the approved evidence and every material answer claim is supported; never treat a crawler request as proof of citation.

The obvious shortcut is to publish a “RAG-friendly” page, ask one chatbot a few questions and call the content ready. That collapses several systems into one result. Public answer engines choose when to crawl, index, retrieve, synthesize and cite. A private RAG owner controls a defined corpus, ingestion schedule, chunking, retrieval settings and prompt.

The reverse shortcut is also wrong: a document that performs well in a private vector index may be unreachable or ineligible on the public web. Our position is firm: one combined AI-readiness score is operationally dishonest. Public discovery and private RAG must have separate pass gates, even when they use the same source text.

The Public-Private Retrieval Split Test

CDM’s Public-Private Retrieval Split Test has four paired gates: Access, Acquisition, Retrieval and Answer. Run the same question set through both lanes, but record different evidence. A public lane is observational because the publisher does not control the engine. A private lane is instrumented because the operator can preserve corpus, configuration and traces.

This table is a CDM testing model, not a claim that all public engines expose every stage. Where a stage is opaque, mark it “not observable.” Do not convert missing evidence into a pass.

Gate 1: prove access in each environment

For the public lane, request the canonical URL without authentication, then inspect status, redirects, headers, HTML and rendered text. Check robots rules for the actual crawler token. OpenAI documents OAI-SearchBot for search and GPTBot for potential training as independent controls. Google likewise separates Googlebot’s search crawl from the Google-Extended control token; Google says Google-Extended does not control Search inclusion or ranking.

Access is necessary, not sufficient. A 200 response from your laptop does not show that a CDN, bot-management rule or geography-specific edge will serve the crawler. A robots allowance does not show that the engine indexed the page. Google explicitly distinguishes crawling, rendering and indexing, and says meeting technical requirements does not guarantee indexing.

For private RAG, test the connector as the service identity, not as an administrator. Record which repository, folder, tenant and document ACLs are in scope. Export parser text for the exact version tested. If a file appears in the index but a legitimate user cannot retrieve it—or an unauthorized user can—the content test has exposed a permission failure, not a writing problem.

Gate 2: distinguish acquisition from usable representation

An HTTP log can document a request. It cannot prove durable indexing, retrieval, training, answer use or citation. Similarly, an ingestion dashboard can report “success” while silently dropping a table, footnote or scanned page.

Create a fact manifest before testing. Each critical fact needs an identifier, approved wording, scope, effective date, source location and permitted audience. Then compare the acquired representation with that manifest.

``yaml fact_id: PLAN-PRICE-US-2026Q3 claim: "The Professional plan costs $79 per month in the United States." scope: "US; monthly billing; excluding tax" effective_from: 2026-07-01 source_url: "https://brand.example/pricing" critical_qualifiers: [United States, monthly billing, excluding tax] expected_access: public ``

In the public lane, inspect what a text-only fetch and rendered page expose. In the private lane, inspect extracted text and chunk boundaries. The test fails if the number survives but “United States” or “excluding tax” is separated from it.

Gate 3: evaluate retrieval with different proof

Public retrieval is usually inferred from citations, first-party publisher reporting or controlled query observation. OpenAI says OAI-SearchBot supports inclusion in ChatGPT search, but inclusion remains discretionary. Bing’s AI Performance reporting, currently described as preview, reports citations and grounding-query samples across supported Microsoft experiences; Microsoft warns those counts do not indicate ranking, placement or page importance.

Run at least 20 stable, non-personalized questions spanning exact facts, paraphrases, comparisons, exclusions and “unknown” cases. Use a clean test protocol: date, locale, product surface, signed-in state, exact prompt, answer, links shown and screenshot or export. Repeat on three separated dates because generated answers are nondeterministic and public indexes change. A citation is evidence that the interface displayed a source for that sample, not proof that every claim came from it.

Private retrieval can be measured directly. Label the expected documents or chunks for each question, capture the top five results, and calculate hit rate at k. A practical critical-fact gate is at least 90% hit@5 with zero unauthorized-document retrievals. This is a CDM recommendation, not a universal standard; regulated or safety-sensitive facts may require 100%.

Gate 4: audit the answer, not merely the source list

For both lanes, break an answer into material claims. Mark each claim supported, contradicted, only partially supported or not in the cited/retrieved evidence. Check whether dates, markets, product tiers and exceptions survive synthesis. Microsoft’s RAG documentation distinguishes retrieval evaluation from groundedness and response completeness for exactly this reason: good retrieval does not guarantee a faithful or complete answer.

The private lane also needs refusal tests. Ask questions outside the approved corpus, requests that combine conflicting versions and prompts from users without permission. A fluent guess is a failure. Record the prompt, model, temperature or equivalent settings, corpus snapshot and retrieval configuration so the result can be reproduced.

Public systems do not offer the same controls. Score only observable output and citations. Do not infer training from an answer, do not infer indexing from a crawl, and do not infer grounding from a citation badge.

Dual-environment retrieval test

Use this compact record for every question:

Run the first baseline before changing content. Make one change—such as keeping a price and its geographic qualifier in one block—then repeat the same questions. Public improvement remains an association unless the platform provides stronger evidence. In private RAG, preserve the prior index and configuration so the change can be isolated.

Diagnose failures without crossing lanes

If public access fails, inspect robots, authentication, CDN and firewall behavior. If access passes but no citation is observed, the cause is unknown: indexing, relevance, competing sources or answer policy may be involved. Do not rewrite endlessly based on one absence.

If private ingestion fails, inspect connectors and parsers. If retrieval misses, examine chunking, metadata, embeddings and filters. If retrieval succeeds but the answer distorts scope, fix answer instructions, evidence blocks and evaluation—not the index alone. This separation prevents content teams from “optimizing” prose for a fault located in infrastructure.

Related guides

• How Do You Combine GA4 and Search Console Into a Content Decision Dashboard?
• How Do You Build an Evidence and Refresh Workflow for WordPress Articles?

Frequently asked questions

Can the same page work for both public AI search and private RAG?

Yes, the same canonical evidence can serve both, but it must pass two test suites. Clear headings, explicit entities, dated claims and nearby qualifiers help humans and both retrieval environments. Public systems still impose crawler, index and product-level choices that the publisher cannot control; private systems impose connector, parser, chunking, permissions and prompt choices that the operator can inspect.

Maintain one approved meaning, then create environment-specific representations only where necessary. The exception is confidential or licensed material: it should remain outside the public lane and enter private RAG only with enforceable access controls.

Does an OAI-SearchBot visit mean ChatGPT will cite the page?

No. A verified request proves that the server received a request associated with that crawler at that time. It does not prove indexing, retrieval for a later question, answer generation, citation or model training. OpenAI documents OAI-SearchBot as the control for search inclusion and says GPTBot is separately related to potential training, but eligibility is not a promise of appearance.

Preserve the log as access evidence, then look for citations in controlled answer samples or first-party reporting. Even a displayed citation must be audited claim by claim because it may not support every sentence beside it.

What is the minimum useful public AI search test set?

Start with 20 questions, not one vanity prompt. Include five exact factual questions, five paraphrases, three comparisons, three scope or date challenges, two outdated formulations and two questions the page should not answer. Run them on at least three dates and record locale, interface, sign-in state, answer and displayed sources.

This is a practical CDM baseline, not a statistically representative measure of all users or prompts. Expand the set from real customer language and search data; for high-risk claims, sample more often and require human review.

Which private RAG metrics matter most for brand facts?

Measure retrieval hit rate, source faithfulness, qualifier preservation, completeness, refusal and permission safety. Hit@5 shows whether expected evidence appears among the first five retrieved items; claim-level grounding shows whether the answer actually uses it. Add a separate test for obsolete documents and access-control leakage.

Fluency and user satisfaction matter, but they cannot rescue an unsupported price, warranty or compliance claim. Thresholds should reflect harm: a 90% retrieval gate may be useful for routine content, while legal, medical or contractual facts can justify a stricter standard and mandatory human escalation.

Should public and private tests use identical questions?

Use a shared core plus environment-specific cases. The shared questions expose whether approved facts retain meaning across systems. Public cases should probe canonical URLs, recency and citations; private cases should probe tenant permissions, internal terminology, conflict resolution and refusal outside the corpus.

Do not publish confidential questions merely to make the sets symmetrical. Version every question and its expected evidence so results remain comparable after edits. If a public engine changes its interface or a private model changes, label a new test run rather than silently combining unlike results.

How often should the split test be repeated?

Repeat it after a critical fact, template, crawler rule, CDN policy, connector, parser, embedding model, retrieval configuration or generation model changes. Also run a scheduled critical-fact suite weekly for fast-changing offers and monthly for stable corporate facts. Public results can move without a site change, so store the engine surface and timestamp.

Private results can drift when the corpus grows even if the model stays fixed. The cadence is a risk decision; the exception is an urgent correction, which should trigger immediate retesting and monitoring rather than waiting for the normal schedule.

Next decision: How Do You Test AI Crawler Access Across Robots.txt, CDN and Firewall Rules?

Related reading: How Do You Build a Brand-Fact Evaluation Set for a Private RAG System? · How Do You Test Whether an AI Answer Is Grounded in the Sources It Cites? · Technical Foundations for AI Discovery: Crawlability, Snippets, Internal Links and Structured Data

Sources and research notes

• OpenAI: Overview of OpenAI crawlers — primary definitions for OAI-SearchBot, GPTBot and user-initiated fetchers, including their independent controls. Checked 26 September 2026.
• Google Search Central: How Google Search works and technical requirements — primary documentation separating crawl, render and index stages and stating that eligibility does not guarantee indexing. Checked 26 September 2026.
• Microsoft Foundry: Retrieval-Augmented Generation evaluators — primary documentation separating document retrieval, retrieval quality, groundedness and completeness evaluation. Checked 26 September 2026.
• Bing Webmaster Blog: AI Performance public preview — first-party description and limits of citation and grounding-query reports. Checked 26 September 2026.
• Liu et al.: Lost in the Middle — original research showing that relevant information placement in long contexts can affect model use; supports testing representation rather than assuming ingestion is enough. Checked 26 September 2026.
• Limitations: public answer systems expose incomplete telemetry and change without notice. The 20-question, three-date and 90% thresholds are CDM recommendations, not platform guarantees or universal standards; teams should calibrate them to risk.