Test PDFs and images with a gold evidence manifest covering OCR, reading order, tables, charts, qualifiers, provenance and retrieval accuracy.
Short answer: answer: Build a gold manifest of the exact claims, qualifiers, table relationships and chart values that must survive extraction, then run the original files through the same parsers and OCR used in production. Pass only when critical evidence is complete, in the right reading order, tied to its source page or region and retrieved by representative questions. A file opening successfully—or a crawler returning 200—does not prove that decisive evidence can be parsed, indexed, retrieved or grounded.
Teams often test a PDF by searching for one visible phrase. That misses the harder failures: a price detached from its currency, a table read column by column, a footnote omitted, a chart legend lost or a scan converted into plausible but wrong digits.
An image-caption feature can also sound convincing while inventing values it cannot read. Our position is uncompromising: if a decisive fact exists only as pixels, it is an unverified asset until a gold-set extraction test passes; visual polish is not evidence accessibility.
The Multimodal Evidence Extraction Test
CDM’s Multimodal Evidence Extraction Test has six checks: Open, Read, Structure, Associate, Retrieve and Ground. It evaluates the production pipeline, not an idealized desktop viewer.
| Check | Question | Failure example |
|---|---|---|
| Open | Can the system access and decode the file? | password, unsupported format or size limit |
| Read | Are characters and numbers correct? | OCR reads 8.5% as 85% |
| Structure | Is reading order and hierarchy preserved? | two columns interleave |
| Associate | Do values remain tied to labels, legends and notes? | chart value loses market and date |
| Retrieve | Do real questions return the evidence? | correct page is indexed but never ranked |
| Ground | Does the answer preserve the extracted evidence? | answer drops “excluding tax” |
Accessibility and retrieval are not identical, but they often share useful foundations. W3C’s PDF7 technique says scanned image-only PDFs lack extractable actual text and recommends checking completeness and reading order after OCR. Tagged structure, meaningful alternatives and logical order help people using assistive technology and give document pipelines better signals; they still require direct testing.
1. Create a gold evidence manifest
Select ten to twenty facts that matter operationally: dates, prices, thresholds, product names, table comparisons, chart values and limitations. Label each fact as critical or supporting. Store the human-verified source page, bounding region, exact value, acceptable normalization and required context.
``json { "evidence_id": "REPORT-2026-FIG4-US", "file": "market-report-2026.pdf", "page": 18, "region": "Figure 4, blue series, 2026 point", "claim": "US aided awareness was 62% in the 2026 wave.", "required_context": ["United States", "aided awareness", "2026", "%"], "critical": true, "tolerance": "exact" } ``
This is an example schema, not a claim from a real report. Two people should verify high-risk values against the visual source. If a chart supports only an approximate reading, the manifest must say so; do not manufacture false precision.
2. Build representative file variants
Test the files users and systems actually receive: born-digital tagged PDF, born-digital untagged PDF, scan-only PDF, mixed text-and-scan document, chart image, screenshot and mobile-compressed image. Include multi-column text, tables with merged cells, footnotes, rotated pages, non-Latin characters and low-contrast figures if they occur in production.
Keep a small synthetic challenge pack in which every answer is known. For example, use one table where the same number appears under different markets, and one chart where the legend—not the color alone—determines the series. Synthetic files test mechanics; real licensed files test operational diversity. Never present synthetic results as performance on the public corpus.
Record resolution, page count, file size, language, encryption, tags and version hash. Google Cloud’s Document AI guidance says scan quality, resolution and font size affect OCR and recommends testing; its documentation suggests at least 200 dpi and generally reports better results at 300 dpi or more. That is product guidance, not a universal guarantee.
3. Inspect raw and structured extraction
Run the exact production parser, OCR version and options. Save raw text plus structured output for pages, lines, tables, figures, coordinates, confidence and reading order where available. Microsoft’s Document Intelligence layout documentation distinguishes geometric elements such as tables and figures from logical roles such as titles and headings. A plain-text dump cannot test those relationships alone.
Compare every critical item with the manifest:
- character accuracy for names, dates, decimals and currencies;
- completeness of required context;
- correct order for columns, captions and footnotes;
- row and column association in tables;
- legend, axis, unit and time association in charts;
- page or region provenance;
- explicit indication when a value is unreadable.
Use exact comparison for identifiers and bounded numeric tolerance only where the source warrants it. Confidence scores are diagnostic; a high model confidence is not human-verified truth.
4. Test association, not just text presence
Search extracted output for all required tokens, then determine whether they appear in a usable unit. “62,” “US” and “2026” somewhere on a page is not enough if the parser pairs 62% with Europe. Serialize tables as headers plus row labels for every cell. Serialize charts with title, series, x-value, y-value, unit, source note and uncertainty.
When important meaning cannot be reliably extracted, publish a nearby text equivalent in HTML or accessible document text. The equivalent should state the claim and limitation, not merely say “chart showing growth.” For images, write alternatives according to their function and context; decorative images should not be treated as evidence.
Maintain the visible source. A hidden machine-only paragraph that materially differs from what readers see undermines trust and can resemble cloaking. The text equivalent should be reviewable by the same subject-matter owner.
5. Run retrieval questions over the extracted representation
Ingest the saved output into a test index. Ask exact, paraphrased and adversarial questions. Capture top-five results with evidence IDs, page references and scores. Critical evidence should be retrieved for its direct question and at least one natural paraphrase. Add negative questions whose answer is absent; the system should refuse or state the gap.
Example query set:
``text Q1: What was aided awareness in the United States in the 2026 wave? Q2: In 2026, what percentage of US respondents recognized the brand when prompted? Q3: What was unaided awareness in Canada? [not present] ``
The first two should retrieve REPORT-2026-FIG4-US; the third should not be answered from that evidence. Log parser version, index snapshot, embedding or search configuration and ranks so another operator can reproduce the run.
6. Audit generated answers at claim level
Give the model only the retrieved test context and require evidence IDs. Split each response into claims, then mark supported, contradicted, partial or absent. Check every required qualifier. A perfect OCR result can still become an ungrounded answer, and a grounded answer can still be incomplete.
Set thresholds by risk. A practical release gate is 100% exact extraction for critical identifiers and values, at least 90% retrieval hit@5 for critical questions, zero unsupported critical claims and correct refusal for every deliberately absent fact. These are CDM recommendations, not formal accessibility or product standards. Human review remains necessary for legal, safety, financial and regulated evidence.
Repair order and diagnostic tree
If Open fails, fix format, encryption, access or platform limit. If Read fails, replace the scan, improve resolution, enable OCR and correct suspect text. If Structure fails, repair tags, order, tables and headings. If Association fails, add explicit textual equivalents and provenance. If Retrieve fails, adjust chunking, metadata and search. If Ground fails, improve context selection, answer instructions and evaluation.
Re-run from the failed stage and then execute the whole end-to-end suite. Preserve before-and-after output; a parser update can improve one layout while regressing another.
Related guides
Frequently asked questions
Is OCR enough to make a scanned PDF usable by AI?
No. OCR creates text candidates, but the output can contain character errors, wrong reading order and lost relationships. W3C’s PDF7 technique calls for checking that all text was converted correctly and read in the correct order, and it notes that tagging may still be needed.
For AI retrieval, also test tables, captions, footnotes, provenance and representative questions. A page can be searchable yet still produce a dangerously wrong price or percentage. Keep the original visual, corrected text and an audit trail for critical evidence.
How should chart data be represented for retrieval?
Represent each decision-relevant point with the chart title, metric, series, x-value, y-value, unit, geography or segment, date, source and limitation. A concise table or prose summary next to the visible chart is usually more robust than a generic image caption. Preserve whether values are exact, rounded or estimated from a graphic.
Test natural questions against the representation and verify that retrieved answers retain the legend and axis meaning. If the chart contains hundreds of points, publish the underlying accessible data where rights and privacy allow.
What extraction accuracy should a team require?
Require 100% accuracy for critical identifiers, dates, thresholds, prices and values that could change a decision, with human verification. For supporting prose, measure word or field accuracy plus retrieval and grounding rather than relying on one aggregate OCR percentage. A 99% character score can still hide a catastrophic decimal error. Set thresholds before testing and report results by evidence class.
The exception is inherently approximate visual evidence; document the allowed tolerance and never score an estimate as exact. Route every failed critical field to correction or explicit refusal before release.
Should we test more than one parser or OCR engine?
Yes when documents are diverse or the evidence is high risk. Run the production parser first, then compare a second engine on failures and representative layouts. Different engines handle native PDF text, scans, tables, handwriting and reading order differently. Keep versions and settings fixed during a comparison.
The goal is not to crown a universal winner; it is to select a routing rule and fallback. Costs, data residency and licensing can constrain alternatives, so a verified HTML equivalent may be safer than adding another opaque extraction step.
Does an accessible PDF guarantee good RAG performance?
No. Accessibility work can provide actual text, structure, alternatives and reading order that materially help extraction, but retrieval also depends on parsing, chunking, metadata, indexing, query design and generation. Conversely, a parser may extract some inaccessible content while the document still fails people using assistive technology. Test both outcomes against their own requirements.
Do not use an AI extraction pass as a substitute for an accessibility audit, and do not assume an accessibility checker measures claim retrieval or groundedness. Use the same source file, but retain separate acceptance records and accountable reviewers for each outcome.
Can public search engines read every PDF they crawl?
No. Access, supported format, file size, indexing choices and extraction quality all matter, and vendor limits differ. Google documents supported-file and byte-processing limits for Googlebot, while a successful server response still says nothing about later indexing or citation. Provide a stable HTML landing page with the document’s title, owner, date, summary and decisive evidence where appropriate.
Use X-Robots-Tag for supported non-HTML indexing controls. Test public search and private document pipelines separately because their parsers and telemetry are not equivalent. If the PDF is the legal record, identify that status without pretending the HTML summary replaces it.
Next decision: How Do You Write Evidence Blocks That Preserve Context During Retrieval?
Related reading: How Should Brand Content Be Tested Differently for Public AI Search and Private RAG? · How Do You Test Whether an AI Answer Is Grounded in the Sources It Cites? · Structured Data That Matches Visible Content: A Practical Governance Guide
Sources and research notes
- W3C WAI: PDF7, Performing OCR on a scanned PDF to provide actual text — accessibility technique and test for complete, correctly ordered OCR text. Checked 26 September 2026.
- Microsoft Learn: Document Intelligence layout model — primary documentation for extracting text, tables, figures, geometric regions and logical roles. Checked 26 September 2026.
- Google Cloud: Document AI supported files and Enterprise Document OCR — primary format, scan-quality, OCR and native-PDF parsing guidance. Checked 26 September 2026.
- Google Search Central: Googlebot — primary supported-file processing limits and verification guidance for public crawling. Checked 26 September 2026.
- Microsoft Learn: RAG preparation phase — primary guidance to examine layout, images, tables, languages and preprocessing before chunking. Checked 26 September 2026.
- Limitations: extraction quality is tool-, version-, language- and document-specific. The schema, challenge pack and release thresholds are CDM recommendations. This protocol does not certify WCAG conformance or guarantee public indexing or citation.
This article is editorial guidance. Apply the principles in proportion to your market, evidence, and responsibilities.
