FUURAA AI Knowledge Library · RAG evaluation

How to evaluate retrieval-augmented generation claims

Use six evidence gates to turn “connected to a knowledge base, fewer hallucinations and answers with citations” into an exact pipeline, versioned corpus, permissions, independent retrieval, claim-level support, abstention, all failures, human correction, cost and a dated target-workflow boundary.

Enter the six evidence gates ↓Open the failure matrix

Published28 August 2026Evidence statusMethod synthesis grounded in primary retrieval and RAG researchScopeEnterprise knowledge bases, search QA, cited generation and retrieval-augmented workflow research

Retrieved is not used correctly; cited is not necessarily supported

RAG capability must keep corpus, permissions, retrieval, reranking, generation, citations, abstention, failures and updates in one evidence chain.

A final answer can look correct through model memory and can carry well-formed citations even when retrieval fails. Test retrieval independently, connect each material answer claim to evidence, and include no-answer, conflict, staleness and permission leakage in the denominator.

Applicability boundaryThis is a public research method, not a FUURAA or FUUVO product-capability claim or an assessment of any model, provider, search engine, knowledge base, organisation, dataset or leaderboard. It is not a guarantee of accuracy, completeness, access control, safety or decision quality.

Six rejectable evidence gates

Each gate requires minimum evidence and stops transfer or narrows the conclusion when material unknowns remain.

Freeze the claim, task and decision

Decision question: What does “RAG works” mean: finding passages, answering, citing, updating, summarising or supporting a specific decision?
Minimum evidence: Verbatim claim and date; pipeline, model, embedding and API versions; user, task, languages, stakes; acceptance thresholds for retrieval, answer support, abstention, latency and cost.
Stop condition: Stop when a demo, one benchmark score or the presence of citations substitutes for an exact task and decision boundary.

Version the corpus, permissions and ingestion chain

Decision question: What could the system search at the evaluation cut-off, who was allowed to see it, and how did source material become retrievable units?
Minimum evidence: Corpus manifest and hashes; owners, licences, effective dates and deletion state; ACL snapshot; parser, OCR, deduplication, chunking, metadata and language handling; failed and excluded documents.
Stop condition: Stop when corpus coverage, freshness, provenance, failed ingestion or access control is unknown.

Test retrieval and reranking independently

Decision question: Does the system retrieve the necessary evidence under realistic queries, languages, paraphrases and distractors before generation begins?
Minimum evidence: Representative query set; relevance judgements; recall and ranking by query slice; lexical, sparse, dense, hybrid and reranker versions; top-k, filters, thresholds, missing-evidence and permission-leak tests.
Stop condition: Stop when only end-answer quality is reported, hiding whether success came from retrieval, model memory or chance.

Verify evidence use, citations and conflict handling

Decision question: Does every material answer claim follow from the cited text, and what happens when sources disagree, are stale or do not contain an answer?
Minimum evidence: Atomic claim-to-passage review; citation correctness and completeness; source authority and date; quote integrity; multi-source synthesis; noise, counterfactual, conflict, no-answer and abstention cases.
Stop condition: Stop when a plausible answer, URL count or citation format is treated as proof of support.

Count every query, failure, correction and resource

Decision question: What happened across all eligible runs, including timeouts, empty retrieval, blocked access, malformed citations, retries and human repair?
Minimum evidence: Full denominator and run IDs; zero-result and error rates; unsafe or unauthorised retrieval; answer defects; retries, caching and fallbacks; reviewer minutes; index, model, storage, network, latency and cost distributions.
Stop condition: Stop when failures are excluded, retries are free, or human correction and infrastructure cost disappear from the claim.

Transfer to the target workflow and expire the conclusion

Decision question: Does the same controlled pipeline help the intended users on their corpus, permissions and consequences after sources, models or indexes change?
Minimum evidence: Shadow or staged deployment; user and query sampling; decision-quality and harm review; monitoring by source and slice; corpus, ACL, parser, embedding, retriever, reranker, prompt and model change triggers; review owner and expiry date.
Stop condition: Stop when benchmark evidence is transferred to production without direct validation, monitoring or a dated recheck.

Minimum RAG-claim failure matrix

Check these conditions deliberately before transferring vector scores, selected answers or citation counts into real RAG capability.

01
The gold evidence was never ingested, was parsed incorrectly or is hidden by the wrong permission filter.
Record the affected corpus, query, retrieval layer, answer claim, user and decision; preserve the narrowest conclusion that remains and specify whether to re-ingest, repair permissions, rerun retrieval, review manually, abstain or withdraw the claim.
02
A dense retriever succeeds on familiar wording but misses acronyms, identifiers, rare terms, another script or a domain shift.
Record the affected corpus, query, retrieval layer, answer claim, user and decision; preserve the narrowest conclusion that remains and specify whether to re-ingest, repair permissions, rerun retrieval, review manually, abstain or withdraw the claim.
03
Relevant evidence falls across chunks, pages or documents and top-k truncation removes the required relationship.
Record the affected corpus, query, retrieval layer, answer claim, user and decision; preserve the narrowest conclusion that remains and specify whether to re-ingest, repair permissions, rerun retrieval, review manually, abstain or withdraw the claim.
04
A highly ranked passage is topically similar but does not entail the answer claim.
Record the affected corpus, query, retrieval layer, answer claim, user and decision; preserve the narrowest conclusion that remains and specify whether to re-ingest, repair permissions, rerun retrieval, review manually, abstain or withdraw the claim.
05
The model follows a noisy, stale or counterfactual passage over stronger evidence or fails to report conflict.
Record the affected corpus, query, retrieval layer, answer claim, user and decision; preserve the narrowest conclusion that remains and specify whether to re-ingest, repair permissions, rerun retrieval, review manually, abstain or withdraw the claim.
06
Citations exist but attach to the wrong claim, omit a material claim, point to an inaccessible page or quote outside context.
Record the affected corpus, query, retrieval layer, answer claim, user and decision; preserve the narrowest conclusion that remains and specify whether to re-ingest, repair permissions, rerun retrieval, review manually, abstain or withdraw the claim.
07
The answer appears correct because of model memory even when retrieval is disabled or deliberately irrelevant.
Record the affected corpus, query, retrieval layer, answer claim, user and decision; preserve the narrowest conclusion that remains and specify whether to re-ingest, repair permissions, rerun retrieval, review manually, abstain or withdraw the claim.
08
Corpus updates, deletions, ACL changes, embedding refreshes or cache behaviour silently invalidate the earlier result.
Record the affected corpus, query, retrieval layer, answer claim, user and decision; preserve the narrowest conclusion that remains and specify whether to re-ingest, repair permissions, rerun retrieval, review manually, abstain or withdraw the claim.

Minimum RAG capability evaluation record

Let the next reviewer reconstruct the conclusion with the same corpus, permissions, pipeline, queries, evidence and complete runs.

01Claim, evaluation date, owner, user, task, language, consequence and acceptance thresholds
02Pipeline, model, prompt, embedding, retriever, reranker, API and deployment versions
03Corpus manifest, hashes, ownership, licences, dates, deletions, permissions and excluded sources
04Parser, OCR, deduplication, chunking, metadata, indexing and refresh configuration
05Query sample, relevance judgements, retrieval and reranking results by meaningful slice
06Atomic answer claims, supporting passages, citation correctness, completeness and source authority
07No-answer, noise, conflict, counterfactual, stale-source and permission-leak results
08All eligible runs, empty results, failures, retries, fallbacks and human corrections
09Latency, storage, indexing, retrieval, generation, network, review and total cost distributions
10Target-workflow validation, monitoring, change triggers, residual limits, review owner and expiry

Common evidence states

Bind conclusions to the exact pipeline, corpus, permissions, query distribution, complete denominator, target workflow and date.

Supported

The exact pipeline meets retrieval, support, permission, failure, cost and target-workflow thresholds on representative evidence with a current review date.

Conditional

Support holds only for named corpora, tasks, languages, query classes, source dates, permission models or operating controls.

Mixed

Retrieval, answer support, citation completeness, refusal, language, freshness, latency or cost vary materially across slices.

Insufficient

Pipeline identity, corpus provenance, permissions, independent retrieval tests, claim-level support, full denominators, target transfer or expiry is missing.

FUURAA analysisThe minimum decision unit for a RAG capability claim is exact pipeline and version × user, task, language and decision × corpus provenance, coverage, freshness, chunking, metadata and permissions × query, embeddings, retrieval, reranking, filters and top-k × atomic-claim support, citation completeness, conflict handling and abstention × all runs, failures, corrections, latency and cost × target-workflow boundary and cut-off date. RAG, BEIR, ALCE, RGB, C-MTEB and CRUD-RAG illuminate generation architecture, retrieval transfer, citations, robustness, Chinese embeddings and the full pipeline; none independently proves production RAG capability.

Primary sources and non-transfer boundaries

These sources constrain retrieval, generation, citations, bilingual robustness, Chinese embeddings and whole-pipeline evaluation.

Sources rechecked 28 August 2026. Each retains its publication timing, role in this method and non-transfer boundary.

First submitted 22 May 2020; revised 12 April 2021RAG — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Introduces a generative architecture that combines parametric memory with a dense, non-parametric Wikipedia index and evaluates it on knowledge-intensive tasks.

BoundaryResults from a fixed Wikipedia index, retriever and research task set do not establish the quality, freshness, provenance, access control or safety of another deployed RAG pipeline.

Open primary source ↗First submitted 17 April 2021; revised 21 October 2021BEIR — Heterogeneous Zero-Shot Information Retrieval

Evaluates lexical, sparse, dense, late-interaction and reranking systems across 18 diverse retrieval datasets, exposing domain-dependent generalisation and cost trade-offs.

BoundaryRetrieval ranking on public text collections does not independently establish answer support, generation quality, permission correctness, freshness or target-workflow performance.

Open primary source ↗First submitted 24 May 2023; revised 31 October 2023ALCE — Automatic Evaluation of LLMs' Citation Quality

Builds end-to-end retrieval-and-generation tasks and separates fluency, answer correctness, citation correctness and citation completeness.

BoundaryA citation benchmark and automatic entailment measures do not prove that every material claim is supported, that sources are authoritative, or that a target user can verify them.

Open primary source ↗First submitted 4 September 2023; revised 20 December 2023RGB — Benchmarking LLMs in Retrieval-Augmented Generation

A Chinese-led bilingual benchmark isolates noise robustness, negative rejection, information integration and counterfactual robustness in RAG systems.

BoundaryControlled English and Chinese testbeds do not cover every corpus, language variety, conflicting source, temporal update, model version or deployment consequence.

Open primary source ↗First submitted 14 September 2023; revised 24 September 2024C-Pack / C-MTEB — General Chinese Embedding Resources

Makes Chinese embedding training resources and C-MTEB, spanning six tasks and 35 datasets, accessible to international retrieval evaluation.

BoundaryEmbedding results across benchmark datasets do not prove chunking quality, corpus coverage, access control, reranking, answer support or end-to-end RAG usefulness.

Open primary source ↗First submitted 30 January 2024; revised 15 July 2024CRUD-RAG — A Comprehensive Chinese RAG Benchmark

A Chinese-led benchmark evaluates retriever, knowledge base, context length and generator across create, read, update and delete scenarios rather than question answering alone.

BoundaryChinese news-derived tasks and benchmark configurations do not establish performance for every enterprise corpus, permission model, update cycle, language, domain or high-stakes decision.

Open primary source ↗

Continue checking

Move from RAG into factuality, long context, multilingual and document-understanding evaluation, or the full library.

Enter the AI Knowledge Library Evaluate factuality Evaluate long context Evaluate multilingual capability Evaluate document understanding Enter AI Evidence Atlas