FUURAA AI Knowledge Library · information-extraction evaluation

How to evaluate AI information-extraction capability claims

Use six evidence gates to turn “extracts fields automatically,” “complete entity relations” or “any document to structured data” into an exact task, schema, sources and annotation contract, ingestion pipeline, field support, complete records, all failures, human correction, cost and a dated target-workflow boundary.

Enter the six evidence gates ↓Open the failure matrix

Published30 August 2026Evidence statusMethod synthesis grounded in primary entity, relation and unified information-extraction researchScopeEntity, field, relation, event and structured information-extraction research

Valid structure is not source support; one F1 is not a complete record

Information-extraction capability must keep sources, schema, ingestion, field support, complete records and every failure in one evidence chain.

Entities, fields, relations, events and complete records are different tasks with different errors. A reviewable capability judgment requires a frozen schema and annotation contract, then source, ingestion, extraction, complete-denominator and real-use checks.

Applicability boundaryThis is a public research method, not a FUURAA or FUUVO product-capability claim or an assessment of any model, provider, language, schema, dataset or leaderboard. It is not a guarantee of field accuracy, record completeness or fitness.

Six rejectable evidence gates

Each gate requires minimum evidence and stops transfer or narrows the conclusion when material unknowns remain.

Freeze the extraction task, schema and decision

Decision question: Which entities, fields, relations, events, links or records must be produced, for whom and for what acceptance decision?
Minimum evidence: Verbatim claim and date; exact system, model, prompt and pipeline versions; source and target formats; schema, required and optional fields, cardinality, languages, users, consequence, thresholds and allowed abstention.
Stop condition: Stop when a benchmark label, JSON demo or broad “extracts anything” statement replaces a versioned schema, eligible population and decision threshold.

Version sources and the annotation contract

Decision question: What exact material was eligible, what counted as truth and how were spans, values, relations and omissions adjudicated?
Minimum evidence: Source provenance, snapshot, permissions, language and document distribution; annotation guide and schema version; span boundaries, nesting, overlap, aliases, normalisation, NULL policy, negative examples, annotator expertise, disagreement and exclusions.
Stop condition: Stop when corpus identity, eligible documents, schema semantics, reference labels or disagreement handling is unknown.

Separate ingestion, perception and extraction

Decision question: Did the system receive the relevant content faithfully before it detected, typed and connected information?
Minimum evidence: File and page coverage; OCR, layout parsing, transcription, decoding and segmentation quality; chunking, overlap, retrieval, context limits and truncation; prompt, tools and post-processing; controlled tests that isolate ingestion from extraction.
Stop condition: Stop when end-to-end scores hide unread pages, lost context, parser errors, retrieval misses or post-processing repair.

Score boundaries, types, links and complete records separately

Decision question: Does each output match the source, schema and record contract without invented or contradictory values?
Minimum evidence: Exact and partial span scores; detection versus typing; entity linking and normalisation; relation direction and arguments; event triggers, roles and coreference; field-level precision, recall and critical errors; schema validation, provenance pointers, duplicate handling and complete-record accuracy.
Stop condition: Stop when token F1, valid JSON or an unverified model judge substitutes for source support, required-field completeness and critical-error review.

Count every document, omission, hallucination and correction

Decision question: What happened across every eligible document, item, empty output, retry and important slice?
Minimum evidence: Complete document and item denominators; missing, extra, malformed, duplicated and conflicting values; no-output, refusal, timeout and tool failures; micro and macro results; document, language, field and subgroup slices; repeats, retries, human correction, latency, tokens and total cost.
Stop condition: Stop when selected records or aggregate F1 hide missing documents, rare fields, invented values, human rescue, slow tails or cost.

Transfer to the target workflow and expire the claim

Decision question: Does controlled extraction remain useful after real sources, schemas, people and downstream systems change?
Minimum evidence: Shadow or staged use; representative documents, languages, schema versions, accessibility needs and exception queues; human acceptance, reconciliation and downstream outcome checks; monitoring; source, OCR, model, prompt, schema and policy change triggers; owner and expiry date.
Stop condition: Stop when a static dataset transfers directly to live documents, new fields or consequential database writes without target validation and rollback.

Minimum information-extraction claim failure matrix

Check these conditions deliberately before transferring selected records, valid JSON or one F1 into real information-extraction capability.

01
A required entity or field is omitted while the output still looks complete
Record the affected document, field, schema, language, user and decision; preserve the narrowest conclusion that remains and specify whether to add sources, relabel, rerun, review manually, restrict use or withdraw the claim.
02
A fluent value or relation is generated without support in the source
Record the affected document, field, schema, language, user and decision; preserve the narrowest conclusion that remains and specify whether to add sources, relabel, rerun, review manually, restrict use or withdraw the claim.
03
Span boundaries, nesting or aliases cause one fact to be split, merged or duplicated
Record the affected document, field, schema, language, user and decision; preserve the narrowest conclusion that remains and specify whether to add sources, relabel, rerun, review manually, restrict use or withdraw the claim.
04
The entity type is right but its normalised identity, unit, date or code is wrong
Record the affected document, field, schema, language, user and decision; preserve the narrowest conclusion that remains and specify whether to add sources, relabel, rerun, review manually, restrict use or withdraw the claim.
05
A relation direction, event argument or cross-sentence link is reversed or missing
Record the affected document, field, schema, language, user and decision; preserve the narrowest conclusion that remains and specify whether to add sources, relabel, rerun, review manually, restrict use or withdraw the claim.
06
OCR, parsing, chunking or retrieval silently removes the evidence before extraction
Record the affected document, field, schema, language, user and decision; preserve the narrowest conclusion that remains and specify whether to add sources, relabel, rerun, review manually, restrict use or withdraw the claim.
07
Aggregate F1 hides weak rare fields, languages, document types or critical cases
Record the affected document, field, schema, language, user and decision; preserve the narrowest conclusion that remains and specify whether to add sources, relabel, rerun, review manually, restrict use or withdraw the claim.
08
A source, schema, model or downstream system change invalidates prior evidence
Record the affected document, field, schema, language, user and decision; preserve the narrowest conclusion that remains and specify whether to add sources, relabel, rerun, review manually, restrict use or withdraw the claim.

Minimum information-extraction capability evaluation record

Let the next reviewer reconstruct the conclusion with the same sources, schema, annotation contract, system and complete runs.

01Claim, date, owner, extraction task, schema, users, consequence and thresholds
02System, model, prompt, tools, runtime, pipeline and deployment versions
03Source corpus, provenance, permissions, snapshot, languages, formats and exclusions
04Schema, required fields, ontology, cardinality, normalisation and NULL policy
05Annotation guide, reference labels, span rules, negative examples and disagreements
06Ingestion, OCR, parsing, chunking, retrieval, context and post-processing settings
07Boundary, type, link, relation, event, schema-validity and record-level results
08Source support, required-field completeness, critical errors and subgroup results
09All documents and items, omissions, extras, failures, repeats, corrections, latency and cost
10Target-workflow evidence, reconciliation, monitoring, change triggers, rollback and expiry

Common evidence states

Bind conclusions to the exact pipeline, task, schema, sources, complete denominator, target workflow and date.

Supported

The exact pipeline meets source support, required-field completeness, critical-error, reliability, subgroup, human-effort, latency and cost thresholds on representative target documents, with a current review date.

Conditional

Support holds only for named source types, schema versions, fields, languages, domains, quality levels or operating controls.

Mixed

Detection, typing, linking, relations, record completeness, critical errors, reliability, latency or cost vary materially across slices.

Insufficient

Source or schema provenance, annotation contract, pipeline identity, support checks, complete denominator, target transfer or expiry is missing.

FUURAA analysisThe minimum decision unit for an AI information-extraction capability claim is exact system, model, prompt and extraction-pipeline version × extraction task, schema, fields, language, domain, user and decision × source provenance, document distribution, annotation guide, references and time × capture, OCR, parsing, chunking, context, retrieval and post-processing × detection, boundary, type, normalisation, relation, event, structural validity and critical errors × all documents, items, omissions, hallucinations, failures, human correction, latency and cost × target-workflow boundary and cut-off date. One correct record is an observation, not transferable proof of capability.

Primary sources and non-transfer boundaries

These sources constrain flat entities, fine-grained Chinese entities, sentence relations, cross-sentence relations, few-shot entities and unified entity/relation/event extraction.

Sources rechecked 30 August 2026. Each retains its publication timing, role in this method and non-transfer boundary.

Published 2003CoNLL-2003 — Language-Independent Named Entity Recognition

Established a shared task for locating and typing person, organisation, location and miscellaneous named entities in English and German news text.

BoundaryFour flat entity types in edited news do not establish nested entities, domain-specific fields, linking, relations, events, multilingual transfer or production-document extraction.

Open primary source ↗Published September 2017TACRED — A Large-Scale Supervised Relation Extraction Dataset

Introduced 119,474 crowdsourced sentence-level examples for TAC knowledge-base relations, including a large no-relation class.

BoundaryIts relation inventory, sentence candidates, crowdsourced labels and benchmark split do not prove open-schema extraction, complete document coverage, current facts or deployed knowledge-base accuracy.

Open primary source ↗Published 14 June 2019DocRED — Document-Level Relation Extraction

Adds human-annotated document-level entities and relations that can require synthesising evidence across multiple Wikipedia sentences.

BoundaryWikipedia documents, Wikidata relations and annotated evidence do not establish transfer to private corpora, OCR inputs, different ontologies, access controls or end-to-end database updates.

Open primary source ↗Published 16 May 2021Few-NERD — Few-Shot Fine-Grained Named Entity Recognition

Provides human annotations for eight coarse and 66 fine-grained entity types plus supervised and few-shot benchmark settings.

BoundaryWikipedia-derived paragraphs, balanced sampling and episodic few-shot tasks do not establish arbitrary new schemas, entity linking, production drift or critical-field reliability.

Open primary source ↗Published 13 January 2020CLUENER2020 — Fine-Grained Chinese Named Entity Recognition

A Chinese-led benchmark defines ten fine-grained entity categories and baseline systems for Chinese named entity recognition.

BoundaryTen categories and its Chinese text distribution do not represent every Chinese variety, specialist ontology, nested span, relation, event or operational extraction workflow.

Open primary source ↗Published 23 March 2022UIE — Unified Structure Generation for Universal Information Extraction

A Chinese-led framework unifies entity, relation, event and sentiment extraction through schema-conditioned text-to-structure generation across multiple datasets and data regimes.

BoundaryResults across selected tasks and datasets do not prove schema validity, source-grounded values, complete records, hallucination control, stable latency or safe downstream writes in a target workflow.

Open primary source ↗

Continue checking

Move from information extraction into document understanding, RAG, factuality or the full library.

Enter the AI Knowledge Library Evaluate factuality Evaluate RAG Evaluate document understanding and OCR Enter AI Evidence Atlas