FUURAA AI Knowledge Library · data-analysis evaluation

How to evaluate AI data-analysis and tabular-reasoning capability claims

Use six evidence gates to turn “upload data for instant insight,” “query databases in natural language” or “generate reliable charts automatically” into an exact task, data snapshot, schema and business definitions, queries and calculations, result support, visual communication, all failures, human review, cost and a dated target-workflow boundary.

Enter the six evidence gates ↓Open the failure matrix

Published30 August 2026Evidence statusMethod synthesis grounded in primary table QA, text-to-SQL, financial reasoning and data-analysis-agent researchScopeTable QA, text-to-SQL, financial numerical reasoning and data-analysis-agent research

A polished chart is not reliable analysis; one correct number is not complete capability

Data-analysis capability must keep data, schema, queries, calculations, result support, visual communication and every failure in one evidence chain.

Table lookup, text-to-SQL, calculation, fact verification, open-ended analysis and chart generation are different tasks with different errors. A reviewable capability judgment requires a frozen data question, snapshot and business definitions, then query, calculation, complete-denominator and real-use checks.

Applicability boundaryThis is a public research method, not a FUURAA or FUUVO product-capability claim or an assessment of any model, provider, database, dataset or leaderboard. It is not a guarantee of data accuracy, analytical completeness, financial conclusions or fitness.

Six rejectable evidence gates

Each gate requires minimum evidence and stops transfer or narrows the conclusion when material unknowns remain.

Freeze the analysis task, data question and decision

Decision question: What exact lookup, comparison, aggregation, calculation, query, chart or recommendation is required, for whom and for what acceptance decision?
Minimum evidence: Verbatim claim and date; exact system, model, prompt, tools and runtime; data question, eligible sources, output contract, languages, users, consequence, thresholds, allowed assumptions and abstention conditions.
Stop condition: Stop when a polished chart, one correct number or broad “analyses any data” statement replaces a named task, output contract and decision threshold.

Version data, schema and reference answers

Decision question: What exact files, tables, rows, columns, definitions and time range were available, and what counted as truth?
Minimum evidence: Data provenance, snapshot, permissions and cut-off; schema, types, units, joins, keys and business definitions; missing, duplicate, censored and outlier rules; reference queries, formulas, programs, intermediate values, answer tolerances, reviewer expertise and disagreements.
Stop condition: Stop when the data snapshot, schema semantics, metric definition, denominator, reference calculation or fact date is unknown.

Separate access, parsing, querying and computation

Decision question: Did the system read the right cells and records, form the right query or code, and execute it in the intended environment?
Minimum evidence: File and sheet coverage; encoding, locale, date and number parsing; table detection and header inference; schema linking, filtering, joins and aggregation; generated SQL, formulas or code; package and engine versions; permissions, sandbox, timeouts, retries and execution traces.
Stop condition: Stop when final-answer accuracy hides skipped sheets, parsing errors, wrong joins, silent coercion, execution failure or human repair.

Verify calculations, support and visual communication

Decision question: Can every material number, statement and chart be reconstructed from the named data and operations?
Minimum evidence: Query execution and result-set checks; formula and intermediate-value verification; units, signs, dates, rounding and denominator checks; source-to-claim mapping; uncertainty and missing-data disclosure; chart data, scale, labels, filters and accessibility; contradiction and sensitivity tests.
Stop condition: Stop when fluent explanation, matching format or an unverified model judge substitutes for executable reconstruction and critical-number review.

Count every task, failure, variance and human correction

Decision question: What happened across the complete eligible set, repeated runs and important data or user slices?
Minimum evidence: Complete task and file denominator; empty, malformed, unsupported and partially correct outputs; query, tool and chart failures; repeated-run variance; source, format, language, complexity and subgroup results; retries, manual edits, review time, latency, tokens, compute and total cost.
Stop condition: Stop when selected successes or average accuracy hide failed files, wrong critical numbers, unstable repeats, human rescue, slow tails or cost.

Transfer to the target workflow and expire the claim

Decision question: Does the controlled result remain useful after live data, schemas, definitions, people and downstream systems change?
Minimum evidence: Shadow or staged use; representative data sizes, formats, languages, accessibility needs and escalation routes; human acceptance, reconciliation and downstream outcome checks; monitoring; data, schema, metric, model, prompt, tool and policy change triggers; owner, rollback and expiry date.
Stop condition: Stop when a static benchmark transfers directly to live reports, financial decisions or database actions without target validation, review and rollback.

Minimum data-analysis claim failure matrix

Check these conditions deliberately before transferring selected answers, successful queries or polished charts into real data-analysis capability.

01
The right number is calculated from the wrong table, date range or denominator
Record the affected data, query, metric, format, user and decision; preserve the narrowest conclusion that remains and specify whether to add data, correct definitions, rerun, review manually, restrict use or withdraw the claim.
02
A date, decimal separator, currency, unit or missing value is parsed incorrectly
Record the affected data, query, metric, format, user and decision; preserve the narrowest conclusion that remains and specify whether to add data, correct definitions, rerun, review manually, restrict use or withdraw the claim.
03
A join duplicates or drops records while the aggregate still looks plausible
Record the affected data, query, metric, format, user and decision; preserve the narrowest conclusion that remains and specify whether to add data, correct definitions, rerun, review manually, restrict use or withdraw the claim.
04
Generated SQL, formula or code is valid but answers a different business definition
Record the affected data, query, metric, format, user and decision; preserve the narrowest conclusion that remains and specify whether to add data, correct definitions, rerun, review manually, restrict use or withdraw the claim.
05
An explanation or chart states a conclusion not supported by the computed result
Record the affected data, query, metric, format, user and decision; preserve the narrowest conclusion that remains and specify whether to add data, correct definitions, rerun, review manually, restrict use or withdraw the claim.
06
A chart scale, filter, label or omitted category changes the reader's interpretation
Record the affected data, query, metric, format, user and decision; preserve the narrowest conclusion that remains and specify whether to add data, correct definitions, rerun, review manually, restrict use or withdraw the claim.
07
One file type, language, rare category or complex query hides materially weaker results
Record the affected data, query, metric, format, user and decision; preserve the narrowest conclusion that remains and specify whether to add data, correct definitions, rerun, review manually, restrict use or withdraw the claim.
08
A data refresh, schema, metric, model or tool change invalidates prior evidence
Record the affected data, query, metric, format, user and decision; preserve the narrowest conclusion that remains and specify whether to add data, correct definitions, rerun, review manually, restrict use or withdraw the claim.

Minimum data-analysis capability evaluation record

Let the next reviewer reconstruct the conclusion with the same data, schema, business definitions, queries or calculations, system and complete runs.

01Claim, date, owner, analysis task, output contract, users, consequence and thresholds
02System, model, prompt, tools, packages, runtime and deployment versions
03Data sources, provenance, permissions, snapshots, cut-offs, formats and exclusions
04Schemas, keys, joins, types, units, business definitions and missing-data rules
05Reference queries, formulas, programs, intermediate values, tolerances and disagreements
06Parsing, schema linking, filters, execution environment, traces and retries
07Calculations, result sets, critical numbers, source support and sensitivity checks
08Chart data, scales, labels, filters, accessibility and narrative claims
09All tasks and files, failures, repeats, corrections, review time, latency and cost
10Target-workflow evidence, reconciliation, monitoring, change triggers, rollback and expiry

Common evidence states

Bind conclusions to the exact pipeline, task, data snapshot, schema, complete denominator, target workflow and date.

Supported

The exact pipeline meets calculation, source-support, critical-error, reliability, subgroup, human-effort, latency and cost thresholds on representative target data, with a current review date.

Conditional

Support holds only for named data sources, schema versions, task types, formats, languages, sizes or operating controls.

Mixed

Parsing, querying, calculation, support, visual communication, repeated-run reliability, latency or cost vary materially across slices.

Insufficient

Data or schema provenance, task contract, reference calculation, execution trace, complete denominator, target transfer or expiry is missing.

FUURAA analysisThe minimum decision unit for an AI data-analysis and tabular-reasoning capability claim is exact system, model, prompt, tool and execution-environment version × analysis task, data question, output contract, user and decision × data provenance, snapshot, permissions, schema, business definitions and time × parsing, schema linking, queries, code, formulas, computation and visualisation × result support, intermediate values, critical numbers, uncertainty, charts and narrative × all tasks, files, failures, variance, human correction, review time, latency and cost × target-workflow boundary and cut-off date. One correct number is an observation, not transferable proof of capability.

Primary sources and non-transfer boundaries

These sources constrain semi-structured table QA, cross-database text-to-SQL, table fact verification, financial numerical reasoning, large databases and end-to-end CSV analysis agents.

Sources rechecked 30 August 2026. Each retains its publication timing, role in this method and non-transfer boundary.

Published July 2015WikiTableQuestions — Compositional Semantic Parsing on Semi-Structured Tables

Introduced natural-language questions over diverse semi-structured Wikipedia tables, requiring operations such as lookup, comparison, aggregation and ordering.

BoundarySingle Wikipedia tables and denotation-based answers do not establish source discovery, spreadsheet formulas, database writes, chart quality, provenance or consequential business analysis.

Open primary source ↗Published October–November 2018Spider — Cross-Domain Semantic Parsing and Text-to-SQL

Tests complex text-to-SQL generalisation across unseen database schemas, with 10,181 questions and 5,693 unique SQL queries over 200 databases.

BoundaryCurated schemas, annotated questions and exact SQL comparison do not prove value grounding, dirty-data handling, query efficiency, permissions or safe execution in a live database.

Open primary source ↗Published 5 September 2019TabFact — Table-Based Fact Verification

Provides 118,000 human-annotated entailed or refuted statements grounded in 16,000 Wikipedia tables to test linguistic and symbolic table reasoning.

BoundaryBinary verification against one table does not establish open-ended analysis, missing-data treatment, causal claims, forecasting, visualisation or complete decision support.

Open primary source ↗Published 1 September 2021FinQA — Numerical Reasoning over Financial Data

Pairs expert-written questions over financial reports with executable reasoning programs, exposing the operations and intermediate steps behind numerical answers.

BoundarySelected reports, question templates and gold programs do not prove current financial accuracy, accounting-policy interpretation, auditability or fitness for investment or management decisions.

Open primary source ↗Published 4 May 2023BIRD — Large-Scale Database-Grounded Text-to-SQL

A Chinese-led benchmark adds large databases, dirty values, external knowledge and SQL-efficiency evaluation across 95 databases and 37 professional domains.

BoundaryBenchmark execution accuracy and efficiency do not establish access control, production schema drift, transaction safety, business-definition correctness or downstream decision quality.

Open primary source ↗Published 10 January 2024InfiAgent-DABench — Agents on Data Analysis Tasks

A Chinese-led benchmark evaluates data-analysis agents end to end on closed-form questions derived from CSV files while interacting with an execution environment.

BoundaryA bounded CSV set and closed-form grading do not establish exploratory analysis, ambiguous stakeholder requests, visual communication, governance, privacy or reliable long-running operations.

Open primary source ↗

Continue checking

Move from data analysis into document understanding, information extraction, factuality or the full library.

Enter the AI Knowledge Library Evaluate factuality Evaluate information extraction Evaluate document understanding and OCR Enter AI Evidence Atlas