FUURAA AI Knowledge Library · Evaluation guide
Use six evidence gates to turn “completed a web task” into a reproducible starting state, observation surface, action authority, independent outcome, side-effect record, recovery test and transfer boundary. Apply the guide in benchmark reading, research design, engineering review and pre-release verification.
Turn task success back into a state transition
An interpretable web-agent claim must connect user intent, starting state, actual observation, authorised action, independently verified final state, and every residual or side effect. If one link is missing, a success rate may count seeded state, hidden assistance, wrong targets or duplicate effects as capability.
Applicability boundaryThis is a public research method, not a FUURAA product-capability claim or an assessment of any specific agent, model, provider or benchmark. It is not procurement advice, a performance warranty, security certification, audit opinion, or legal or compliance conclusion.
Six rejectable evidence gates
Minimum failure matrix
Record the affected task, actual-versus-expected state, authority boundary, whether the evaluator detected it, the narrowest conclusion that remains, and the scope that must be rerun after repair.
Record the affected task, actual-versus-expected state, authority boundary, whether the evaluator detected it, the narrowest conclusion that remains, and the scope that must be rerun after repair.
Record the affected task, actual-versus-expected state, authority boundary, whether the evaluator detected it, the narrowest conclusion that remains, and the scope that must be rerun after repair.
Record the affected task, actual-versus-expected state, authority boundary, whether the evaluator detected it, the narrowest conclusion that remains, and the scope that must be rerun after repair.
Record the affected task, actual-versus-expected state, authority boundary, whether the evaluator detected it, the narrowest conclusion that remains, and the scope that must be rerun after repair.
Record the affected task, actual-versus-expected state, authority boundary, whether the evaluator detected it, the narrowest conclusion that remains, and the scope that must be rerun after repair.
Record the affected task, actual-versus-expected state, authority boundary, whether the evaluator detected it, the narrowest conclusion that remains, and the scope that must be rerun after repair.
Record the affected task, actual-versus-expected state, authority boundary, whether the evaluator detected it, the narrowest conclusion that remains, and the scope that must be rerun after repair.
Minimum web-agent evaluation record
Common evidence states
The exact task, state, observation, authority, action budget and independently verified outcome are reconstructable, including side effects and repeated runs.
The result is credible only for named sites, versions, accounts, observation channels, permissions, tasks or evaluation windows.
Task completion, side effects, languages, sites, runs or recovery tests point in different directions.
Initial state, observation, authority, assistance, evaluator or outcome evidence is missing; a success trace alone is not evidence.
FUURAA analysisThe evidence most easily lost in web-agent evaluation is not one click but the chain from intent to consequence. FUURAA recommends preserving every capability claim as task and starting state × observation surface × authority and budget × independent outcome × date, with side effects, human assistance, recovery and expiry inside the conclusion. Task completion rate can describe an outcome; only this chain can show whether it came from authorised, reviewable and transferable action.
Primary sources and non-transfer boundaries
Sources rechecked 22 August 2026. Each retains its publication timing, role in this method and non-transfer boundary.
Introduces reproducible, functional websites and evaluates agents by end-to-end task correctness rather than isolated action prediction.
BoundaryIts tasks, websites and reported model results are a dated research snapshot; they do not establish current performance on arbitrary public websites.
Open primary source ↗24 January 2024VisualWebArena — Evaluating Multimodal Agents on Realistic Visual Web TasksExtends realistic web evaluation to visually grounded tasks and shows why text or DOM-only observation can miss task-critical evidence.
BoundaryVisual grounding results are conditional on the benchmark sites, observation setup, models and evaluators used in the paper.
Open primary source ↗11 April 2024OSWorld — Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer EnvironmentsUses real computer environments and execution-based evaluators to test whether final application state—not merely a plausible trace—satisfies a task.
BoundaryDesktop-task coverage and historical results do not prove safety, reliability or suitability for another operating system, application or workflow.
Open primary source ↗8 February 2024WebLINX — Real-World Website Navigation with Multi-Turn DialogueContributes large-scale expert demonstrations across real websites and exposes the difficulty of generalising navigation to unseen sites.
BoundaryDemonstration data measures named navigation settings; it does not verify live transaction outcomes or authorise action on third-party services.
Open primary source ↗12 March 2024WorkArena — How Capable Are Web Agents at Solving Common Knowledge Work Tasks?Frames browser-agent evaluation around reproducible knowledge-work tasks and the policies, pages and workflows of a functional enterprise platform.
BoundaryResults for the benchmark platform and task set cannot be transferred directly to every enterprise system, role or permission model.
Open primary source ↗7 July 2024WorkArena++ — Towards Compositional Planning and Reasoning-based Common Knowledge Work TasksAdds realistic compositional tasks that require multiple application steps, planning and reasoning rather than a single page interaction.
BoundaryCompositional difficulty is benchmark-specific; it does not by itself measure authority control, side effects, recovery or production readiness.
Open primary source ↗Continue checking