Percival

The LLM evaluation platform documented on this site is retired and no longer offered. Percival is retired. Related research on patronus.ai is TRAIL, a benchmark for finding the step where an agent went wrong.

Patronus AI is a research lab developing training data and simulated environments for frontier AI. Its work spans reinforcement learning environments and digital world models for agents operating in software.

TRAIL on patronus.ai (opens in a new tab)Read on arXiv (opens in a new tab)

Related research

TRAIL

TRAIL is a benchmark of 148 human-annotated agent traces and an error taxonomy, built to test whether models can find where an agentic workflow went wrong. The traces come from GAIA and SWE-bench agents.

Traces: 148, annotated by hand
Errors: 841 labeled in them
Best score: 11% for any model tested

Cite

TRAIL: Trace Reasoning and Agentic Issue Localization

Darshan Deshpande, Varun Prashant Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan, Rebecca Qian. May 13, 2025. Also as a PDF (opens in a new tab).

Darshan Deshpande, Varun Prashant Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan, Rebecca Qian. TRAIL: Trace Reasoning and Agentic Issue Localization. arXiv:2505.08638, 2025.