Retired

Research

The LLM evaluation platform documented on this site is retired and no longer offered. Patronus AI studies model failures, world modeling and synthetic data to develop high-quality tasks and environments for reinforcement learning.

Patronus AI is a research lab developing training data and simulated environments for frontier AI. Its work spans reinforcement learning environments and digital world models for agents operating in software.

Recent papers

• GlobeBenchA benchmark for whether text world models keep track of state, rules and side effects.
  Oct 15, 2026 · arXiv 2610.22708 (opens in a new tab)
• SpeedrunBenchVideo game speedrunning as a test of whether agents can beat strategies humans already know.
  Sep 3, 2026 · arXiv 2610.08076 (opens in a new tab)
• FigmaTraceHuman design workflows in Figma, captured as training data for creative design tasks.
  Aug 20, 2026 · arXiv 2608.21460 (opens in a new tab)
• HalluWorldA controlled benchmark that measures hallucination against reference world models.
  May 19, 2026 · arXiv 2605.19341 (opens in a new tab)
• World Modeling for Agentic RLA language model that stands in for an environment while an agent trains.
  May 7, 2026 · arXiv 2607.16204 (opens in a new tab)

Earlier papers

• TRAILA benchmark for finding the step where an agent went wrong.
  May 13, 2025 · arXiv 2505.08638 (opens in a new tab)
• GLIDERA small, explainable judge model for grading LLM interactions.
  Dec 18, 2024 · arXiv 2412.14140
• LynxAn open-source model for detecting hallucinations in retrieval-augmented generation.
  Jul 11, 2024 · arXiv 2407.08488
• FinanceBenchOpen-book financial question answering about publicly traded companies.
  Nov 20, 2023 · arXiv 2311.11944
• SimpleSafetyTests100 prompts for finding critical safety risks in language models.
  Nov 14, 2023 · arXiv 2311.08370 (opens in a new tab)