Retired

Evaluators

The LLM evaluation platform documented on this site is retired and no longer offered. Its evaluators and evaluation API are retired.

Patronus AI is a research lab developing training data and simulated environments for frontier AI. Its work spans reinforcement learning environments and digital world models for agents operating in software.

What LLM evaluators are

LLM evaluators are tools or models that assess the output of an LLM for factual correctness, safety, style, and other quality aspects. The models can be trained specifically or be general-purpose, with evaluation-specific prompts. The guide on patronus.ai (opens in a new tab) covers evaluator types, use cases and the evaluation workflow.

Guides and papers

• LLM Evaluators: Tutorial & Best PracticesLearn about the various types and use cases of LLM evaluators and how to effectively incorporate them into your AI-based applications.
  Guide → (opens in a new tab)
• LLM As a JudgeChapter 4 of the LLM Testing guide.
  Guide → (opens in a new tab)
• GLIDERA small, explainable judge model for grading LLM interactions.
  Paper →
• Contrastive Decoding for LLM JudgesReducing score range bias when language models act as judges.
  Paper → (opens in a new tab)