Judge Evaluators (UI)

The following guide describes how to create a new Judge Evaluator with user defined criteria to tailor evaluations to your use case and preferences.

Create a Judge Evaluator in the UI

1. Create an Evaluator in the UI

From the Evaluators view, click on "Define your own Criteria" in the top right.

The Pass Criteria specifies to the judge evaluator how to evaluate your LLM's behavior. Pass criteria should be written in a declarative tone, eg. "MODEL OUTPUT should contain ..." or "MODEL OUTPUT should be free from...".

Your pass criteria must refer to at least one of the following entities in your criteria description. You may refer to multiple entities.

• MODEL OUTPUT: The output of your task. In most cases, this is the text that will be evaluated for the requirement. For example, this can be the output of your chat assistant.
• USER INPUT: The input to your task. For example, a user query to your LLM application.
• RETRIEVED CONTEXT: Context associated with the task. This can be retrieved document chunks, conversational histories, or any metadata that is relevant to the task. Often MODEL OUTPUT is assessed against the RETRIEVED CONTEXT field.
• GOLD ANSWER: The gold answer or gold label, if provided. An example criteria using the gold answer checks for semantic similarity.

See How entities reach the Judge for how these tags map to evaluation request fields and when an entity is omitted from the evaluation.

2. Validate Pass Criteria

Click on the Validate Pass Criteria assistant to check that there are no spelling issues or ambiguities present in the requirements you've provided.

If all looks good, click "save" to register the new profile.

3. Use your New Judge Evaluator

You can now use your new evaluator! You can test it in the playground by clicking on the playground.

Here, our comprehensive-bias-detection evaluator is able to catch a biased response in a model output. You can click View Code for each query:

import os
import patronus
from patronus.evals import RemoteEvaluator
 
patronus.init(
    # This is the default and can be omitted
    api_key=os.environ.get("PATRONUS_API_KEY")
)
 
patronus_evaluator = RemoteEvaluator("judge", "comprehensive-bias-detection")
 
result = patronus_evaluator.evaluate(
 
)
 
print(result)

How entities reach the Judge

Each entity tag in your pass criteria maps to a field on the evaluation request:

The Judge sees an entity only when both conditions hold: the tag appears in the pass criteria, and the corresponding request field is populated. Two rules follow:

• A tag referenced in your criteria must be populated in the evaluation request. A referenced field left empty is omitted from the evaluation, and the Judge fails criteria that require a missing entity.
• Fields sent but not referenced by the criteria are ignored. For example, data passed in task_context is not visible to the Judge unless the criteria mentions RETRIEVED CONTEXT.

Optional entities

If an entity is only available for some of your data, handle its absence in the criteria itself rather than leaving the reference unconditional:

If GOLD ANSWER is provided, the MODEL OUTPUT must agree with it. Otherwise, the MODEL OUTPUT must be polite and relevant to the USER INPUT.

Rows without a gold_answer are then evaluated against the fallback condition instead of failing on the missing entity.