Description
API ReferenceEvaluations

Evaluate

Requires either input or output field to be specified. Absence of both leads to an HTTP_422 (Unprocessable Entity) error.

POST
/v1/evaluate
/v1/evaluate

The Authorization access token

Authorization

X-API-KEY<token>

In: header

Request Body

application/jsonRequired

evaluators
Required
Evaluators

List of evaluators to evaluate against.

evaluated_model_system_promptEvaluated Model System Prompt

The system prompt provided to the LLM.

evaluated_model_retrieved_contextEvaluated Model Retrieved Context

Optional context retrieved from vector database. This is a list of strings, with the following restrictions:

  • Number of items must be less/equal than 50.
  • The sum of tokens in all elements must be less/equal than 120000, using o200k_base tiktoken encoding

evaluated_model_inputEvaluated Model Input

The input (prompt) provided to LLM.

evaluated_model_outputEvaluated Model Output

LLM's response to the given input.

evaluated_model_gold_answerEvaluated Model Gold Answer

Gold answer for given evaluated model input

captureCaptureOptions

Capture evaluation result based on given option, default is none:

  • all captures the result of all evaluations (pass + failed).
  • fails-only captures the evaluation result when evaluation failed.
  • none does not capture evaluation result
Default: "all"Value in: "all" | "fails-only" | "none"

appApp

Assigns evaluation results to the app.

  • app cannot be used together with experiment_id.
  • If app and experiment_id is omitted, app is set automatically to "default" on capture.
  • Automatically creates an app if it doesn't exist.
  • Only relevant for captured results. If will capture the results under given app.

experiment_idExperiment Id

Assign evaluation results to the experiment.

  • experiment_id cannot be used together with app.
  • Only relevant for captured results. If will capture the results under experiment.

dataset_idDataset Id

The ID of the dataset from which the evaluated sample originates. This field serves as metadata for the evaluation. This endpoint does not ensure data consistency for this field. There is no guarantee that the dataset with the given ID is present in the Patronus AI platform, as this is a self-reported value.

dataset_sample_idDataset Sample Id

The ID of the sample within the dataset. This field serves as metadata for the evaluation. This endpoint does not ensure data consistency for this field. There is no guarantee that the dataset and sample are present in the Patronus AI platform, as this is a self-reported value.

tagsTags

Tags are key-value pairs used to label resources

confidence_interval_strategyConfidenceIntervalStrategies

Create confidence intervals based on one of the following strategies:

  • 'none': returns None
  • 'full-history': calculates upper boundary, median, and lower boundary of confidence interval based on all available historic records.
  • 'generated': calculates upper boundary, median, and lower boundary of confidence interval based on on-flight generated sample of evaluations.
Default: "none"Value in: "none" | "full-history"

evaluated_model_attachmentsEvaluated Model Attachments

Optional list of attachments to be associated with the evaluation sample. This will be added to all evaluation results in this request. Each attachment is a dictionary with the following keys:

  • url: URL of the attachment.
  • media_type: Media type of the attachment (e.g., "image/jpeg", "image/png").
  • usage_type: Type of the attachment (e.g., "evaluated_model_system_prompt", "evaluated_model_input").

curl -X POST "https://api.patronus.ai/v1/evaluate" \
  -H "X-API-KEY: <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "evaluators": [
      {
        "evaluator": "string",
        "criteria": "string",
        "explain_strategy": "never"
      }
    ],
    "evaluated_model_system_prompt": "string",
    "evaluated_model_retrieved_context": [
      "string"
    ],
    "evaluated_model_input": "string",
    "evaluated_model_output": "string",
    "evaluated_model_gold_answer": "string",
    "capture": "all",
    "app": "string",
    "experiment_id": "string",
    "dataset_id": "string",
    "dataset_sample_id": 0,
    "tags": {},
    "confidence_interval_strategy": "none",
    "evaluated_model_attachments": [
      {
        "url": "http://example.com",
        "media_type": "image/jpeg",
        "usage_type": "evaluated_model_system_prompt"
      }
    ]
  }'

Successful Response

{
  "results": [
    {
      "evaluator_id": "string",
      "status": "string",
      "error_message": "string",
      "evaluation_result": {
        "id": "string",
        "log_id": "14b5977f-7a80-40ca-bb79-eca6c2abdb34",
        "app": "string",
        "project_id": "405d8375-3514-403b-8c43-83ae74cfe0e9",
        "experiment_id": "string",
        "created_at": "2019-08-24T14:15:22Z",
        "evaluator_id": "string",
        "profile_name": "string",
        "evaluated_model_system_prompt": "string",
        "evaluated_model_retrieved_context": [
          "string"
        ],
        "evaluated_model_input": "string",
        "evaluated_model_output": "string",
        "evaluated_model_gold_answer": "string",
        "evaluated_model_attachments": [
          {
            "url": "string",
            "media_type": "string",
            "usage_type": "string"
          }
        ],
        "explain_strategy": "never",
        "pass": true,
        "score_raw": 0,
        "text_output": "string",
        "additional_info": {
          "positions": [
            null
          ],
          "extra": {},
          "confidence_interval": {
            "strategy": "string",
            "alpha": 0,
            "lower": 0,
            "median": 0,
            "upper": 0
          }
        },
        "evaluation_metadata": {},
        "explanation": "string",
        "evaluation_duration": "string",
        "explanation_duration": "string",
        "evaluation_run_id": 0,
        "evaluator_family": "string",
        "evaluator_profile_public_id": "fe6c9202-ffdf-40e1-8f9b-304d0cb5a8db",
        "evaluated_model_id": "string",
        "evaluated_model_name": "string",
        "evaluated_model_provider": "string",
        "evaluated_model_params": {},
        "evaluated_model_selected_model": "string",
        "dataset_id": "string",
        "dataset_sample_id": 0,
        "tags": {
          "property1": "string",
          "property2": "string"
        },
        "external": true,
        "favorite": true,
        "evaluation_feedback": true,
        "usage_tokens": 0,
        "metric_name": "string",
        "metric_description": "string",
        "evaluation_type": "string",
        "annotation_criteria_id": "e5d4b10b-c239-4b00-9620-5b9e8428bf29"
      }
    }
  ]
}