Documentation

Deprecated Fields Guide

Suggest Edits

🚧

We are deprecating API input and output fields across our APIs. We highly recommend you do not use the deprecated fields as they will be changed in the future.

For now, all deprecated API input fields are supported, and all deprecated API output fields are included in the output so as not to break API consumers.

These are the APIs with deprecated input and output fields:

Evaluate: POST /v1/evaluate

Request Body:

  • criteria - replaced by evaluators
  • evaluators[..].evaluator_id - replaced by evaluators[..].evaluator. This field accepts both direct references to evaluators (evaluator_id) and evaluator aliases.
  • retrieved_context - replaced by evaluated_model_retrieved_context
  • input - replaced by evaluated_model_input
  • output - replaced by evaluated_model_output
  • label - replaced by evaluated_model_gold_answer
  • explain - replaced by evaluators[..].explain_strategy
# New Fields
{
    "evaluators": [
        {
            "evaluator": "custom",
            "profile_name": "system:is-code",
            "explain_strategy": "always"
        }
    ],
    "evaluated_model_retrieved_context": ["abc"],
    "evaluated_model_input": "Say hello world in Python",
    "evaluated_model_output": "print('hello world')",
    "evaluated_model_gold_answer": "print('hello world')"
}

# Deprecated Fields
{
    "criteria": [
        {
            "criterion_id": "custom",
            "profile_name": "system:is-code"
        }
    ],
    "retrieved_context": ["abc"],
    "input": "Say hello world in Python",
    "output": "print('hello world')",
    "label": "print('hello world')",
    "explain": true
}

Response Body:

  • results[..].criterion_id - replaced by results[..].evaluator_id
  • results[..].result{*} - replaced by results[..].evaluation_result
  • results[..].evaluation_result[..] - the following fields are deprecated:
    • system_prompt - replaced by evaluated_model_system_prompt
    • retrieved_context - replaced by evaluated_model_retrieved_context
    • input - replaced by evaluated_model_input
    • output - replaced by evaluated_model_output
    • label - replaced by evaluated_model_gold_answer
    • explain - replaced by explain_strategy
    • score_normalized. Please use score_raw instead which provides more flexibility.
    • additional_info.score_raw - moved outside of additional_info to the root of evaluation_result object.

Here's an example of the updated response body after the migration:

# New Response Body
{
    "results": [
        {
            "evaluator_id": "custom-large-2024-08-08",
            "profile_name": "system:is-code",
            "status": "success",
            "error_message": null,
            "evaluation_result": {
                "id": "112886856302501115",
                "app": "default",
                "created_at": "2024-08-19T19:09:25.048106Z",
                "evaluator_id": "custom-large-2024-08-08",
                "profile_name": "system:is-code",
                "evaluated_model_system_prompt": null,
                "evaluated_model_retrieved_context": [
                    "abc"
                ],
                "evaluated_model_input": "Say hello world in Python",
                "evaluated_model_output": "print('hello world')",
                "evaluated_model_gold_answer": "print('hello world')",
                "explain_strategy": "always",
                "pass": true,
                "score_raw": 1.0,
                "additional_info": {
                    "positions": null,
                    "extra": null,
                    "confidence_interval": null
                },
                "explanation": "The input asks for a Python code snippet to say 'hello world'. The output provides the code snippet `print('hello world')`, which meets the pass criteria as it is purely a code snippet without any additional text. Therefore, the pass flag is 1.",
                "evaluation_duration": "PT2.055S",
                "explanation_duration": "PT2.049S",
                "evaluation_run_project_id": null,
                "evaluation_run_id": null,
                "evaluator_family": "custom",
                "evaluator_profile_public_id": "07a39005-da80-48b1-89f5-17f5fdc8dfe8",
                "evaluated_model_id": null,
                "evaluated_model_name": null,
                "evaluated_model_provider": null,
                "evaluated_model_params": null,
                "evaluated_model_selected_model": null,
                "dataset_id": null,
                "dataset_sample_id": null,
                "tags": null,
                "external": false,
            },
        }
    ]
}

Get Evaluation Result: GET /v1/evaluation-results/{id}

Request Body: No Changes

Response Body:

  • evaluation_results - selected fields are deprecated:
    • system_prompt - replaced by evaluated_model_system_prompt
    • retrieved_context - replaced by evaluated_model_retrieved_context
    • input - replaced by evaluated_model_input
    • output - replaced by evaluated_model_output
    • label - replaced by evaluated_model_gold_answer
    • explain - replaced by explain_strategy
    • score_normalized- will be deleted in favor of score_raw, which provides more flexibility
    • additional_info.score_raw - moved outside of additional_info to the root of evaluation_result object.

Create Evaluator Profile: POST /v1/evaluator-profiles

Request Body:

  • evaluator_id - replaced by evaluator_family

Response Body:

  • evaluator_id - replaced by evaluator_family
# New Response Body
{
    "evaluator_family": "custom", 
    "name": "my-test-profile", 
    "config": {
        "pass_criteria": "The MODEL OUTPUT should include the word test.",
        "active_learning_enabled": false
    },
    "description": "Check if test model"
}
# Old Response Body
{
    "evaluator_id": "custom-large-2024-05-16",
    "name": "my-test-profile", 
    "config": {
        "pass_criteria": "The MODEL OUTPUT should include the word test.",
        "active_learning_enabled": false
    },
    "description": "Check if test model"
}

List Evaluator Profiles: GET /v1/evaluator-profiles

Request Body:

  • evaluator_id - replaced by evaluator_family

Response Body:

  • evaluator_id - replaced by evaluator_family

Archive Evaluator Profile: PATCH /v1/evaluator-profiles/{public_id}/archive

Request Body:

  • evaluator_id - replaced by evaluator_family

Response Body:

  • evaluator_id - replaced by evaluator_family

Add Evaluator Profile Revision: POST /v1/evaluator-profiles/{public_id}/revision

Request Body:

  • evaluator_id - replaced by evaluator_family

Response Body:

  • evaluator_id - replaced by evaluator_family

Search Evaluation Results:POST /v1/evaluation-results/search

Request Body:

  • criterion_profile_name - replaced by profile_name
  • evaluator_profile_name - replaced by profile_name

Response Body:

  • results[..] - selected fields are deprecated:
    • system_prompt - replaced by evaluated_model_system_prompt
    • retrieved_context - replaced by evaluated_model_retrieved_context
    • input - replaced by evaluated_model_input
    • output - replaced by evaluated_model_output
    • label - replaced by evaluated_model_gold_answer
    • explain - replaced by explain_strategy
    • score_normalized . Please use score_raw instead which provides more flexibility.
    • additional_info.score_raw - moved outside of additional info to the root of EvaluationResult object.

Updated 2 months ago