Active Learning

Active learning is an option a user can choose during the process of creating a custom profile. Active learning increases the performance of the custom evaluator using feedback provided by the user on evaluation results.

On an evaluation, a sample of the feedback you've given is used to improve the evaluation performance.

---

Providing Feedback

A user can provide feedback to the results of:

• Evaluation runs executed from the context of the application (Feedback on Evaluation field),
• Evaluation API responses that were run with the strategy "capture": "always".

📘

Good to Know

Feedback is a type of labeling mechanism, where a user can agree (thumbs up) or disagree (thumbs down) with a result provided by any evaluator.

Feedback provisioning is available for all types of evaluation defined in the Evaluators page. Available feedback is processed and used for the transformation of relevant evaluation prompts.

---

Feedback Examples

The following examples show a correct way of labeling evaluations with feedback for no-comma profile of the custom evaluator.

Pass Criteria: "The output cannot contain a comma character."
Evaluator Output: "This output contains comma character."
Evaluator Result: "Pass"

Pass Criteria: "The output cannot contain a comma character."
Evaluator Output: "This output, contains comma character."
Evaluator Result: "Fail"

Pass Criteria: "The output cannot contain a comma character."
Evaluator Output: "This output doesn't contain comma character."
Evaluator Result: "Pass"

Pass Criteria: "The output cannot contain a comma character."
Evaluator Output: "This output, doesn't contain comma character."
Evaluator Result: "Pass"

---

Setup

To use the active learning functionality within the application, a profile needs to be created for the evaluator. For more details, visit Profiles page.

During profile creation, or after it is created, turn on the Active Learning enabled option.

🚧

Take a Note

Every defined profile is immutable. Because of that, a profile with an enabled active learning will use it all the time. To have an option of running the same evaluation without active learning, create a second profile with the same pass_criteria definition and without enabling Enable active learning flag.

Parameter Description:

• Enable active learning: A flag that sets up active learning
• Number of samples injected to prompt: Feedback samples used for extending evaluation prompts (max value of 10)
• No. of samples with positive feedback: Feedback samples with thumbs up less/equal to the above (max value of 10)