Webhooks (Beta)

📘

Enterprise-only feature

If you are not on the Patronus Enterprise plan, but want access to Webhooks, reach out to the team at [email protected].

📘

Beta feature

This feature is currently in Beta and may not capture all events. We recommend monitoring your integration closely.

About

When particular events on the Patronus platform occur, like evaluation failures, you can notify external services through Webhooks.

You can find the Webhooks management page in the User Menu (accessed by clicking on your Account Name in the top left), or by following the link here.

Creating a Webhook

1. Navigate to the Webhooks management page through the User Menu or following this link.

2. 

   Click Create Webhook in the top-right corner. You should now see the Webhook creation screen:

3. 

   Type in the URL that Patronus will sent a POST request to with the payload about the specified event. For now, the only supported event type is evaluation.failed.

4. By default, the Event Webhook is set to Active, and you will start receiving POST requests at the specified URL when the event is triggered. You can optionally set it to Inactive, and enable the Webhook integration when you are ready.

5. Click Create - and you're all set!

   

6. You can Edit or Delete your integration at any time by clicking the three dot menu on a Webhook

   

API Shape

When the specified event happens, Patronus makes a POST API call to your Webhook URL. The body of the POST request has two objects, request and events:

Request Object

The request object provides metadata about the API call:

• id (string): A unique identifier for each API call.
• idempotency_key (string): A unique identifier of the Webhook request. This key remains constant across retries.

Note: If our system doesn't receive a timely response, it may retry the API request. Retried requests will have a new id but the same idempotency_key. Use the idempotency_key to ignore duplicate requests.

Events Array

The events array contains one or more event objects. These event objects may have different event types.

Each event object includes:

• id (string): A unique identifier for the event.
• object (string): The name of the object in the data field (e.g., evaluation_result).
• type (string): The event trigger (e.g., evaluation.failed).
• created (integer): Unix timestamp of event creation.
• data (object): Contains the event details. The key inside this object corresponds to the value specified in the object field.

Example Request Body

{
  "request": {
    "id": "req_TrKEAwnlisalRT", # Request unique ID generated for each api call
    "idempotency_key": "8c8b168b-56ec-47d8-bb47-0155fc1552ed" # Idempotency key generated for each webhook request
  },
  "events": [
    {
      "id": "evt_1Px8lr00YY7lRe4kbL9Y9ZRI", # `Event` unique id
      "object": "evaluation_result", # `Event` object type
      "type": "evaluation.failed", # `Event` type
      "created": 1725892851, # `Event` creation timestamp
      "data": {
        "evaluation_result": { # Patronus Evaluation Result
          ...
        }
      }
    }
  ]
}

Supported Event Types

• Evaluation Failures (evaluation.failed)

Currently, Patronus only supports notifying external services when an evaluation fails. In the API call made these services, the event type will be evaluation.failed.

Evaluation Failures (evaluation.failed)

Note, this event will only be fired by Experiment and Evaluation API failures. Evaluation Runs and Playground failures will not trigger the evaluation.failed event.

Example evaluation.failed request body

{
  "request": {
    "id": "req_TrKEAwnlisalRT", # Request unique ID generated for each api call
    "idempotency_key": "8c8b168b-56ec-47d8-bb47-0155fc1552ed" # Idempotency key generated for each webhook request
  },
  "events": [
    {
      "id": "evt_1Px8lr00YY7lRe4kbL9Y9ZRI", # `Event` unique id
      "object": "evaluation_result", # `Event` object type
      "type": "evaluation.failed", # `Event` type
      "created": 1725892851, # `Event` creation timestamp
      "data": {
        "evaluation_result": { # Patronus Evaluation Result
          "id": "112983595726627839",
          "app": "production-gpt4o",
          "created_at": "2024-09-05T21:11:31.515781Z",
          "evaluator_id": "retrieval-hallucination-small-2024-07-23",
          "profile_name": "system:detect-rag-hallucination",
          "evaluated_model_retrieved_context": [
            "Donald Trump used to be the president",
            "Joe Biden is the current president"
          ],
          "evaluated_model_input": "Who is the president of the US?",
          "evaluated_model_output": "Barack Obama.",
          "pass": false,
          "score_raw": 0.0,
          ... # additional `evaluation_result` fields not included for brevity
        }
      }
    }
  ]
}