GLIDER

GLIDER is a 3B parameter custom evaluator model trained by Patronus AI. GLIDER can score any text input and associated context on arbitrary user defined criteria.

• It shows higher Pearson’s correlation than GPT-4o on FLASK
• It outperforms prior judge models, achieving comparable performance to LLMs 17× its size.
• Supports fine-grained scoring, multilingual reasoning and span highlighting.

We train and align a Phi-3.5-mini-instruct model on synthetic data that spans 183 different research and industrial
evaluation metrics from 685 relevant domains of application to prove that Grading LLM Interactions and Decisions using Explainable Ranking can help improve performance. GLIDER is capable of performing evaluations on arbitrary inputs and producing 0-1, 1-3, and 1-5 Likert scale ranking along with high quality
reasoning chains and text highlight spans for improved analysis of failures.

Training

We use a mixture of synthetic datasets and openly available datasets to train the model.

We created a detailed taxonomy of potential metrics to cover along with their definitions covering 685 unique domains like finance, medicine and technology to more creative domains like art, fashion and films. To ensure that the model does not overfit to a single evaluation field like user input or model output, we diversify our dataset by forcing associations arbitrarily between random tag names representing inputs, outputs, contexts and gold answer. We create pointwise data points and then prompt the model to output a correct and incorrect score and reasoning for the generated instance. This pairwise data generation is used for RLAIF alignment training phase where we use the rejected samples to lower their probabilities and increase the probabilities of the chosen samples.

We choose phi-3.5-mini-instruct as our base model. We perform supervised fine-tuning (SFT) for one epoch. Following this, we align this model with the APO zero loss since our synthetic data contains noise and APO has been shown to be more robust in such situations. In addition to this preference optimization loss, we add a standard cross entropy term, ensuring that the model continues to capture data nuances in the alignment phase.

To read more details about the data generation and training, refer to our paper: https://arxiv.org/abs/2412.14140

Results

GLIDER achieves state of the art performance on the FLASK benchmark, beating GPT-4o while still performing close to models 17× its size on the Feedback Collection dataset.

Pearson correlation for various models on ranking tasks against human ratings

Table 1: bolded text indicates best overall and underline indicates best open-source judge model.

Performance (F1 score) comparison of models on pairwise ranking datasets.

Table 2: Bolded text indicates best overall and underline indicates best open-source judge model. Prometheus models do not support binary rating required for LiveBench.