Hallucination Detection (Lynx)

Lynx is a State-of-the-Art hallucination detection model. Lynx is capable of advanced reasoning on challenging real-world hallucination scenarios. Lynx is novel for several reasons:

• Trained on challenging perturbed data.
• Trained on multiple real world domains, such as finance and medicine.
• Outperforms GPT-4o, Claude-3-Sonnet and other open source models.

The Hallucination Problem

Deploying RAG applications to production can be risky due to hallucinations. They occur when LLM outputs are not faithful to the retrieved contexts. Detecting hallucinations requires nuanced and complex reasoning. LLMs hallucinate more in specialized domains such as finance and medicine. Real-time, automated hallucination detection is important to make RAG systems reliable. However, this is challenging to achieve.

• Using GPT-4 and proprietary LLMs as judges to catch hallucinations is unreliable, costly and hard to scale.
• Human evaluation is expensive and not feasible in real-time AI workflows.
• Consequences of misinformation include incorrect medical diagnosis, poor financial advice, embarrassing outputs...and more!

Lynx

Lynx is the largest and first open source model that outperforms GPT-4o, Claude-3-Sonnet and LLama-3-70B on hallucination tasks. Lynx is available as 8B and 70B variants. The 8B parameter variant only has a 3% performance gap compared to GPT-4o.

To use Lynx, you can make the following request:

curl --location 'https://api.patronus.ai/v1/evaluate' \
--header 'X-API-KEY: YOUR_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
  "app": "default",
  "evaluators": [
    {
      "evaluator": "retrieval-hallucination-lynx",
      "explain_strategy": "always"
    }
  ],
  "evaluated_model_input": "Who are you?",
  "evaluated_model_output": "My name is Barry.",
  "evaluated_model_retrieved_context": ["I am John."],
  "tags": {"experiment": "question_answering"}
}'

HaluBench

To evaluate Lynx, we present HaluBench , a comprehensive hallucination evaluation benchmark, consisting of 15k samples sourced from various real-world domains. It consists of a high proportion of hard-to-detect, ambiguous hallucinations. Our experiment results show that Lynx outperforms GPT-4o, Claude-3-Sonnet, and closed and open-source LLM-as-a-judge models on HaluBench. We released Lynx, HaluBench and our evaluation code for public access.