Dataset still available

FinanceBench

The LLM evaluation platform documented on this site is retired and no longer offered. FinanceBench is not part of it. It remains a public research dataset.

Patronus AI is a research lab developing training data and simulated environments for frontier AI. Its work spans reinforcement learning environments and digital world models for agents operating in software.

Open-book financial question answering

FinanceBench is a test suite for evaluating LLMs on open-book financial question answering. It comprises 10,231 questions about publicly traded companies, with corresponding answers and evidence strings. In the paper, GPT-4-Turbo used with a retrieval system incorrectly answered or refused to answer 81% of a sample of 150 cases.

Where to find the dataset

The paper describes FinanceBench and its open-source cases. Read it on arXiv (opens in a new tab) or as a PDF (opens in a new tab). The current home for FinanceBench is its research page on patronus.ai (opens in a new tab).

FinanceBench: A New Benchmark for Financial Question Answering

Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, Bertie Vidgen. Nov 20, 2023.

Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, Bertie Vidgen. FinanceBench: A New Benchmark for Financial Question Answering. arXiv:2311.11944, 2023.