Test Dataset Generation Overview

Patronus is also proud to offer Test Dataset Generation. Please reach out to us if you are interested in our automated data generation. The team will talk through your use case, collect relevant documents, and generate the data you need efficiently and at scale.

Dataset Generation Types

Our dataset generation feature is flexible, and covers a wide range of use cases and generation formats. Generated datasets are high quality and have diverse distribution. Our research team has developed the following types of dataset generation methods.

Document Based Generation

We generate customized, domain-specific datasets over multiple modalities such as text, tables, code and images. Customers share their data in supported formats (e.g., PDF, JSON, JPG) and we extract relevant content from the data to ground the generation.

These prompts can be only questions or question-answer pairs with answers grounded in the data. We can ensure good coverage or focus prompts around a user-provided list of topics.

Criteria Based Generation

Criteria driven generation creates sets of prompts to assess model behavior against a particular criterion (e.g., whether models output copyright, unsafe information, PII).

These prompts can either be adversarial or focused on ensuring good coverage with test cases. For coverage, we take a curriculum extraction approach and label prompts with categories in the final dataset.

Adversarial Attacks

We can generate adversarial prompts for different model families based on red-teaming techniques. These include various methods to recursively branch and iterate to find prompt improvements that will lead a model to output unsafe information. The prompts consist of questions or are of completion style. They can be used to test different aspects of model safety.

Conversational Datasets

We can generate multi-turn conversational datasets with different user personas. Here is an example of a conversational transcript that we constructed:

Jordan Thompson: Absolutely, Emily. Being able to play the sport I love and get paid for it is a dream come true. Another pro is the platform it provides. As a professional athlete, I have the chance to inspire and influence others, especially young fans who look up to me.

Emily Stevens: That's a great point, Jordan. Professional athletes have the power to make a positive impact on society. Another pro is the financial aspect. Many athletes earn substantial salaries and have access to lucrative endorsement deals, providing financial security for themselves and their families.

Jordan Thompson: Yes, the financial aspect is definitely a pro. However, it's important to acknowledge the cons as well. One major challenge is the immense pressure to perform consistently at a high level. The expectations from fans, coaches, and sponsors can be overwhelming at times.

Perturbations and Data Augmentation

We support various perturbation methods that can be used to perturb existing datasets to create new evaluation or fine-tuning data. We support various perturbations for language, demographic, style & tone, semantic and syntactic. These generic-purpose perturbers can be used across different datasets across domains. Example: A semantic perturber can be used to introduce subtle variations in the text to introduce hallucinations.