OWASP Assessment

What is the OWASP AI Risk Assessment?

The OWASP Top 10 for LLM Applications Cybersecurity and Governance Checklist aims to help leaders develop a comprehensive list of critical areas and tasks needed to defend and protect the organization as they develop a Large Language Model strategy.

How Patronus Supports OWASP Assessments

Patronus supports OWASP assessments by providing test datasets and corresponding evaluators for each category.

OWASP testing sets are provided in Patronus Datasets and prepended with owasp*, eg. owasp-llm09-overreliance-toxic-prompts.

We support the following OWASP categories:

LLM01: Prompt Injection Attacks

Prompt Injection Vulnerability occurs when an attacker manipulates a large language model (LLM) through crafted inputs, causing the LLM to unknowingly execute the attacker's intentions. owasp-llm01-prompt-injection dataset contains jailbreaking prompts that test if a model is susceptible to prompt injection attacks.

LLM02: Insecure Outputs

Insecure Output refers specifically to insufficient validation, sanitization, and handling of the outputs generated by LLMs. Models can produce outputs that contain insecure code which could be used for various cybersecurity exploits. owasp-llm02-insecure-outputs consists of prompts to test if an LLM would produce outputs that can be harmful if executed directly.

LLM07: Data Leakage

LLMs may inadvertently reveal confidential data in their responses, leading to unauthorized data access, privacy
violations, and security breaches. Its crucial to implement data sanitization and strict user policies to mitigate this. owasp-llm07-data-leakage aims to find these vulnerabilities. The prompts ask for a range of personal and sensitive information ranging from contact information, medical history to financial details. We recommend using our pii evaluator along with this dataset.

LLM08: Excessive Agency

LLM-based systems may undertake actions leading to unintended consequences. The issue arises from
excessive functionality, permissions, or autonomy granted to the LLM-based systems. To test this, we provide owasp-llm08-excessive-agency, consisting of prompts to evaluate the actions the model can take, such as providing discounts, having access to security credentials etc.