Test and evaluate prompts, agents and RAG systems.
Test your prompts, agents, and RAGs. Red teaming, pentesting and vulnerability scanning for AI. Compare GPT, Claude, Gemini, DeepSeek and more.
Pricing: not verified yet, check the tool's site
Engineering note: Declarative configs with CLI and CI/CD integration, so evals can run as a pipeline step. Repo says it is used by OpenAI and Anthropic.
The LLM Evaluation Framework.
Pricing: not verified yet, check the tool's site
Engineering note: Repo lives under the confident-ai org. Check the repo docs for the metric list before assuming it covers agent-specific cases.
Supercharge Your LLM Application Evaluations.
Pricing: not verified yet, check the tool's site
Engineering note: The repo has moved to the vibrantlabsai org (GitHub redirects the old explodinggradients path). Pin to the current org URL in your docs.
A framework for large language model evaluations.
Pricing: not verified yet, check the tool's site
Engineering note: Published by the UK Government BEIS org on GitHub. A framework to write evals as code.
A framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
Pricing: not verified yet, check the tool's site
Engineering note: Combines a framework with a registry of benchmarks. Useful as a reference for eval structure.