31 tools across MCP, evals, vector databases, observability and agent frameworks. Every fact is checked against the official source.
Servers, SDKs and registries for the Model Context Protocol.
5 toolsTest and evaluate prompts, agents and RAG systems.
5 toolsStorage and retrieval for embeddings.
6 toolsTracing, monitoring and evals for LLM apps and agents.
6 toolsLibraries for building agents and multi-agent workflows.
9 toolsModel Context Protocol Servers.
Pricing: not verified yet, check the tool's site
Engineering note: Collection of MCP servers maintained under the official modelcontextprotocol GitHub org. Start here to see how reference servers are structured before writing your own.
The official Python SDK for Model Context Protocol servers and clients.
Pricing: not verified yet, check the tool's site
Engineering note: Official SDK, so it tracks the protocol directly. Covers both the server and client side in Python.
The fast, Pythonic way to build MCP servers and clients.
Pricing: not verified yet, check the tool's site
Engineering note: Python library for building MCP servers and clients. Compare its API against the official SDK before choosing.
Visual testing tool for MCP servers.
Pricing: not verified yet, check the tool's site
Engineering note: Use it to poke at a server you are building without wiring up a full agent first.
Registry and connection layer that connects agents to thousands of tools and services, with auth, credentials and sessions handled for you.
Pricing: not verified yet, check the tool's site
Engineering note: Homepage lists hosted MCP servers with usage counts and a CLI (npx smithery) for adding and calling tools. Its connection auth is powered by the open-source agent.pw. Check how credentials are stored before sending real accounts through any hosted gateway.
Test your prompts, agents, and RAGs. Red teaming, pentesting and vulnerability scanning for AI. Compare GPT, Claude, Gemini, DeepSeek and more.
Pricing: not verified yet, check the tool's site
Engineering note: Declarative configs with CLI and CI/CD integration, so evals can run as a pipeline step. Repo says it is used by OpenAI and Anthropic.
The LLM Evaluation Framework.
Pricing: not verified yet, check the tool's site
Engineering note: Repo lives under the confident-ai org. Check the repo docs for the metric list before assuming it covers agent-specific cases.
Supercharge Your LLM Application Evaluations.
Pricing: not verified yet, check the tool's site
Engineering note: The repo has moved to the vibrantlabsai org (GitHub redirects the old explodinggradients path). Pin to the current org URL in your docs.
A framework for large language model evaluations.
Pricing: not verified yet, check the tool's site
Engineering note: Published by the UK Government BEIS org on GitHub. A framework to write evals as code.
A framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
Pricing: not verified yet, check the tool's site
Engineering note: Combines a framework with a registry of benchmarks. Useful as a reference for eval structure.
Vector database (repo: qdrant/qdrant) with a managed Qdrant Cloud and a free OSS option.
Pricing: Cloud: free tier forever (single node, 0.5 vCPU / 1GB RAM / 4GB disk). Standard tier is usage-based (vCPU, memory, storage, backups, paid inference tokens, billed hourly). Premium: minimum spend. Source: qdrant.tech/pricing
Engineering note: Free tier is capped at 1GB RAM and 4GB disk and has no high availability. Paid Standard tier lists a 99.5% uptime SLA. Migration tool from OSS to Cloud is offered.
Open-source search infrastructure for AI; Chroma Cloud offers serverless vector, full-text and metadata search.
Pricing: Cloud Starter $0/mo + usage (with $5 free credits); Team $250/mo + usage; Enterprise custom. Usage: $2.50/GiB written, $0.33/GiB-month stored, $0.0075/TiB queried, $0.09/GiB returned. Source: trychroma.com/pricing
Engineering note: Pricing is purely usage-based on writes, storage, queries and egress, so estimate with their calculator. Chroma says data export is easy since it is open source.
Open-source vector database that stores objects and vectors, combining vector search with structured filtering.
Pricing: Cloud: Free plan (100k objects, 1GB memory, 10GB disk, 1 collection); Flex from $45/mo pay-as-you-go; Premium from $400/mo (contact sales). Source: weaviate.io/pricing
Engineering note: Pricing is by total vector dimensions plus storage, with compression affecting price. Hybrid search is available on all plans including free. Site notes a September 2026 price book update.
High-performance, cloud-native vector database built for scalable vector ANN search.
Pricing: not verified yet, check the tool's site
Engineering note: Described as cloud-native and built for scalable approximate nearest neighbor search.
Open-source vector similarity search for Postgres.
Pricing: not verified yet, check the tool's site
Engineering note: Runs inside Postgres, so embeddings can sit next to your relational data and use your existing database ops.
Developer-friendly OSS embedded retrieval library for multimodal AI.
Pricing: not verified yet, check the tool's site
Engineering note: Embedded library rather than a separate server, which changes the deployment model compared with Qdrant or Milvus.
Open source agent evals and observability: trace, evaluate and improve LLM applications.
Pricing: Cloud: Hobby free (50k units/mo, 30 days data, 2 users); Core $29/mo; Pro $199/mo; Enterprise $2,499/mo. Extra usage from $8 per 100k units. Self-hosting is free. Source: langfuse.com/pricing
Engineering note: Billing counts traces, observations and scores as units, so a chatty multi-step agent consumes units fast. Supports Python/JS SDKs and OpenTelemetry. Includes prompt management and datasets.
AI observability and evaluation.
Pricing: not verified yet, check the tool's site
Engineering note: Repo lives under the Arize-ai org. Verify the self-host and hosted options in their docs.
Open source LLM observability platform. One line of code to monitor, evaluate and experiment.
Pricing: Hobby: 10,000 free requests, 7-day retention, 1 seat. Pro $79/mo; Team $799/mo; both usage-based on top. Source: helicone.ai/pricing
Engineering note: Integration is described as one line of code. Pricing page also lists gateway features such as caching, rate limits and fallbacks.
Open-source observability for your GenAI or LLM application, based on OpenTelemetry.
Pricing: not verified yet, check the tool's site
Engineering note: Built on OpenTelemetry, which helps if you already ship traces to an existing backend.
Debug, evaluate and monitor LLM applications, RAG systems and agentic workflows with tracing, automated evaluations and production dashboards.
Pricing: not verified yet, check the tool's site
Engineering note: Repo lives under the comet-ml org. Combines tracing, evals and dashboards in one tool.
LangChain's platform for tracing, evaluation, deployment and agent tooling.
Pricing: Developer $0/seat (5k base traces/mo); Plus $39/seat/mo (10k base traces/mo); Enterprise custom. Usage in LSUs at $1.00 each. Source: langchain.com/pricing
Engineering note: Base traces are retained 14 days; extended traces 180 days for an extra fee. Pricing page says LangSmith does not train on your data.
The agent engineering platform.
Pricing: not verified yet, check the tool's site
Engineering note: Same org as LangGraph and LangSmith (listed here), so the three are designed to be used together.
Build resilient agents.
Pricing: not verified yet, check the tool's site
Engineering note: Same org as LangChain. LangSmith's pricing page mentions deployment for LangGraph APIs.
Framework for orchestrating role-playing, autonomous AI agents.
Pricing: not verified yet, check the tool's site
Engineering note: Role-based multi-agent design: you define agents with roles that collaborate on tasks.
A programming framework for agentic AI.
Pricing: not verified yet, check the tool's site
Engineering note: Microsoft project. Check the repo README for current status and recommended successor before starting new work.
The document processing platform for AI.
Pricing: not verified yet, check the tool's site
Engineering note: Positions itself as a document processing platform; check docs for its agent features.
How Python does AI. Agents, realtime voice, image generation, embeddings. Every model, every interface, typed end to end.
Pricing: not verified yet, check the tool's site
Engineering note: From the Pydantic team. Selling point is end-to-end typing in Python.
A barebones library for agents that think in code.
Pricing: not verified yet, check the tool's site
Engineering note: From Hugging Face. Agents write actions as code, so sandbox execution is something to plan for.
A lightweight, powerful framework for multi-agent workflows.
Pricing: not verified yet, check the tool's site
Engineering note: Official OpenAI library for multi-agent workflows in Python.
The modern TypeScript framework for AI-powered applications and agents.
Pricing: not verified yet, check the tool's site
Engineering note: One of the TypeScript-first options in this list; the rest are mostly Python.