Promptfoo

aid: promptfoo
url: https://raw.githubusercontent.com/api-evangelist/promptfoo/refs/heads/main/apis.yml
name: Promptfoo
type: Index
image: https://kinlane-productions.s3.amazonaws.com/apis-json/apis-json-logo.jpg
tags:
- LLM Evaluation
- Red Teaming
- AI Security
- Guardrails
- Open Source
- CLI
- Developer Tools
description: Promptfoo is an open-source LLM evaluation and red-teaming framework distributed as a TypeScript CLI and Node.js
  library under the MIT license. Developers use it to evaluate prompts, models, and RAG pipelines side by side, run automated
  red team attacks against LLM applications, scan code for LLM vulnerabilities in IDE and CI, and proxy Model Context Protocol
  traffic. Promptfoo also operates a commercial Enterprise platform at promptfoo.app for teams that need centralized governance,
  remediation reports, and shared evaluation history.
created: '2026-05-23'
modified: '2026-05-23'
specificationVersion: '0.19'
apis:
- aid: promptfoo:promptfoo-cli
  name: Promptfoo CLI
  tags:
  - CLI
  - LLM Evaluation
  - Open Source
  - Testing
  humanURL: https://www.promptfoo.dev/docs/getting-started/
  properties:
  - url: https://www.promptfoo.dev/docs/getting-started/
    type: GettingStarted
  - url: https://www.promptfoo.dev/docs/usage/command-line/
    type: Documentation
  - url: https://github.com/promptfoo/promptfoo
    type: SourceCode
  description: The Promptfoo CLI is the primary entry point for running prompt and model evaluations from the command line
    or CI. It is installable via npm, Homebrew, pip, or npx and reads a YAML configuration to fan out evaluations across
    providers including OpenAI, Anthropic, Azure, Bedrock, Google, and local Ollama models. Outputs include a local web
    viewer, JSON reports, and CI-friendly exit codes.
- aid: promptfoo:promptfoo-library
  name: Promptfoo Node.js Library
  tags:
  - SDK
  - Node.js
  - Library
  - LLM Evaluation
  humanURL: https://www.promptfoo.dev/docs/usage/node-package/
  properties:
  - url: https://www.promptfoo.dev/docs/usage/node-package/
    type: Documentation
  - url: https://www.npmjs.com/package/promptfoo
    type: SDK
  description: The Promptfoo Node.js package exposes the same evaluation engine programmatically so developers can embed
    evaluations, assertions, and dataset runs directly into JavaScript and TypeScript applications and pipelines.
- aid: promptfoo:promptfoo-red-team
  name: Promptfoo Red Team
  tags:
  - Red Teaming
  - AI Security
  - Vulnerability Scanning
  - Adversarial Testing
  humanURL: https://www.promptfoo.dev/docs/red-team/
  properties:
  - url: https://www.promptfoo.dev/docs/red-team/
    type: Documentation
  description: Promptfoo Red Team generates adversarial test cases against LLM applications targeting prompt injection,
    jailbreaks, PII leakage, bias, and other OWASP LLM Top 10 categories. Runs from the CLI and produces structured findings
    that can be triaged in the Promptfoo viewer or Enterprise platform.
- aid: promptfoo:promptfoo-enterprise
  name: Promptfoo Enterprise
  tags:
  - Enterprise
  - SaaS
  - Governance
  - Team Collaboration
  humanURL: https://www.promptfoo.dev/docs/enterprise/
  properties:
  - url: https://www.promptfoo.dev/docs/enterprise/
    type: Documentation
  - url: https://promptfoo.app/
    type: ApplicationURL
  description: Promptfoo Enterprise is the commercial SaaS at promptfoo.app providing centralized evaluation history, shared
    red team findings, remediation reports, role-based access control, SSO, and team-wide governance over LLM evaluation
    workflows.
- aid: promptfoo:promptfoo-mcp-proxy
  name: Promptfoo MCP Proxy
  tags:
  - MCP
  - Proxy
  - AI Security
  - Model Context Protocol
  humanURL: https://www.promptfoo.dev/docs/usage/mcp/
  properties:
  - url: https://www.promptfoo.dev/docs/usage/mcp/
    type: Documentation
  description: MCP Proxy is Promptfoo's security gateway for Model Context Protocol traffic, inspecting tool calls and responses
    flowing between agents and MCP servers to enforce policies and surface adversarial activity.
- aid: promptfoo:promptfoo-code-scanning
  name: Promptfoo Code Scanning
  tags:
  - Code Scanning
  - IDE
  - CI/CD
  - Static Analysis
  humanURL: https://www.promptfoo.dev/docs/usage/code-scanning/
  properties:
  - url: https://www.promptfoo.dev/docs/usage/code-scanning/
    type: Documentation
  description: Promptfoo Code Scanning analyzes source code in IDEs and CI pipelines to find LLM-related vulnerabilities
    including unsafe prompt construction, missing guardrails, and risky tool usage in agent code.
common:
- type: Website
  url: https://www.promptfoo.dev/
- type: Documentation
  url: https://www.promptfoo.dev/docs/
- type: APIReference
  url: https://www.promptfoo.dev/docs/api-reference/
- type: GettingStarted
  url: https://www.promptfoo.dev/docs/getting-started/
- type: Blog
  url: https://www.promptfoo.dev/blog/
- type: Pricing
  url: https://www.promptfoo.dev/pricing/
- type: Login
  url: https://promptfoo.app/
- type: GitHubRepository
  url: https://github.com/promptfoo/promptfoo
- type: LinkedIn
  url: https://www.linkedin.com/company/promptfoo/
- type: Discord
  url: https://discord.gg/promptfoo
- type: Contact
  url: https://www.promptfoo.dev/contact/
- type: License
  url: https://github.com/promptfoo/promptfoo/blob/main/LICENSE
- type: Features
  data:
  - name: Side-by-Side Evaluation
    description: Compare prompts and models across providers with assertion-based scoring.
  - name: Red Teaming
    description: Automated adversarial test generation across prompt injection, jailbreaks, PII, bias, and more.
  - name: RAG Evaluation
    description: Evaluate retrieval-augmented generation pipelines for accuracy, faithfulness, and groundedness.
  - name: CI/CD Integration
    description: Run evaluations on every pull request with exit-code-driven gating.
  - name: Local Web Viewer
    description: Inspect evaluation runs and diffs in a local browser UI without sending data to a third party.
  - name: MCP Proxy
    description: Inspect and govern Model Context Protocol traffic between agents and tool servers.
  - name: Code Scanning
    description: Static analysis of LLM application code in IDEs and CI to surface risky patterns.
  - name: Self-Hosting
    description: Open source CLI and library run locally with no telemetry to Promptfoo required.
- type: UseCases
  data:
  - name: Prompt Iteration
    description: Compare prompt variants against datasets to choose the best-performing version.
  - name: Model Selection
    description: Benchmark candidate models across providers before committing to one in production.
  - name: RAG Quality Assurance
    description: Evaluate chunking, retrieval, and generation choices in RAG systems.
  - name: AI Red Teaming
    description: Probe pre-production LLM applications for adversarial failure modes.
  - name: Continuous LLM Testing
    description: Wire evaluations and red team scans into CI so regressions block deploys.
- type: Integrations
  data:
  - name: OpenAI
    description: Native provider for OpenAI Chat Completions, Assistants, and Responses APIs.
  - name: Anthropic
    description: Native provider for Anthropic Claude models.
  - name: Azure OpenAI
    description: Provider for Azure-hosted OpenAI deployments.
  - name: AWS Bedrock
    description: Provider for Anthropic, Meta, Mistral, and other models on Bedrock.
  - name: Google Vertex AI
    description: Provider for Gemini and other models on Vertex.
  - name: Ollama
    description: Provider for locally hosted open source models via Ollama.
  - name: LangChain
    description: Evaluate LangChain chains and agents through custom providers.
  - name: GitHub Actions
    description: Run promptfoo evaluations in CI on every pull request.
  - name: Model Context Protocol
    description: MCP Proxy and MCP provider support for agentic workflows.
maintainers:
- FN: Kin Lane
  email: [email protected]