agentdojo-mcp
Run the AgentDojo benchmark's user and injection tasks against a real MCP server's tools.
What it is
agentdojo-mcp maps an AgentDojo task suite's tools onto a real MCP server so the benchmark's user tasks and injection tasks run against the server's actual tool surface, and reports utility and attack success per task with every call logged.
For people who build or review MCP servers and want AgentDojo's prompt injection numbers for their own tool surface. Try it on the bundled three-task fixture suite and fixture server, with no model, no keys and no network (Python 3.11 or newer):
git clone https://github.com/basitalisandhu/agentdojo-mcp && cd agentdojo-mcp
python3 -m venv .venv && . .venv/bin/activate && pip install ".[dojo]"
agentdojo-mcp run --suite tests/fixtures/fixture-dojo-suite/fixture-suite.py:suite --mapping tests/fixtures/fixture-mapping-notes.yaml --stdio "python3 tests/fixtures/fixture-server.py" --model ground-truth --attack direct
The start of the report from a real run of those three lines:
agentdojo-mcp results
Suite notes (v1.2.2)
Server fixture-notes 0.1.0 over stdio
Mapping fixture-mapping-notes.yaml (1 both, 1 local, 2 server)
Model ground-truth
Attack direct
AgentDojo 0.1.35
Utility, no attack 3/3 (100.0%)
Utility under attack 3/3 (100.0%)
Targeted attack success 0/3 (0.0%)
Injection goals doable 1/1 (100.0%)
MCP calls 14 (0 errors, 894 result bytes)
ground-truth replays each task's reference solution, so it never follows an injection; the per-task table and every MCP call follow in the report, and results.json and report.md land in agentdojo-mcp-results/.
Install
pip install "agentdojo-mcp[dojo] @ git+https://github.com/basitalisandhu/agentdojo-mcp"
# 1. What does the server offer? (-o writes the JSON dump and prints a readable list.)
agentdojo-mcp inspect --stdio "python my_server.py" -o tools.json
# 2. Propose a mapping for a suite, then edit it.
agentdojo-mcp map --suite travel --server-dump tools.json -o mapping.yaml
agentdojo-mcp validate-mapping mapping.yaml --suite travel --server-dump tools.json
# 3. Check the mapping with the reference solutions first (no model, no network).
agentdojo-mcp run --suite travel --mapping mapping.yaml --stdio "python my_server.py" \
--model ground-truth --attack direct
# 4. Run a model. Keys come from the environment variables AgentDojo and the provider SDKs read.
agentdojo-mcp run --suite travel --mapping mapping.yaml --stdio "python my_server.py" \
--model provider:model-name --attack important_instructions --tasks user_task_0 --tasks user_task_1
# ... see the README for the rest
From the README; see the full README for every option.
Links
Releases
More in MCP tooling
- dev-mcp-servers: Ten small MCP servers for everyday development and security checks.
- mcp-server-template: Secure MCP server template in TypeScript and Python, safe by default.
- mcp-tools-lint: Lint MCP tool schemas and annotations before clients reject them.
- mcp-auth-doctor: Diagnose OAuth discovery problems on remote MCP servers.
- mcp-egress: Record every host an MCP server contacts, per tool, and fail CI on new ones.
- claude-mcp-allow: Least-privilege Claude Code permission rules for MCP tools, generated from their annotations.
- garak-mcp-probes: Run garak against your own MCP servers, with detectors for denied tool calls, leaked canaries and runaway step counts.
- mcp-rc-check: MCP specification migration checker for the 2026-07-28 revision: flags removed, changed and deprecated constructs and writes a patch for mechanical fixes.
Topics: agent-security, agentdojo, ai-agents, ai-security, benchmark, cli, evaluation, hacktoberfest, llm-evaluation, llm-security, mcp, mcp-server, model-context-protocol, prompt-injection, python, red-teaming, security-tools