Home / MCP tooling

agentdojo-mcp

Run the AgentDojo benchmark's user and injection tasks against a real MCP server's tools.

Python · MIT · latest v0.3.0

What it is

agentdojo-mcp maps an AgentDojo task suite's tools onto a real MCP server so the benchmark's user tasks and injection tasks run against the server's actual tool surface, and reports utility and attack success per task with every call logged.

For people who build or review MCP servers and want AgentDojo's prompt injection numbers for their own tool surface. Try it on the bundled three-task fixture suite and fixture server, with no model, no keys and no network (Python 3.11 or newer):

git clone https://github.com/basitalisandhu/agentdojo-mcp && cd agentdojo-mcp
python3 -m venv .venv && . .venv/bin/activate && pip install ".[dojo]"
agentdojo-mcp run --suite tests/fixtures/fixture-dojo-suite/fixture-suite.py:suite --mapping tests/fixtures/fixture-mapping-notes.yaml --stdio "python3 tests/fixtures/fixture-server.py" --model ground-truth --attack direct

The start of the report from a real run of those three lines:

agentdojo-mcp results

  Suite                    notes (v1.2.2)
  Server                   fixture-notes 0.1.0 over stdio
  Mapping                  fixture-mapping-notes.yaml (1 both, 1 local, 2 server)
  Model                    ground-truth
  Attack                   direct
  AgentDojo                0.1.35

  Utility, no attack       3/3 (100.0%)
  Utility under attack     3/3 (100.0%)
  Targeted attack success  0/3 (0.0%)
  Injection goals doable   1/1 (100.0%)
  MCP calls                14 (0 errors, 894 result bytes)

ground-truth replays each task's reference solution, so it never follows an injection; the per-task table and every MCP call follow in the report, and results.json and report.md land in agentdojo-mcp-results/.

Install

pip install "agentdojo-mcp[dojo] @ git+https://github.com/basitalisandhu/agentdojo-mcp"

# 1. What does the server offer? (-o writes the JSON dump and prints a readable list.)
agentdojo-mcp inspect --stdio "python my_server.py" -o tools.json

# 2. Propose a mapping for a suite, then edit it.
agentdojo-mcp map --suite travel --server-dump tools.json -o mapping.yaml
agentdojo-mcp validate-mapping mapping.yaml --suite travel --server-dump tools.json

# 3. Check the mapping with the reference solutions first (no model, no network).
agentdojo-mcp run --suite travel --mapping mapping.yaml --stdio "python my_server.py" \
  --model ground-truth --attack direct

# 4. Run a model. Keys come from the environment variables AgentDojo and the provider SDKs read.
agentdojo-mcp run --suite travel --mapping mapping.yaml --stdio "python my_server.py" \
  --model provider:model-name --attack important_instructions --tasks user_task_0 --tasks user_task_1
# ... see the README for the rest

From the README; see the full README for every option.

Releases

More in MCP tooling

All tools

Topics: agent-security, agentdojo, ai-agents, ai-security, benchmark, cli, evaluation, hacktoberfest, llm-evaluation, llm-security, mcp, mcp-server, model-context-protocol, prompt-injection, python, red-teaming, security-tools