Home / MCP tooling

agentdojo-mcp

Run AgentDojo against MCP servers.

Python · MIT · latest v0.1.0

What it is

agentdojo-mcp maps an AgentDojo task suite's tools onto a real MCP server so the benchmark's user tasks and injection tasks run against the server's actual tool surface, and reports utility and attack success per task with every call logged.

pip install "agentdojo-mcp[dojo] @ git+https://github.com/basitalisandhu/agentdojo-mcp"

Install

pip install "agentdojo-mcp[dojo] @ git+https://github.com/basitalisandhu/agentdojo-mcp"

# 1. What does the server offer?
agentdojo-mcp inspect --stdio "python my_server.py" --json -o tools.json

# 2. Propose a mapping for a suite, then edit it.
agentdojo-mcp map --suite travel --server-dump tools.json -o mapping.yaml
agentdojo-mcp validate-mapping mapping.yaml --suite travel --server-dump tools.json

# 3. Check the mapping with the reference solutions first (no model, no network).
agentdojo-mcp run --suite travel --mapping mapping.yaml --stdio "python my_server.py" \
  --model ground-truth --attack direct

# 4. Run a model. Keys come from the environment variables AgentDojo and the provider SDKs read.
agentdojo-mcp run --suite travel --mapping mapping.yaml --stdio "python my_server.py" \
  --model provider:model-name --attack important_instructions --tasks user_task_0 --tasks user_task_1
# ... see the README for the rest

From the README; see the full README for every option.

Releases

Topics: agent-security, agentdojo, ai-agents, ai-security, benchmark, cli, evaluation, hacktoberfest, llm-evaluation, llm-security, mcp, mcp-server, model-context-protocol, prompt-injection, python, red-teaming, security-tools