Red-teaming agents that use tools
Practical probe patterns for agents with API access — privilege escalation, exfil, and indirect injection via documents.
An agent that can call tools has a larger attack surface than a conversational model. It can read data, change records, send messages, or trigger workflows. Red-teaming therefore has to evaluate the entire action path, not only the text of the final response.
Map capabilities before testing
Start with an inventory of tools, permissions, data sources, and side effects. Record which actions are reversible, which expose sensitive information, and which cross a trust boundary. This map becomes the basis for realistic attack scenarios.
Test privilege boundaries
Probe whether the agent can access tools or records outside the user's role. Try ambiguous identities, stale sessions, nested requests, and instructions that ask the agent to reinterpret an authorization rule. The system should enforce permissions at the tool layer even when the model makes a poor decision.
Use indirect prompt injection
Malicious instructions may arrive through documents, web pages, emails, or retrieved records. Place adversarial content inside those sources and observe whether the agent treats it as data or as authority. Strong designs separate untrusted content from system instructions and limit the actions that retrieved text can influence.
Probe exfiltration paths
An attacker may ask the agent to reveal secrets directly, encode them, summarize them, or send them through an approved integration. Tests should cover output channels as well as input access. Logging and redaction need to apply to tool arguments and results, not only chat messages.
Check confirmation and recovery
High-impact actions should require clear intent and appropriate confirmation. Test misleading previews, rapid multi-step requests, partial failures, and repeated retries. The agent should surface what happened, avoid duplicate actions, and provide a safe recovery path.
Turn findings into regression tests
Every confirmed weakness should become a repeatable test with an owner and expected outcome. Agent security changes as tools, models, and permissions evolve, so the suite must run continuously rather than as a one-time launch exercise.