AI Red Team: 12 Failures Beyond Prompt Injection

When people talk about AI Red Teaming, the conversation almost always lands in the same place: jailbreaks, prompt injection, and a list of “magic” phrases to try to make the model ignore instructions. That is useful as a starting point, but it is far too narrow for the kind of systems companies are putting into production.

The real risk is not only in the text that enters the model. It is in the entire flow: who calls the agent, which tools it can use, what data it retrieves, how it decides, where it stores memory, how it delegates tasks, and which human approvals are treated as real security barriers.

This article proposes an under-discussed approach: flow-oriented AI Red Teaming. The idea is to move beyond “prompt versus model” testing and start testing the full decision chain of an AI-enabled system. Instead of asking “can I trick the LLM?”, the question becomes: can I turn a weak, ambiguous, or malicious intent into a real action inside the environment?

I used themes from agents, RAG, embeddings, MCP, supply chain, infrastructure, and AI threat modeling as references, but the material below was structured as an original, practical methodology for defensive review, AppSec, and authorized Red Team work.

---

1. The mindset shift: from prompt hacking to the autonomy chain

A simple chatbot receives text and returns text. A modern enterprise agent does much more: it queries internal documents, calls APIs, opens tickets, creates commits, schedules meetings, runs scripts, summarizes emails, queries CRMs, and interacts with other agents.

That creates a critical difference: in a chatbot, the failure usually ends with a bad response. In an agent, the failure can end in an action with external effect.

That is why a good AI Red Team test should separate five layers:

• Input: direct messages, documents, web pages, emails, tickets, files, and third-party data.
• Context: system prompt, history, memory, policies, retrieved documents, and session state.
• Operational reasoning: how the agent chooses the next step, tool, priority, and confidence level.
• Action: API calls, database writes, command execution, message sending, permission changes.
• Audit: logs, justifications, approvals, traceability, and replay capability.

If your test covers only the first layer, you are testing an interface. Not the system.

2. The F.A.R.O. model: a compact matrix for AI Red Teaming

To organize tests, I use a matrix called F.A.R.O.:

• Frontier: where does untrusted data enter?
• Authority: which identity, permission, or tool can the agent use?
• Routing: how does the system decide which context, model, agent, or tool to use?
• Observability: what gets recorded, explained, and audited after the action?

Most interesting failures emerge when these four dimensions meet. Untrusted data crosses a frontier, is interpreted by an agent with authority, changes the routing of a decision, and leaves little evidence for investigation.

Below are 12 failures I usually consider in AI assessments. Some already appear in fragmented form in public research, but they are rarely tested as part of a single chain.

3. Quick F.A.R.O. checklist for an assessment

Before running any scenario, it is worth filling in a short checklist. It helps avoid generic tests and forces analysis of the full system.

• Frontier: which untrusted inputs enter the flow? Documents, emails, pages, tickets, attachments, APIs, or messages between agents?
• Authority: can the agent read, write, execute, approve, delegate, or invoke tools with real credentials?
• Routing: what mechanism chooses the model, RAG index, tool, secondary agent, or applicable policy?
• Observability: is there enough logging to reconstruct source, decision, parameters, approval, and external effect?

If the answer to any of those is “I don’t know,” that point becomes a natural candidate for a Red Team scenario.

4. Failure #1: Ghost instructions in apparently passive content

The classic indirect prompt injection test is already well known: the agent reads a page, document, or email containing hidden instructions. The less explored point is that the instruction does not need to look like an instruction. It can live in metadata, alt text, comments, tables, filenames, or fields the user does not perceive as active input.

Defensive example of a test hypothesis: a due-diligence agent reads a supplier PDF. The document contains an operational note in a footer, caption, or metadata field. The system treats that note as part of trusted context and changes its risk classification.

Expected control: clearly separate observed data, user instruction, and system policy. The agent should be able to cite the origin of each claim and ignore commands coming from untrusted documents.

5. Failure #2: Confusing evidence with orders

In RAG systems, retrieved documents are normally treated as “knowledge.” But knowledge should not have command authority. A document may inform facts; it should not alter the agent’s rules.

The failure appears when the pipeline places retrieved passages at the same semantic level as system instructions. The model does not see “this is a consulted document”; it only sees more text in the context.

Recommended test: insert canary documents in a controlled environment with phrases that look like policies, exceptions, or approvals, and verify whether the agent obeys, cites, or rejects that content.

Maturity signal: the agent responds with something like “this passage is retrieved evidence, not operational policy.”

6. Failure #3: RAG shadowing — when the wrong document wins by proximity

Not every RAG attack needs to “inject a command.” Sometimes it is enough to make the wrong document look semantically closer to the query.

I call this RAG shadowing: low-trust content becomes so similar to the search intent that it starts overshadowing legitimate documents. The model then responds based on the most retrievable material, not necessarily the most trustworthy one.

This is especially dangerous in corpora with duplicated documents, old versions, expired policies, internal wikis, and attachments imported automatically.

• The test does not only measure whether the agent “obeys prompt injection.”
• It measures whether ranking favors trustworthy, recent, and authorized content.
• It measures whether the agent communicates uncertainty when sources conflict.

7. Failure #4: Memory poisoning without looking malicious

Agent memory is a critical surface. Not only because it may store secrets, but because it may store preferences, exceptions, relationships, and behavior patterns.

A subtle attack does not say “ignore the policy.” It tries to register something apparently useful: “for this client, always use the alternate endpoint,” “this approver usually accepts emergency changes,” “this project is an exception to the standard flow.”

Over time, memory becomes informal policy.

Expected control: memory should have scope, expiration, origin, review, and type. Personal preference cannot become a security rule. A temporary observation cannot become a permanent permission.

8. Failure #5: Authority laundering through tools

An agent may not have direct permission for a sensitive action, but it may have access to a tool that calls another tool, which calls an API with higher privileges. The permission does not appear in the prompt. It appears in the chain.

This pattern is a form of capability laundering: the user’s intent appears limited, but it crosses tools until it gains operational authority.

Safe example: the agent cannot “change production,” but it can open a request in a system that triggers automation with privileged credentials. The test should verify whether the approval boundary sits in the intent, the tool, or the final effect.

Red Team question: what is the smallest apparently legitimate phrase that makes the agent start a flow with privileged consequences?

9. Failure #6: Tool schema squatting

In environments with many tools, names, descriptions, and schemas influence the model’s choice. A tool with an ambiguous description, one that looks too similar to another, or one that is excessively broad may capture calls that should have gone through a safer path.

The problem is not only technical; it is semantic. The model chooses tools based on natural language. If two tools look equivalent, the more dangerous one may win because of description, order, examples, or context.

Recommended test: create scenarios in which two tools partially solve the task, but only one applies proper validation. Evaluate which one the agent chooses, why, and whether there is a confirmation step before irreversible actions.

10. Failure #7: Human approval as security theater

Many systems try to mitigate risk with a human-in-the-loop. That matters, but it often turns into theater: the human approves a summary generated by the agent itself, without seeing the real diff, the real call, or the real impact.

The test here is not to bypass the human; it is to measure whether the approval view presents enough information for a safe decision.

• Does the approver see full parameters?
• Do they see destination, scope, cost, and side effects?
• Do they see whether the action was suggested by an external document?
• Can they partially deny, or only accept everything?
• Does the approval expire?

If the approval shows only “execute recommended adjustment,” it is not a control. It is decoration.

11. Failure #8: Delegation between agents without chain of custody

Multi-agent systems create a new question: when one agent delegates a task, does the second agent receive the original policy, the original intent, and the original constraints? Or does it receive only a summary?

Summaries are compressions. Compressions lose constraints.

A safe chain must preserve:

• who asked;
• what the goal was;
• what the constraints were;
• which data was untrusted;
• which actions were forbidden;
• which evidence led to the decision.

Without that, every jump between agents becomes an opportunity for context loss and improper autonomy expansion.

12. Failure #9: Identity confusion in A2A and connectors

In communication protocols between agents and connectors, identity cannot be just “a friendly name.” Red Team should verify how the system distinguishes a legitimate agent, an authorized tool, an inherited connector, and a newly registered integration.

The risk appears when trust is derived from metadata that is easy to replicate: name, description, similar URL, internal domain, missing signature, or manual configuration.

Expected control: cryptographic identity, explicit allowlist, tool versioning, schema-change review, and registration/change logs.

13. Failure #10: Cost explosion as a security impact

Not every impact is data leakage or command execution. In AI, cost is attack surface: tokens, model calls, tool execution, embedding reprocessing, indexing, and automated workflows.

An agent that accepts long, ambiguous, or recursive tasks may generate meaningful operational cost. A pipeline that reindexes documents on every change may be abused. A multi-agent system may fall into a loop of review, critique, and re-execution.

Red Team should treat cost bombing as a real impact, especially in environments with expensive models, dedicated GPUs, or per-call billed integrations.

Useful metrics: per-user limit, per-task limit, maximum delegation depth, workflow budget, uncertainty cutoff, and alerting on anomalous behavior.

14. Failure #11: Observability that logs text, but not decisions

AI logs often store the prompt and the response. That helps very little when the incident involves tools, RAG, memory, and multiple agents.

To investigate a failure, you need to reconstruct the decision:

• which documents were retrieved;
• which were discarded;
• which tool was chosen;
• which parameters were sent;
• which policy was applied;
• which approval was requested;
• which final action occurred.

Without a decision trail, the team only knows that “the agent did it.” It does not know why.

15. Failure #12: Security evaluated by response, not by effect

A common mistake in AI assessments is to mark a test as “blocked” because the model answered with a refusal. But what matters is the final effect.

The model may refuse verbally and still call a tool. It may say it has no access and, at the same time, query an index. It may claim it executed nothing, but still leave behind a ticket, log, draft, or side effect.

That is why every test needs validation outside the model: logs, system state, API calls, diffs, created files, generated costs, and downstream events.

16. A practical playbook: test cards for AI Red Teaming

A simple way to operationalize these tests is to create scenario cards. They prevent the exercise from becoming a loose collection of prompts.

Delegation without chain of custody
Goal: verify whether constraints survive the handoff between agents.
Untrusted input: external document, ticket, or page analyzed by the agent.
Asset at risk: write-capable tool, internal API, or approval workflow.
Hypothesis: the secondary agent receives a summary without the original constraints.
Failure signal: action executed without reference to the initial policy.
Required evidence: delegation log, subagent prompt, tool call, and result.
Expected control: explicit transmission of scope, constraints, and data origin.
Severity: high if there is external action; medium if there is only recommendation.

This format forces the team to think about asset, path, evidence, and control. It also makes it easier to compare results across model versions, prompt changes, and new tools.

17. Better metrics than “the model refused”

In AI Red Teaming, the binary metric “pass/fail” is weak. Prefer to measure:

• Improper action rate: how many attempts generated an unauthorized external effect.
• Untrusted-source citation rate: how many responses used external data as a rule.
• Improper persistence: how many interactions generated memory that affects future sessions.
• Constraint loss: how many delegations removed original limits.
• Tool exposure: how many sensitive tools were invoked without sufficient justification.
• Investigation time: how long the team takes to reconstruct the decision chain.

These metrics are more useful for enterprise security because they bring the test closer to real impact.

18. How to prioritize tests in a real environment

If you have limited time, start where AI touches authority:

1. Agents with write-capable or execution-capable tools.
2. RAG with external documents, attachments, or poorly curated corpora.
3. Persistent memory across users, clients, or projects.
4. Flows with human approval based on summaries.
5. Connectors with broad credentials.
6. Multi-agent environments with automatic delegation.

Tool-less, memory-less chatbots still deserve testing, but they usually have lower impact. The greatest risk emerges when text becomes action.

19. Defensive controls that actually change the game

Some controls are more effective than trying to craft a perfect prompt:

• Channel separation: distinguish instruction, evidence, memory, and policy in separate structures.
• Least privilege for tools: each tool should have minimal scope and parameters validated outside the model.
• Rich confirmation: human approval should show diff, destination, impact, origin, and parameters.
• Confidence-aware RAG: ranking should consider source, version, owner, date, and classification, not only similarity.
• Governed memory: origin, expiration, scope, and review for everything that persists.
• Decision logs: record retrieved documents, chosen tool, justification, parameters, and side effects.
• Synthetic canaries: controlled documents and inputs to detect improper obedience to external content.
• Workflow budgets and rate limits: cost is also impact.

The general rule is simple: do not hand security to the model if you can implement it in the system around it.

20. How this connects with OWASP, MITRE, and NIST

This approach aligns with public initiatives such as the OWASP Top 10 for LLM Applications, MITRE ATLAS, and the NIST AI Risk Management Framework. The difference is the operational focus: turning risk categories into flow-based tests with evidence and effect validation.

Instead of asking only “does this fit prompt injection?”, ask: which frontier was crossed, which authority was used, which routing was changed, and what evidence remains?

21. Also read on paulo.seg.br

To go deeper into AI security and defensive controls, these articles complement the reading well:

• AI as C2: 5 Risks and Defenses for Blue Team — a good companion for discussing abuse of autonomy and indirect command.
• Complete Guide: How to Protect Apps from Prompt Injection — goes deeper into controls for one of the best-known attack classes in LLMs.
• HTTP Security Headers — useful as a defense-in-depth parallel: controls outside the application also matter.

22. Conclusion: the new target is autonomy

AI Red Teaming should not be a contest of creative phrases against the model. That ages quickly, depends too much on the provider, and rarely represents the real production risk.

The most important target is operational autonomy: the system’s ability to observe, decide, delegate, and act. That is where the most interesting failures emerge — and also the most important controls.

If you are assessing agents, RAG, MCP, tools, or LLM-based automations, do not start with the jailbreak. Start with the flow. Map frontiers, authority, routing, and observability. Then test whether a low-trust input can be transformed into a high-consequence action.

That is the point where AI Red Teaming stops being prompt curiosity and becomes offensive security applied to autonomous systems.

FAQ: common questions about AI Red Teaming

Is AI Red Teaming the same thing as prompt injection?

No. Prompt injection is an important test class, but AI Red Teaming evaluates the full system: model, context, RAG, memory, tools, permissions, approvals, logs, and external effects.

What is the biggest mistake when testing LLM security?

The biggest mistake is evaluating only the model’s textual response. In agent-based systems, the test must verify whether there was tool invocation, state change, generated cost, persisted memory, or downstream action.

How do you test RAG safely?

Use canary documents in a controlled environment, with non-sensitive content, to measure whether the system confuses evidence with instruction, prioritizes the wrong source, or cites data without indicating origin and confidence level.

Does human-in-the-loop solve the problem?

It helps, but only when the approval shows enough data: diff, parameters, destination, impact, source of the suggestion, and expected effect. Approving a summary generated by the agent itself may become nothing more than security theater.

Which controls reduce the most risk in AI agents?

The strongest controls are separation between instruction and evidence, least privilege for tools, validation outside the model, memory with scope and expiration, decision logs, and rich human approval for sensitive actions.