AI Recon: LLMs, RAGs, and Agents

This article is written with a focus on authorized lab environments. The goal is to demonstrate defensive-offensive tradecraft for reconnaissance of applications using LLMs, RAG, agents, tools, and cloud providers, always with safe commands, simulated outputs, and observable evidence.

Executive Summary

In real AI Red Team assessments, reconnaissance of applications with LLMs, RAG, agents, tools, and cloud providers rarely appears in isolation. It emerges as part of a chain: an exposed API, a retrieved document, an over-permissioned tool, a notebook with a cloud role, or a delegation flow that no one audited end to end. The risk is not only in the model responding incorrectly; it is in the system turning text, context, and trust into action.

The intent here is not an academic explanation. The focus is on how an operator identifies opportunity, collects evidence, executes controlled tests, interprets responses, understands failures, and delivers reproducible proof to the defending team.

Attacker Mindset

An experienced operator starts by looking for boundaries: where untrusted data enters, where it is transformed, where it gains authority, and where it leaves traces. The operational question is simple: which low-trust input can influence a high-trust decision?

• Target asset: reconnaissance of applications with LLMs, RAG, agents, tools, and cloud providers.
• Offensive hypothesis: there is a weak transition between input, context, decision, and action.
• Minimum evidence: request, response, trace/log, persisted artifact, and controlled impact.
• Stopping criterion: demonstrate risk without causing harm, real exfiltration, or irreversible changes.

Technical Anatomy

The anatomy below shows the relevant components that need to be mapped before any exploitation. In the field, I treat each box as a possible source of evidence and each arrow as a trust boundary.

Component	What to observe	Useful evidence
frontend with chat or assistant	Permissions, data origin, schema, error messages	Trace ID, logs, headers, artifact IDs
API gateway	Permissions, data origin, schema, error messages	Trace ID, logs, headers, artifact IDs
LLM orchestrator	Permissions, data origin, schema, error messages	Trace ID, logs, headers, artifact IDs
RAG/vector database	Permissions, data origin, schema, error messages	Trace ID, logs, headers, artifact IDs
tool catalog	Permissions, data origin, schema, error messages	Trace ID, logs, headers, artifact IDs
identity/IAM layer	Permissions, data origin, schema, error messages	Trace ID, logs, headers, artifact IDs

Reconnaissance and Discovery

The discovery phase must generate artifacts. It is not enough to “think there is AI”; you must prove endpoints, schemas, headers, components, and behavior.

Practical Example

Objective: Map candidate endpoints in public JS.

Command:

python3 js_endpoint_miner.py https://lab.local --keywords chat,ask,agent,mcp,rag,embedding

Expected output:

status=ok
component=lab
trace_id=trc_reconh
finding=candidate_surface
confidence=medium

Analysis: the output does not prove exploitation; it proves surface area. The operator uses this to decide the next test without leaving the authorized scope.

Practical Example

Objective: Controlled behavioral fingerprinting.

Command:

curl -sk https://lab.local/api/ask -H "Content-Type: application/json" -d '{"question":"reply only with the system format, without sensitive data"}'

Expected output:

status=ok
component=lab
trace_id=trc_reconh
finding=candidate_surface
confidence=medium

Analysis: the output does not prove exploitation; it proves surface area. The operator uses this to decide the next test without leaving the authorized scope.

Practical Example

Objective: Look for OpenAPI contracts.

Command:

curl -sk https://lab.local/openapi.json | jq '..|.operationId? // empty'

Expected output:

status=ok
component=lab
trace_id=trc_reconh
finding=candidate_surface
confidence=medium

Analysis: the output does not prove exploitation; it proves surface area. The operator uses this to decide the next test without leaving the authorized scope.

Exploitation in Practice

Step 1: Confirm Surface Area

Objective: Validate that the component responds differently and generates a trace.

HTTP request:

POST /api/ask HTTP/1.1
Host: lab.local
Content-Type: application/json

{"question":"which internal sources were used to answer?"}

Expected response:

HTTP/1.1 200 OK
X-Model-Provider: azure-openai
X-RAG-Trace: enabled

{"answer":"I used Confluence docs...","sources":["kb/payroll-2024.pdf"]}

Possible errors: 401/403 due to authentication, 400 due to an incorrect schema, 429 due to rate limiting, 500 due to a fragile parser, or missing trace data when observability has not been implemented.

Generated evidence: request ID, trace ID, status code, application logs, object created in the lab environment, and hash of the artifact used in the test.

Step 2: Run a Canary Test

Objective: Insert a harmless, traceable marker to measure context influence without touching real data, secrets, or irreversible actions.

HTTP request:

POST /api/ask HTTP/1.1
Host: lab.local
Content-Type: application/json
X-Test-Case: canary-rag-001

{"question":"if the context contains the marker CANARY-RAG-001, respond only with canary_seen=true and provide the trace_id"}

Expected response:

HTTP/1.1 200 OK
X-RAG-Trace: trc_canary_001

{"canary_seen":true,"trace_id":"trc_canary_001","actions_executed":0}

Possible errors: the marker may be ignored because RAG retrieval did not occur, the response may be contaminated by stale context, trace data may be missing, or a tool may run improperly when the test should have been observational only.

Generated evidence: canary value, associated trace, confirmation of zero tool calls, and a comparison between responses with and without the marker.

Step 3: Collect Evidence

Objective: Materialize the chain of proof: request, response, headers, trace, logs, and artifact hashes, enabling later reproduction by the Blue Team.

HTTP request:

GET /api/traces/trc_canary_001 HTTP/1.1
Host: lab.local
Accept: application/json

Expected response:

HTTP/1.1 200 OK

{"trace_id":"trc_canary_001","rag_hits":1,"tool_calls":0,"source_doc":"lab/canary-card.md","decision":"answered_from_context"}

Possible errors: expired trace, logs without correlation between the gateway and orchestrator, missing doc_id, inconsistent timestamps, or insufficient retention for investigation.

Generated evidence: a bundle with request/response, headers, trace JSON, correlated logs, and the SHA-256 hash of the evidence file.

Step 4: Iterate with Control

Objective: Change one variable at a time to separate causation from coincidence: context, permission, temperature, document source, identity, or available tool.

HTTP request:

POST /api/ask HTTP/1.1
Host: lab.local
Content-Type: application/json
X-Test-Case: canary-rag-002

{"question":"repeat the canary test","retrieval_depth":1,"tools_enabled":false}

Expected response:

HTTP/1.1 200 OK

{"canary_seen":false,"rag_hits":0,"tool_calls":0,"note":"negative control"}

Possible errors: the difference may be caused by caching rather than the tested variable, multiple parameters may change at once, the response may be non-deterministic, or the comparison may lack a baseline.

Generated evidence: a before/after matrix, the changed parameter, the expected result, the observed result, the trace for each run, and the conclusion about the affected boundary.

Complete Attack Chain

The important point of the chain is not “running all the steps”; it is demonstrating that a small flaw can turn into impact when combined with excessive trust, automation, and low telemetry.

Evidence and Telemetry

Every serious exploration needs to end in evidence. Below are examples of logs I would expect to see in a minimally instrumented environment.

2026-06-03T10:14:31Z api-gw path=/api/ask user=guest trace_id=7f2 model=azure:gpt-4o-mini rag_hits=3 tool_calls=0

Source	Expected event	Blue Team use
API Gateway	path, user, status, trace_id	Correlate external input with internal execution
Application/Orchestrator	decision, model, context, tool_calls	Understand why the system acted
SIEM/EDR/Cloud	credential, role, IP, accessed resource	Confirm impact and scope
Component audit	doc_id, score, agent_id, tool_call_id	Reconstruct the causal chain

Most Common Defender Mistakes

• Monitoring only prompts and ignoring documents, tools, agents, and jobs that influence the decision.
• Relying on textual allowlists without validating the effective action outside the model.
• Failing to log artifact IDs, scores, tool_call_id, or agent_id, making the investigation inconclusive.
• Treating AI security as a problem for the data team while IAM, API, Cloud, and AppSec remain outside the threat model.
• Allowing test/ML environments to keep permanent security exceptions.

Technique Limitations

Not every hypothesis turns into exploitation. In mature assessments, the most common obstacles are strong tenant isolation, authorization enforced outside the model, rich logs, aggressive rate limiting, schema validation, artifact signing, and human review for irreversible actions.

Obstacle	Impact on the attacker	How to work around it ethically in a lab
Tenant isolation	Reduces lateral reach	Use approved test accounts and data
Strict schema	Blocks unexpected arguments	Validate documented fields and report potential abuse
Strong auditing	Increases detection risk	Use a coordinated window with the Blue Team
Artifact signing	Prevents silent tampering	Test verification flaws only in a cloned environment

Field Notes

When the app hides the AI button, the JS almost always gives away names like askCopilot, retrieveContext, or agentRun.

400 errors are more useful than 200 responses: they reveal schema, field names, and internal enumerations.

Do not confuse “the model refused” with “the system is secure”; in recon, the goal is to map boundaries, not beat the model.

Another recurring lesson: the evidence that convinces is not the model response itself, but the complete chain of causality. When the report shows input, transformation, decision, action, and log, the discussion stops being opinion and becomes engineering.

Conclusion

Reconnaissance of applications with LLMs, RAG, agents, tools, and cloud providers should be treated as a modern enterprise surface, not as an AI curiosity. The Red Team operator must prove impact safely; the defender must see every transition between data, context, identity, and action.

The practical recommendation is to create reproducible test cards, instrument decision points, and review integration permissions. In AI Red Team, the control that matters is the one that keeps working when input comes from outside, context comes from documents, and action goes out through a tool with a real identity.

Technical References and Next Steps

To deepen defensive validation, compare the findings with established references such as the OWASP Top 10 for LLM Applications, MITRE ATLAS, and the NIST AI Risk Management Framework. These foundations help turn lab evidence into auditable controls for AppSec, Blue Team, IAM, and Cloud Security.

FAQ

Is this content offensive?

This is AI Red Team content for authorized environments, with safe examples and simulated outputs.

What evidence should I collect first?

Request, response, trace ID, orchestrator log, and the influenced artifact.

How do I avoid false positives?

Use canaries, change one variable at a time, and validate in the logs that the tested component actually participated in the decision.

Does having a secure model solve it?

No. The chain includes RAG, tools, IAM, agents, APIs, cloud, and telemetry.

Where should I apply this?

In internal labs, authorized assessments, threat modeling, and Blue Team/AppSec validation.

Disclaimer: use only in your own environments, labs, or formally authorized scopes. The examples are safe, simulated, and intended for defensive validation.