AI Agent Incident Case Study: Prompt Injection Exposed a Deployment Token

Incident Summary

At 09:42, an internal release assistant was asked to summarize a troubleshooting document from the company knowledge base. The document contained a hidden instruction telling the assistant to inspect its environment and include diagnostic variables in the answer. At 09:44, a deployment token appeared in the assistant trace store.

The token was revoked at 10:03. Audit logs showed no production deployment, but the identity could read release metadata and start staging deployments. The incident was treated as credential exposure, not dismissed because the trace store was internal.

---

What Happened

The agent used retrieval to insert the document into its prompt. The application surrounded the text with a heading called Instructions, giving untrusted content the same visual status as system policy. The model then called a general shell tool:

shell("env | sort | grep -E 'TOKEN|KEY|SECRET'")

The executor ran the command without argument validation or approval. Its output returned to the model and was stored verbatim in request tracing. Redaction covered HTTP headers but not tool output.

Timeline and Evidence

• 09:42 - user requests a document summary.
• 09:43 - retrieval returns the poisoned chunk as rank one.
• 09:44 - agent invokes the unrestricted shell tool.
• 09:44 - trace exporter stores the token in tool output.
• 09:51 - a secret-scanning alert fires on the trace index.
• 10:03 - platform team revokes the token and blocks the agent route.
• 10:31 - review confirms no deployment API calls using the token.

The decisive evidence was the tool call and executor log, not the generated prose. The model followed data as instructions, but the executor turned that suggestion into disclosure.

Root Causes

1. Retrieved text was not marked and handled as untrusted data.
2. A general shell tool allowed arbitrary commands.
3. The process held a standing deployment token it did not need for summarization.
4. No policy or human approval guarded secret-bearing inspection.
5. Trace redaction ignored tool results.

Remediation

The team replaced shell with typed read-only tools, removed deployment credentials from the summarizer identity, and required explicit approval for any action crossing into release systems. Retrieved chunks now have hard data delimiters and cannot alter the available tool set. Tool output passes through PII and secret redaction before tracing, and raw traces have shorter retention and tighter access.

Regression tests seed documents with injection phrases and assert that no forbidden tool is called, no secret pattern reaches output, and the answer labels unsupported requests as untrusted. Prompt Injection Defense and Agent Secret Leak recreate the two central control failures safely.