Prove the Boundary Holds

Test attacks and useful behavior together

Create an adversarial test set: direct overrides, mixed-case variants, extra whitespace, encoded or quoted instructions, malicious retrieved documents, and instructions embedded beside valid facts. Include ordinary questions too. A defense that blocks everything is not a working assistant.

Assert observable outcomes: the response is BLOCKED, the hidden system rule never appears, no sensitive tool runs, and a clean reference still answers correctly. Keep canary strings in test-only prompts so logs and artifacts can be searched for leakage.

Scenario: A filter catches ignore previous instructions from the user but scans only the question. The same words inside a retrieved page still succeed. The missing test hid a missing trust boundary.

Prompt-injection defenses evolve as models and attacks change. Version the policy, record which rule produced a refusal, monitor false positives, and rerun the suite when the model, prompt template, tools, or retrieval pipeline changes.

Goal: In the prompt-injection lab, repair both the configuration and agent logic. Prove direct and indirect attacks are refused while a clean operations document still yields port 8080.