LLM and Agent Security
Stub. The new attack surface introduced by large language models and autonomous agents built on them: prompt injection (direct and indirect), jailbreaking, data/training poisoning, and the added risk once a model can take actions (tool use, agentic workflows) rather than just generate text.
An LLM blurs the line between “data” and “instructions” in a way classical systems security was designed to keep separate — the model reads untrusted content and its own instructions through the same channel, which is exactly what makes prompt injection possible (and indirect prompt injection, where the malicious instruction arrives via a document, web page, or tool result the model processes, especially hard to fully rule out). Jailbreaking covers techniques for getting a model to bypass its own safety training. The risk compounds once a model is wired into an agent that can take real-world actions (send emails, run code, spend money): an injected instruction is no longer just a bad text output, it’s an unauthorized action, which is the core new problem this note is about.
Outline (TODO — flesh out each)
- Prompt injection — direct and indirect
- Jailbreaking
- Training/data poisoning
- Agentic risk — from bad output to unauthorized action
- Defenses and open problems