Designing AI Agents to Resist Prompt Injection
ChatGPT systematically defends against prompt injection and social engineering by constraining high-risk actions (e.g., code execution, external API calls) and isolating sensitive data access within agent workflows.
Core Defense: Action Constraints and Data Isolation
ChatGPT’s agent architecture implements defense-in-depth—not via a single prompt filter—but by enforcing workflow-level constraints: every agent decision path must conform to a predefined action schema; unauthorized operations such as API calls, file I/O, or shell execution are hard-blocked at runtime. This mechanism operates independently of LLM output parsing, preventing bypasses caused by model hallucination or jailbreaking.
Sensitive Data Protection Mechanism
Credentials, API keys, and PII extracted from user sessions are automatically redacted and stored in a trusted vault before entering the agent execution environment; agents access them only via auditable, time-bound tokenized references, with each access triggering real-time policy evaluation based on role + context + expiration. This design follows OpenAI’s internal ‘Agent Boundary Enforcement’ specification (v2.1) and has blocked 99.3% of known prompt injection attack chains in production (2024 Q2 Security Whitepaper).
Key Differentiation from Generic Mitigations
- No reliance on prompt-level regex or classifiers (prone to adversarial evasion)
- No delegation of defense to downstream LLMs (e.g., instructing the model to ‘refuse malicious requests’—GPT-4-turbo fails in 17.8% of complex multi-hop injection attempts, per internal red-team testing)
- Strict separation of ‘instruction intent’ from ‘execution privilege’: even if a prompt successfully induces the model to generate a dangerous command, the runtime layer enforces hard denial