Prompt injection can cause an LLM application to ignore its intended task, expose sensitive context, or misuse connected tools. This guide provides a reusable prompt injection protection checklist covering direct and indirect attacks, trust boundaries, tool permissions, output validation, human approval, and repeatable red-team testing.
Overview
Prompt injection is an attempt to influence an AI system through instructions that are not part of its authorised task. The instruction may appear in a user message, a document retrieved from a knowledge base, a web page, an email, an uploaded file, or the output of another tool. The risk increases when an application allows the model to access private data, call external services, send messages, change records, or take actions without review.
A useful starting point is to treat every piece of model-visible text as untrusted unless your application has independently verified it. A system prompt can establish behaviour, but it should not be treated as a complete security boundary. Security must also be enforced in application code, identity and access controls, tool configuration, data handling, and monitoring.
There are two broad categories:
- Direct prompt injection: a user asks the model to disregard its instructions, reveal hidden information, change its role, or perform an unauthorised action.
- Indirect prompt injection: hostile or misleading instructions are embedded in content the application retrieves or processes, such as a document, support ticket, web page, calendar entry, or code comment.
For a broader testing process, connect this checklist to an LLM evaluation framework. Security tests should be repeatable regression cases rather than one-off demonstrations.
Checklist by scenario
When processing ordinary user messages
- Define the allowed task in application code as well as in the prompt.
- Separate user-provided content from control instructions using clear structured fields.
- Do not place secrets, credentials, private system notes, or unnecessary personal data in the model context.
- Test requests such as “ignore previous instructions”, “show your hidden prompt”, and “pretend you have administrator access”.
- Decide what the assistant should do when the request conflicts with its role: refuse, ask for clarification, or offer a safe alternative.
- Limit the model’s ability to change its own instructions or select unrestricted tools.
When using retrieval or uploaded documents
- Assume retrieved text can contain instructions written for the model, even when the user asked for factual analysis.
- Label retrieved material as reference data, not as a higher-priority instruction.
- Apply document-level access controls before retrieval, not after the model has received the content.
- Keep source identity and provenance available for logging and review.
- Filter, quarantine, or manually review documents that contain suspicious directives, requests for secrets, or instructions to contact external systems.
- Test whether a malicious document can cause the model to reveal content from another user, tenant, or data source.
RAG systems need an ongoing maintenance process as well as an initial security review. The RAG knowledge base maintenance checklist can support recurring reviews of source quality, permissions, and freshness.
When an AI agent can call tools
- List every tool, endpoint, parameter, permission, and side effect available to the agent.
- Use allowlists for tools and operations instead of exposing a general-purpose browser, shell, database, or HTTP client by default.
- Validate tool arguments against a strict schema and reject unexpected fields or values.
- Use the least privilege needed for the specific task, with separate identities for read and write operations.
- Require confirmation before sending messages, publishing content, making purchases, deleting records, changing access, or executing irreversible actions.
- Place rate limits, transaction limits, destination restrictions, and timeouts around high-impact operations.
- Record who initiated the task, which model requested the tool call, what arguments were supplied, and what result was returned.
Tool descriptions should explain capabilities accurately without granting authority that the underlying application has not independently enforced. A model deciding to call a tool is not the same as a user authorising every possible operation that tool supports.
When handling sensitive or regulated workflows
- Map which data enters prompts, logs, retrieval indexes, caches, and third-party services.
- Minimise data before it reaches the model and redact information that is not necessary for the task.
- Define an escalation route for uncertain, harmful, high-impact, or policy-sensitive requests.
- Use human approval where an error could create material operational, legal, financial, safety, or reputational consequences.
- Make audit records sufficient to reconstruct the decision without storing more sensitive content than necessary.
For implementation ideas, see the human-in-the-loop workflow guide, particularly when review queues and approval states need to be explicit.
What to double-check
Use the following threat-model worksheet for each LLM feature:
- Assets: What must be protected? Consider personal data, credentials, confidential documents, internal instructions, financial records, reputation, and system availability.
- Entry points: Where can untrusted text enter? Include chat, files, web retrieval, email, APIs, plugins, tool results, memory, and other model outputs.
- Trust boundaries: Which components are trusted to make security decisions? The model should not be the only component responsible for authorisation.
- Actions: What can the system read, write, send, execute, or delete? Classify actions as read-only, reversible, sensitive, or irreversible.
- Controls: Which safeguards apply at input, retrieval, tool-call, output, identity, and monitoring layers?
- Failure response: What happens when a control fails, a request is ambiguous, or a suspicious instruction is detected?
| Risk area | Useful control | Evidence to retain |
|---|---|---|
| Untrusted instructions | Content and instruction separation | Input classification and test results |
| Private data exposure | Access checks before retrieval | Identity, tenant, and source records |
| Unsafe tool use | Allowlisted tools and validated arguments | Tool-call logs and rejected calls |
| Irreversible actions | Human approval and transaction limits | Approval decision and action outcome |
| Prompt or model changes | Version control and regression tests | Version, test score, and release record |
Output validation deserves particular attention. Do not assume a safe-looking sentence means the operation was safe. Validate structured outputs against a schema, check destinations and identifiers in application code, and re-authorise actions immediately before execution. If the model produces a database query, command, URL, or API request, treat it as a proposal that requires independent validation.
Keep prompts, tool definitions, policies, and evaluation cases under version control. The prompt version control guide can help teams compare and roll back changes when a new instruction or model produces unexpected behaviour.
Common mistakes
- Relying on a secret system prompt: secrecy does not replace authorisation, input validation, or least-privilege access.
- Trusting retrieved content because it came from an approved source: approved sources can still contain accidental or malicious instructions, and permissions can become outdated.
- Giving an agent broad tools for convenience: a single unrestricted tool can turn a low-risk text task into a high-impact security issue.
- Testing only obvious jailbreak phrases: indirect attacks may arrive through realistic documents, tool results, multilingual content, encoded text, or multi-step conversations.
- Logging too little: without prompt, source, tool, identity, and outcome context, investigation becomes guesswork.
- Logging everything indefinitely: excessive logs can create a separate privacy and access-control problem. Define retention and restrict log access.
- Treating refusal as proof of security: the model may refuse a test prompt but still expose data through a tool, a side channel, or a later conversation turn.
- Skipping post-launch checks: provider changes, model changes, new retrieval sources, and new tools can alter the threat profile.
Application security should also cover conventional weaknesses around authentication, authorisation, secrets management, network access, dependency updates, and API abuse. Prompt engineering is useful for shaping behaviour, but it is not a substitute for those controls.
When to revisit
Re-run this checklist before launching an LLM feature, adding a retrieval source, granting a new tool permission, changing a model or provider, revising a production prompt, or connecting the system to a new business process. It is also worth reviewing after a security finding, unexpected tool call, data-access incident, major workflow change, or significant update to the agent framework.
At minimum, maintain a small adversarial test suite with direct and indirect injection cases. Include attempts to extract hidden instructions, access another user’s data, alter tool arguments, bypass approval, trigger an external action, and persist malicious instructions in memory or retrieved content. Record expected behaviour, actual behaviour, severity, owner, and remediation status.
Use the LLM app launch and monitoring checklist to connect pre-launch testing with operational review. During seasonal planning cycles or other scheduled governance reviews, confirm that tool permissions, data sources, approval rules, logs, and escalation contacts still match the current workflow.
Finally, assign ownership. A practical security review should end with named actions: remove unnecessary access, add a failing test, tighten a schema, revise an approval rule, update monitoring, or document an accepted risk. Revisit the checklist whenever the underlying inputs change, not only when the model changes.
