System Prompt vs User Prompt vs Tool Instructions: A Practical Guide

Overview

The short version is this: not all instructions in an LLM application should live in the same place. A stable application usually separates instruction layers by purpose.

System prompts define the durable operating rules for the assistant. They usually cover role, boundaries, output style, safety constraints, and what the model should optimise for across a whole session or workflow.

User prompts contain the task request, context supplied by the person or calling application, and the variables that are expected to change from one interaction to the next.

Tool instructions define how the model should use external tools, APIs, retrieval systems, or functions. They clarify when a tool is available, what inputs it needs, what outputs it returns, and what the model should do before and after calling it.

This distinction matters because many teams still treat prompting as one long string. That can work for demos, but it becomes fragile in production. Once you add retrieval, function calling, browser actions, code execution, summarisation pipelines, or operational workflows, prompt hierarchy becomes part of application design rather than a writing exercise.

A useful mental model is:

• System prompt: who the assistant is and how it should behave
• User prompt: what the user wants right now
• Tool instructions: how the assistant may interact with external capabilities

In practice, these layers overlap. A system prompt may mention that tools exist. A user prompt may request tool use. A tool schema may include descriptions that shape model behaviour. But the more clearly you separate responsibilities, the easier it is to reason about failures.

For example, if a model gives the wrong tone, that is often a system prompt issue. If it answers the wrong question, that is often a user prompt or context issue. If it passes malformed arguments to a function or calls the wrong tool, that is usually a tool instruction or interface design issue.

This is why system prompt vs user prompt is not just a terminology question. It affects evaluation, safety, prompt versioning, latency, and maintenance. If your application includes retrieval, this also intersects with document injection strategy and evidence handling. If you are building multi-step agents, the distinction becomes even more important because the model is no longer only answering questions; it is making choices inside a workflow.

How to compare options

When teams discuss prompt hierarchy, they often ask which layer is “best.” That is the wrong comparison. The useful question is: which instruction belongs at which layer?

Here is a practical framework for comparing system prompts, user prompts, and tool instructions in an LLM app development workflow.

1. Compare by stability

Ask how often the instruction should change.

• If it should remain stable across many sessions, it likely belongs in the system prompt.
• If it changes per task or per user, it belongs in the user prompt.
• If it changes when tools, schemas, or external APIs change, it belongs in tool instructions.

Example: “Respond in concise bullet points and mention uncertainty where relevant” is usually system-level. “Summarise this incident report for the infrastructure team” is user-level. “Use the ticket lookup function before answering questions about incident status” is tool-level.

2. Compare by authority

Instruction layers often have an implied precedence. Exact behaviour depends on the API and orchestration stack, but a common design assumption is:

1. System-level rules establish the operating frame
2. User instructions define the task inside that frame
3. Tool instructions govern how the model interacts with external systems

The lesson is not to rely on hidden precedence rules alone. Instead, remove avoidable conflicts. If the system prompt says “never speculate” while a user asks for imaginative brainstorming, you need a design decision, not a random outcome. If the tool description says “use this tool for account status” but the system prompt tells the model to answer directly, you have created an avoidable contradiction.

3. Compare by failure mode

Each instruction layer tends to fail in different ways.

• System prompt failures: wrong tone, wrong format, policy drift, role confusion, verbosity issues
• User prompt failures: ambiguous objectives, missing context, weak constraints, unclear success criteria
• Tool instruction failures: unnecessary tool calls, missing tool calls, malformed arguments, poor tool selection, weak handling of returned data

When debugging, identify the failure first, then inspect the likely layer. This is faster than rewriting everything at once.

4. Compare by testability

Good prompt engineering makes each layer testable on its own.

A system prompt should be testable across a suite of representative tasks. A user prompt template should be testable with controlled input variables. Tool instructions should be tested using realistic schemas, edge-case arguments, and failure responses from the tool.

If you cannot tell which layer caused a bad result, your architecture is probably too entangled. This is one reason teams benefit from prompt versioning and change logs. If you need a process for that, see Prompt Versioning Best Practices: How Teams Track Changes, Tests, and Rollbacks.

5. Compare by token cost and maintenance cost

Long prompts are not always bad, but every extra instruction has a cost. Stable instructions repeated in every user message create unnecessary token overhead and make maintenance harder. Tool descriptions that are overly verbose can also degrade clarity.

A practical rule is to keep stable rules high-level and central, task-specific details local to the request, and tool instructions explicit but minimal. If a detail belongs in code or schema design instead of natural language, prefer the structured option.

Feature-by-feature breakdown

This section turns the comparison into an implementation reference you can revisit as model APIs change.

System prompts: what they are good for

System prompts work best for durable behavioural guidance. They are well suited to:

• Role definition
• Output conventions
• Risk boundaries
• Reasoning style constraints, where supported
• Global instructions for evidence use, uncertainty, and tone
• Persistent workflow rules such as “ask clarifying questions when required data is missing”

A strong system prompt is usually short enough to remain legible and specific enough to shape behaviour. It should not try to encode every business rule in prose. If you put too much into it, you create a hard-to-maintain policy blob.

Good pattern: define stable norms.

You are an internal support assistant for technical teams.
Be concise, prefer bullet points, and state uncertainty clearly.
Do not invent missing system state.
If account or incident data is required, use available tools before answering.

Common mistake: stuffing changing task details into the system prompt. If the task changes often, it probably does not belong there.

User prompts: what they are good for

User prompts are the task layer. They should express the current objective, relevant context, desired output, and any local constraints.

They are well suited to:

• Current questions and requests
• Task-specific background
• Audience and formatting needs for this output
• Input documents or variables
• Success criteria for the current turn

Good pattern: let the user prompt specify the job to be done.

Summarise the following postmortem for the on-call team.
Highlight root cause, customer impact, remediation steps, and unresolved risks.
Keep it under 150 words.

Common mistake: asking the model to resolve missing context it does not have. If the task depends on account data, retrieval, or an external system, pair the user prompt with tool access or a retrieval layer rather than expecting the model to guess.

This distinction also matters in AI productivity tools such as a text summarizer, keyword extractor, or sentiment analyzer. The user request should define the specific job, while the system prompt sets stable behaviour like output format and confidence handling. If you publish browser-based utilities, this same design principle helps keep product behaviour consistent across use cases.

Tool instructions: what they are good for

Tool instructions are often under-designed. Teams focus on the prompt but ignore the fact that tool descriptions, parameter schemas, and call policies strongly influence behaviour.

Tool instructions are well suited to:

• When a tool should be used
• When a tool should not be used
• How to map natural language requests to tool arguments
• How to interpret tool outputs
• What fallback behaviour to follow if a tool fails

Good pattern: make tool selection criteria clear and concrete.

Use get_ticket_status when the user asks about a live incident, ticket number, owner, or current status.
Do not use it for general explanations of incident process.
If the tool returns no matching ticket, say that no live record was found and ask for a valid ID.

Common mistake: vague tool descriptions such as “retrieves useful information.” That pushes ambiguity back onto the model.

Another common mistake is writing tool guidance only in natural language while leaving parameters underspecified. If the tool expects structured JSON, validate the schema carefully. This is similar to why developers use a formatter before shipping config or API payloads. Related references on bot365 include the JSON Formatter and Validator Guide and the SQL Formatter Guide.

Prompt hierarchy in agent prompt design

In agentic systems, the hierarchy usually expands beyond three simple layers. You may have:

• Application-level system instructions
• Step-specific planner prompts
• Tool descriptions and schemas
• Retrieved documents
• Intermediate memory or state summaries
• User messages

This is where LLM instruction layers become a design concern. If the model receives conflicting guidance from retrieved text, previous steps, and tool descriptions, your orchestration logic needs to decide what wins.

A practical approach is to separate layers by function:

1. Policy layer: stable rules and behavioural constraints
2. Task layer: current objective and output requirements
3. Capability layer: tools, APIs, retrieval, and allowed actions
4. Evidence layer: retrieved documents or structured results
5. State layer: what has already happened in the workflow

That architecture reduces confusion when you later add retrieval-augmented generation, browser actions, or workflow automation. If your app depends on retrieval, see How to Build a RAG Pipeline for the evidence side of this problem.

Where people usually get prompt hierarchy wrong

Across prompt engineering examples, a few mistakes appear repeatedly:

• Using the system prompt as a dumping ground for every rule
• Repeating permanent instructions in every user turn
• Relying on tool names alone without clear usage criteria
• Letting retrieved content behave like hidden instructions
• Failing to define fallback behaviour for missing or failed tool responses
• Changing multiple instruction layers at once, making evaluation impossible

These mistakes create the illusion that the model is inconsistent, when the real issue is often inconsistent application design.

Best fit by scenario

To make the comparison practical, here is how to decide which layer should carry the instruction in common build scenarios.

Scenario 1: Customer support assistant

Best use of system prompt: set tone, refusal boundaries, escalation rules, and rules against inventing account state.

Best use of user prompt: include the current customer question and relevant case notes.

Best use of tool instructions: define when to check CRM records, ticket systems, or account status tools before answering.

If the assistant answers confidently about an account without checking records, that is probably a tool instruction or orchestration problem, not just a wording problem.

Scenario 2: Internal developer copilot

Best use of system prompt: require concise technical answers, explicit assumptions, and safe handling of credentials or tokens.

Best use of user prompt: provide the bug, stack trace, file excerpt, or expected behaviour.

Best use of tool instructions: specify when to search code, run tests, inspect logs, or validate payloads.

If your workflow touches auth debugging, a structured inspection tool is often better than plain prompting. For related troubleshooting patterns, see the JWT Decoder Guide.

Scenario 3: Content operations workflow

Best use of system prompt: define editorial standards, brand tone, and handling of uncertainty.

Best use of user prompt: specify the content asset, audience, channel, and desired transformation such as summary, keyword extraction, or sentiment review.

Best use of tool instructions: route the task to the right utility or API, such as a text summarizer, keyword extractor, or sentiment analyzer.

If you are comparing those utility patterns, related reading includes Best Text Summarizer Tools Compared, Keyword Extraction Tools Compared, and Sentiment Analysis Tools Compared.

Scenario 4: Multi-step agent with scheduling or operations tasks

Best use of system prompt: define the agent’s role, safe operating boundaries, and requirement to confirm destructive actions.

Best use of user prompt: state the desired operational task, such as creating a recurring job or checking schedule intent.

Best use of tool instructions: define when to use a cron helper, validator, or execution service, and how to explain the result back to the user.

Where structured syntax matters, helper tools reduce ambiguity. A useful companion reference is the Cron Expression Builder Guide.

A simple decision rule

If you are unsure where an instruction belongs, ask three questions:

1. Should this remain true across most or all tasks? Put it in the system layer.
2. Is this specific to the current request? Put it in the user layer.
3. Does this govern how external capabilities are selected or used? Put it in the tool layer.

That rule will not solve every edge case, but it prevents most prompt hierarchy problems before they spread through your app.

When to revisit

Prompt hierarchy is not a one-time design choice. It should be revisited whenever the surrounding system changes.

Review your system prompt, user templates, and tool instructions when:

• You switch to a new model or API format
• You add function calling, retrieval, browser actions, or code execution
• Your output quality changes after a model update
• You add new tools or modify existing schemas
• You see repeated failures in tool selection or argument generation
• Your compliance, security, or approval workflow changes
• You expand from single-turn prompting to agent workflows

A practical review process looks like this:

1. Map each instruction to a layer. Remove duplicates and contradictions.
2. Create a small test set. Include normal tasks, edge cases, and failure cases.
3. Change one layer at a time. Do not rewrite the whole stack unless you are intentionally redesigning it.
4. Measure the right outcome. Check instruction following, tool use quality, output format, and failure recovery separately.
5. Version everything. Track prompt changes, tool schema changes, and orchestration changes together.

It also helps to define an evaluation rubric before you start tuning. Otherwise teams often optimise for fluency and miss factual or operational failures. For a practical framework, see AI Output Evaluation Rubric for Marketing Teams. Even if your use case is not marketing, the underlying evaluation logic is still useful.

The most durable takeaway is simple: treat prompt hierarchy as interface design. System prompts, user prompts, and tool instructions are not competing alternatives. They are different control surfaces in the same application. When each layer has a clear job, your prompts become easier to reason about, your tools become easier to trust, and your agent workflows become much easier to maintain.

If you want one practical action to take after reading this article, audit one live prompt flow this week. Split the instructions into three columns: system, user, and tool. Then remove anything that does not clearly belong. That small exercise usually reveals why a model feels unreliable—and gives you a cleaner foundation for future prompt engineering work.