AI Agent Tool-Calling Design Guide: Permissions, Schemas, Retries, and Human Approval
Reliable AI agent tool calling depends on more than connecting an LLM to an API. This guide provides a practical design workflow for defining tools, limiting permissions, validating arguments, handling failures, recording decisions, and introducing human approval where automation could create material risk.
Overview
Tool calling, also called function calling, allows an AI model to request an operation from your application. The model may ask to search a knowledge base, retrieve an account record, create a ticket, send a message, or update a database. Your application—not the model—should validate the request, enforce permissions, execute the operation, and return a controlled result.
The safest design treats every tool as a small, governed application interface. A tool should have a narrow purpose, an explicit input schema, a defined permission boundary, predictable error responses, and an audit trail. Avoid giving an agent a broad “run anything” capability when several focused tools would provide better control.
Tool calling should also be evaluated as part of the wider LLM application. Combine it with prompt injection defences, regression tests, monitoring, cost controls, and human review. The LLM Evaluation Framework can help you build a repeatable test set, while the LLM App Launch and Post-Launch Monitoring checklist is useful when moving from testing into production.
What to track
1. Tool purpose and description
Start with a tool register. For each tool, record its owner, purpose, data accessed, systems affected, expected users, and operational limits. The description supplied to the model should explain when the tool is appropriate and when it must not be used.
Prefer descriptions such as “Retrieve the current status of an order using an authenticated order ID” over vague labels such as “Manage orders”. State whether the tool only reads information or can change it. If a tool has side effects, say so plainly in both the model-facing description and the application documentation.
2. Argument schemas and validation
Use a strict schema for every argument. Define required fields, permitted values, formats, maximum lengths, and sensible ranges. Validate arguments again in application code; a schema shown to the model is not a security boundary.
For example, a refund tool might require an order identifier, a currency amount, and a reason selected from an allowlist. The server should confirm that the order belongs to the authenticated user, that the amount is within the permitted limit, and that the refund has not already been processed. Do not rely on the model to make those checks.
Reject unknown fields where your framework supports that behaviour, normalise values before processing, and return errors that explain what can be corrected without exposing internal implementation details.
3. Permission boundaries
Assign permissions to the calling user, service, or workflow—not simply to the model. A tool should run with the minimum access needed for its stated purpose. Separate read permissions from write permissions, and separate low-impact writes from actions that are difficult to reverse.
Use server-side checks for tenant, identity, ownership, role, environment, and resource scope. A prompt instruction such as “only access this customer’s records” is useful context but cannot replace access control. Keep credentials outside prompts and tool arguments, and pass secrets through controlled application configuration.
4. Idempotency and duplicate calls
Models may repeat a request after a timeout, or an application may retry a request whose outcome is uncertain. Any tool that creates, sends, charges, publishes, or updates should have an idempotency strategy. Accept a unique operation key, store the result associated with that key, and return the original result when the same operation is received again.
For non-idempotent actions, consider a two-stage design: first create a proposed action, then require an explicit confirmation or approval before execution. This reduces the chance that a repeated model response creates duplicate side effects.
5. Timeouts, retries, and failure states
Set a timeout for every external call. Without one, a slow dependency can consume the entire agent run and encourage uncontrolled retries. Retry only failures that are likely to be temporary, such as a recoverable network problem. Use bounded exponential backoff with a maximum attempt count, and avoid retrying validation errors, permission failures, or rejected business rules.
Define distinct failure states, such as invalid input, unauthorised request, unavailable dependency, conflicting update, rate limit, and unknown outcome. Return structured error codes to the orchestration layer. The agent can then explain the next step rather than inventing a successful result.
6. Audit logs and privacy
Record the user or service identity, agent run ID, tool name and version, validated arguments or a privacy-safe representation, approval decision, execution status, timestamps, latency, retry count, and resulting error code. Logs should support investigation without unnecessarily copying sensitive data into every record.
Define retention and access rules for tool logs. Redact secrets and consider masking personal or financial information. If you change a tool schema or permission, preserve enough version information to reconstruct which contract was active when an action occurred.
Decision table for tool risk
| Tool category | Typical examples | Default control |
|---|---|---|
| Read-only | Search documents, check status, retrieve an account detail | Automatic execution with scoped access, validation, logging, and output filtering |
| Reversible write | Create a draft, add an internal note, open a low-impact ticket | Automatic execution only within defined limits; use idempotency and clear user feedback |
| Material or external write | Send a customer message, change a record, publish content | Require confirmation, policy checks, or human approval depending on context |
| Irreversible or high-impact | Delete data, transfer funds, disclose sensitive information | Human approval, strong authentication, explicit scope, and a complete audit trail |
7. Human approval workflow
Human approval should be specific, not a vague “review this” step. Show the proposed action, affected resource, material arguments, reason, expected consequence, and any policy warnings. The reviewer should be able to approve, reject, edit, or request clarification.
Keep approval separate from execution. A valid approval should expire after a defined period or when important arguments change. Bind it to the exact user, resource, tool version, and action being approved. If the model changes the amount, recipient, or target after approval, send the action back through review.
For a practical review-queue pattern, see the AI Human-in-the-Loop Workflow guide.
Tool-review template
- Tool name and version: What stable identifier appears in logs and documentation?
- Purpose: What single job does the tool perform?
- Inputs: Which fields are required, constrained, normalised, or rejected?
- Data scope: Which users, tenants, records, and environments can it access?
- Side effects: What changes, messages, costs, or disclosures can occur?
- Permission: Which identity and role may invoke it?
- Idempotency: How are duplicate and uncertain requests handled?
- Failure handling: Which errors are retried, surfaced, or escalated?
- Approval: Which conditions require confirmation or human review?
- Observability: What is logged, monitored, redacted, and retained?
- Tests: Which normal, invalid, adversarial, and permission-boundary cases pass?
Cadence and checkpoints
Review tool calling at several levels rather than waiting for a serious incident.
- Per change: Review the tool whenever its schema, prompt description, permissions, dependency, retry policy, or output format changes. Run focused tests before release.
- Monthly: Inspect failed calls, repeated retries, timeouts, approval rejection rates, unknown outcomes, and unusual access patterns. Sample successful calls as well as failures; a successful but incorrectly scoped action is still a defect.
- Quarterly: Reconfirm owners, service accounts, access scopes, approval thresholds, log retention, and dormant tools. Re-run permission-boundary and prompt-injection tests, and check that provider or SDK changes have not altered supported tool-calling behaviour.
- After an incident: Freeze or narrow the affected tool, preserve relevant logs, reproduce the failure safely, and document the corrective change. Do not treat a prompt edit as a complete fix when the underlying issue is missing server-side validation.
Keep a changelog for tool definitions and agent prompts. The Prompt Versioning System guide offers a useful model for naming, reviewing, and rolling back changes. Where tools are used in content or operations workflows, pair the review with the AI Content Workflow Audit.
How to interpret changes
A rise in tool-call failures does not automatically mean the model has become less capable. First separate the causes: schema changes, stricter validation, expired credentials, dependency outages, user behaviour, prompt changes, or a model update. Compare the same test set and segment production data by tool, version, error class, workflow, and user permission.
Watch for combinations that indicate risk. More retries alongside higher latency may point to an unstable dependency. More successful calls with more human rejections may indicate that the model is producing plausible but unsuitable actions. A lower error rate after loosening validation may represent reduced detection rather than improved reliability.
Review quality, not just availability. For read tools, check whether returned data is relevant and correctly scoped. For write tools, verify the intended state change, duplicate prevention, approval binding, and user-visible confirmation. Connect tool results to end-to-end evaluations instead of measuring function-call syntax alone. For input-manipulation risks, use the Prompt Injection Defense Guide as part of the test plan.
When to revisit
Revisit this guide whenever you add a tool, broaden access, change a model or provider, modify an approval threshold, or connect a new external system. Schedule a monthly operational review and a more complete quarterly access and design review. Also review immediately after a duplicate action, unauthorised data exposure, unexplained tool result, provider migration, or material change to the agent’s role.
For each review, take one representative read-only tool, one reversible write, and one high-impact action through the full workflow: generate a request, validate it, enforce permissions, simulate failure, retry where appropriate, request approval, execute, and inspect the audit record. Record the result and assign an owner for every gap.
Before deployment, use this short checklist:
- Limit every tool to a narrow, documented purpose.
- Validate all arguments and permissions on the server.
- Define idempotency, timeouts, bounded retries, and explicit failure states.
- Separate proposals, approvals, and execution for risky actions.
- Log enough context to investigate while protecting sensitive data.
- Test normal, malformed, repeated, unauthorised, delayed, and adversarial requests.
- Set a next review date and record the tool version, owner, and decision.
Tool calling becomes easier to trust when it is treated as a governed interface rather than an informal extension of a prompt. A recurring review gives your team a practical way to detect permission drift, identify fragile integrations, and keep AI automation within clearly understood boundaries.