Deploying an LLM app is not simply a matter of connecting a model API to a user interface. This production checklist shows how to estimate operating costs, control risk, protect secrets, manage latency, monitor quality, and prepare reliable rollback procedures before real users depend on the system.
Overview
A prototype can appear successful with a small number of test prompts, generous timeouts, and manual review. Production changes the conditions. Requests arrive concurrently, users submit unexpected inputs, providers change models or pricing, and a single slow or incorrect response can affect trust in the entire application.
A reliable LLM app should therefore be assessed across five dimensions:
- Security: API keys, user data, tool permissions, and administrative functions must be protected.
- Economics: expected request volume, token usage, retries, and supporting infrastructure should be modelled before launch.
- Reliability: timeouts, rate limits, provider errors, and partial failures need defined behaviour.
- Quality: outputs should be evaluated against representative tasks rather than judged only through demonstrations.
- Operations: logs, alerts, feedback, versioning, and rollback paths should be ready before traffic increases.
For a broader build sequence, use the LLM App Development Checklist: From Prototype to Production. This article focuses on the production decision: whether the application is controlled and observable enough to operate, and how to recalculate that decision as the inputs change.
How to estimate
Start with a simple unit model. Define one unit of work, such as a completed chat turn, document summary, classification request, or tool-assisted task. Then estimate how many units the application will process in a normal period.
A basic monthly operating estimate is:
Total monthly cost = model usage + embedding or retrieval usage + application infrastructure + observability + storage + contingency
Keep model usage separate from the other categories. This makes it easier to compare providers, models, prompt changes, and caching strategies without hiding operational costs inside a single total.
For token-based pricing, use the provider's current published inputs rather than a permanent figure in application code. A practical calculation is:
Model cost = (input tokens ÷ billing unit × input rate) + (output tokens ÷ billing unit × output rate)
Apply the calculation to a representative request, not an ideal one. Include system instructions, conversation history, retrieved context, tool results, and the expected answer. If the application retries failed requests, estimate retry volume separately. Retries can affect both cost and rate-limit pressure.
For throughput, estimate peak demand as well as monthly demand:
Peak concurrent work ≈ requests per second × average response time in seconds
This is a planning approximation, not a capacity guarantee. It helps identify whether the application needs a queue, concurrency limit, streaming responses, or a smaller fallback model. Test the result under realistic load before relying on it.
Quality should be estimated with a test set. Create a fixed collection of representative inputs and record measures such as valid structured-output rate, task success, citation or retrieval accuracy where relevant, refusal behaviour, latency, and human review outcomes. The Prompt Engineering Evaluation guide provides a useful framework for comparing prompt and model changes without relying on isolated examples.
Inputs and assumptions
1. Request and token assumptions
Record the expected requests per day, peak requests per minute, average input size, average output size, conversation history length, and proportion of requests that use retrieval or tools. Separate new conversations from long-running conversations because history can cause input size to grow over time.
Do not assume that a short user message produces a small request. System prompts, schemas, retrieved passages, previous turns, and tool definitions may account for much of the context. Measure representative payloads where possible and keep a buffer for unusually long inputs.
2. Reliability assumptions
Define an explicit timeout for each external call and an overall deadline for the user request. Decide which failures are safe to retry. A read-only model request may be retried with care; a tool call that creates an order, sends a message, or changes a record requires idempotency protection before retrying.
Set rate limits at more than one level: per user or account, per application, and according to the provider's permitted capacity. Return a clear response when a limit is reached rather than allowing requests to accumulate indefinitely.
3. Security and privacy assumptions
Keep provider credentials in a secrets manager or protected deployment environment. Never place them in browser code, public repositories, prompts visible to users, or client-side logs. Rotate credentials using a documented procedure and limit each key's permissions where the provider supports it.
Map the data that enters the system. Identify personal, confidential, regulated, or commercially sensitive information, then decide whether it should be sent to a model provider, stored in conversation history, included in logs, or retained for evaluation. Redact or minimise data where full content is not needed. Protect administrative tools with authentication, authorisation, validation, and audit records.
For retrieval-augmented applications, treat retrieved text as untrusted input. It can contain irrelevant instructions or content that attempts to influence the model. Keep system-level controls separate from retrieved content, restrict tool access, and validate the model's proposed actions in application code. See the RAG tutorial for a deeper treatment of retrieval design.
4. Output and evaluation assumptions
Use structured outputs when the application needs predictable fields. Validate the response against a schema before it reaches downstream code. Handle missing fields, invalid values, duplicate records, and unexpected extra content explicitly. A valid JSON response is not necessarily a correct response, so schema validation should be followed by business-rule checks.
Store prompt, model, configuration, and application versions alongside evaluation results. This makes regressions traceable when a provider updates a model or a developer changes instructions. Avoid logging sensitive prompt content by default; use redaction, sampling, access controls, and retention limits.
Worked examples
Example 1: estimating monthly model usage
Assume an internal assistant is expected to process 1,200 requests in a month. A representative request contains 1,800 input tokens and produces 500 output tokens. The application uses one model for every request, with no retries in the base estimate.
Total input tokens are 1,200 × 1,800 = 2,160,000. Total output tokens are 1,200 × 500 = 600,000. Convert these totals using the provider's current billing unit and rates, then add infrastructure, logging, storage, and a clearly labelled contingency. If testing shows that 5% of requests retry once, add 60 additional request equivalents to the model estimate rather than treating retries as free.
This example is deliberately expressed as a formula instead of a fixed price. Provider rates, model choices, billing units, and discounts can change. Keeping the inputs in a worksheet or configuration document makes the estimate reusable.
Example 2: comparing a large model with a fallback
Suppose the application handles two classes of work: complex requests that need the primary model and routine requests that can use a smaller fallback. Begin by measuring task success on a labelled evaluation set. If the fallback meets the application's quality threshold for routine tasks, estimate the monthly mix using:
Blended model cost = (complex request share × primary model cost) + (routine request share × fallback model cost)
Do not select the cheaper route on cost alone. Compare structured-output validity, escalation rate, latency, refusal behaviour, retrieval performance, and human correction time. A lower per-request charge can be outweighed by extra reviews or failed downstream actions.
Example 3: capacity and user experience
Assume a peak period produces 0.5 requests per second and the average model response takes 8 seconds. The simple concurrency estimate is 0.5 × 8 = 4 concurrent requests. Add headroom for variation, retrieval calls, retries, and other application work. If the resulting load is close to the tested limit, introduce a queue or concurrency control before launch. Streaming can improve perceived responsiveness, but it does not remove the need to control total work or handle an incomplete response.
When to recalculate
Revisit the production estimate whenever an underlying input changes. At minimum, recalculate after:
- a model, provider, billing unit, or pricing arrangement changes;
- prompt instructions, output schemas, conversation history, or retrieved context changes;
- traffic, peak concurrency, user mix, or average input and output size changes;
- retry rates, timeout rates, tool calls, or fallback usage increase;
- new data types, integrations, regions, retention rules, or access levels are introduced;
- monitoring shows a quality, latency, cost, safety, or uptime regression.
Use a practical maintenance schedule. Review spend, request volume, latency, error rates, rate-limit events, and fallback use weekly or at an interval suited to the application's risk. Run the evaluation set after every prompt, model, retrieval, or schema change. Review logs and user feedback for recurring failure modes, but remove or protect sensitive content before wider access.
Keep a deployment record containing the active model and prompt versions, environment variables, feature flags, evaluation results, known limitations, and rollback steps. A rollback should be tested, not merely documented: confirm that the previous application version, prompt configuration, model route, and database behaviour can be restored without creating duplicate or destructive tool actions.
Before launch, confirm that you can answer yes to these questions: Are secrets outside client code? Are inputs and logs governed? Are timeouts, retries, and rate limits defined? Are outputs validated? Is there a representative evaluation set? Can the team see cost, latency, errors, and quality signals? Is a fallback or degraded experience available? Can the last known-good release be restored? If any answer is no, record the gap, assign an owner, and decide whether it blocks launch.
Production readiness is not a one-time approval. It is a repeatable review of changing inputs, observed behaviour, and acceptable risk. Recalculate the model, test the application, and update the checklist whenever the system or its operating conditions change.
