Moving an LLM application to a new model or provider is more than changing an endpoint. This checklist gives you a repeatable way to compare compatibility, estimate cost, test behaviour, plan the cutover, and preserve a rollback path before migration affects users.
Overview
An LLM API migration can change several layers of an application at once: authentication, request formatting, model capabilities, tool calling, structured output, streaming, rate limits, safety responses, latency, and billing. A successful migration treats these as separate compatibility questions rather than assuming that two providers with similar chat APIs are interchangeable.
The safest approach is to create a migration baseline before selecting a replacement. Record what the current system sends, what it expects back, how much traffic it handles, and which outcomes matter to users. Then compare the target provider against that baseline using representative requests, not only a simple test prompt.
Use the migration decision in three stages:
- Inventory: document the current integration, workloads, limits, costs, and failure handling.
- Validate: run a fixed evaluation set against the existing and target systems.
- Cut over safely: release gradually, monitor the right signals, and keep rollback available.
This process also applies when upgrading a model within the same provider. A model version change may preserve the endpoint while changing output style, tool behaviour, context handling, latency, or refusal patterns.
How to estimate
Estimate migration impact across four dimensions: implementation effort, operating cost, quality, and operational risk. Do not reduce the decision to the advertised token rate or a single benchmark result.
1. Calculate expected usage
Start with a representative period, such as a week or month. Separate input and output usage because providers may measure or price them differently. A basic workload estimate is:
Total input units = requests × average input units per request
Total output units = requests × average output units per request
If your application uses retries, tool calls, or multi-step workflows, include them explicitly:
Effective requests = user requests × average model calls per workflow × retry multiplier
For cost comparison, use the target provider's current published pricing and unit definitions at the time of review. Keep the formula separate from the price so that the worksheet remains reusable when pricing changes:
Estimated usage cost = input units × input unit rate + output units × output unit rate
Add any separately billed features, such as hosted retrieval, tool execution, storage, or audio processing, only when they are part of your actual architecture. Mark estimates as estimates; observed invoices and production telemetry should replace assumptions after launch.
2. Compare outcomes, not just responses
Build a test set from real application tasks, with sensitive data removed or safely substituted. Include routine, difficult, and failure-case prompts. For each case, record:
- Whether the response satisfies the task;
- Whether the output matches the required schema;
- Whether citations, classifications, or extracted fields are correct;
- Whether the model calls the correct tool with valid arguments;
- Response latency and timeout behaviour;
- Unexpected refusals, unsupported claims, or formatting changes.
Where judgement is subjective, define a small scoring rubric before testing. For example, score factual usefulness, instruction following, format validity, and safety handling from zero to two. The purpose is not to create a universal model ranking; it is to identify regressions against your application's requirements.
3. Use a compatibility worksheet
| Area | Current integration | Target integration | Migration action |
|---|---|---|---|
| Authentication | Key, token, project, or role | Required credential and scope | Update secret storage and access checks |
| Messages | Roles, content parts, metadata | Accepted message structure | Map fields and test multimodal content if used |
| Generation controls | Temperature, limits, stop rules | Supported controls and defaults | Translate or remove unsupported parameters |
| Tools | Schema, call format, parallel calls | Supported tool protocol | Adapt schemas and validation |
| Structured output | JSON mode or schema enforcement | Equivalent response feature | Add parsing, repair, and rejection handling |
| Streaming | Event format and termination | Stream protocol and errors | Update parser and disconnect handling |
| Limits | Context, requests, concurrency | Target limits | Set queues, backoff, and truncation rules |
Inputs and assumptions
Before testing, create a migration record with the following inputs. This makes the decision auditable and gives the team a clear basis for recalculation.
- Traffic: average and peak requests, concurrency, geographic distribution, and expected growth.
- Prompt shape: system instructions, conversation history, retrieved context, attachments, and typical input length.
- Output shape: average response length, maximum response length, streaming requirement, and structured fields.
- Workflow depth: tool calls, retries, routing between models, summarisation steps, and human review.
- Service requirements: acceptable latency, availability expectations, timeout values, and recovery behaviour.
- Data controls: retention requirements, logging rules, access controls, and the information that must not enter test prompts.
- Application dependencies: SDK version, framework adapters, observability integrations, background jobs, and deployment configuration.
Keep assumptions visible. If average output length is unknown, use a conservative range and test both ends. If provider limits are unclear, verify them in the current documentation or account dashboard before committing to the design. Do not treat a successful request as proof that production traffic will fit within limits.
Plan for semantic incompatibility as well as technical incompatibility. A target model may return valid JSON but use different field values, tool arguments, tone, refusal language, or ordering. Downstream code should validate outputs and fail safely rather than relying on a particular wording pattern.
Worked examples
Example 1: A support assistant with tool calls
Assume a support assistant receives 10,000 user requests in a review period. Each workflow makes an average of 1.3 model calls, and 5% of calls are retried. The effective call estimate is:
10,000 × 1.3 × 1.05 = 13,650 model calls
The team should then measure average input and output units separately for those calls. It should also test the tool path, not just conversational answers: account lookup, order search, permission failures, malformed arguments, and a tool timeout. A migration that produces good text but calls the wrong function or omits a required argument is not compatible for this workload.
Example 2: A document summarisation workflow
Assume an internal workflow processes 2,000 documents per month. Each document is split into four sections, summarised separately, and then passed through one final consolidation step. The baseline is therefore approximately 10,000 model calls before retries or failed jobs are included:
2,000 × (4 + 1) = 10,000 model calls
For this workload, compare total input units, not only document count. Test short and long documents, tables, incomplete text, duplicate sections, and documents that exceed the target context limit. Record whether the final summary preserves required names, dates, actions, and uncertainty markers. If the target model produces shorter summaries, confirm that the reduction does not remove information needed by downstream users.
Example 3: A staged migration decision
Suppose the target passes the evaluation set but has different streaming events and a stricter request limit. The migration decision is not simply pass or fail. The implementation plan should include a stream adapter, concurrency control, queue monitoring, and a limited traffic release. Keep the original integration available until error rates, latency, output validation, and user feedback remain acceptable over an agreed observation period.
Cutover checklist
- Pin the target model or version in configuration.
- Store credentials in the approved secret-management system.
- Validate request and response schemas at the application boundary.
- Set explicit timeouts, retry limits, and backoff behaviour.
- Test rate-limit responses, malformed output, partial streams, and provider errors.
- Enable dashboards for volume, latency, failures, retries, validation errors, and estimated usage.
- Release to a controlled subset of traffic before full cutover.
- Document the rollback switch and confirm that the previous integration still works.
When to recalculate
Revisit the worksheet whenever a provider changes pricing, usage units, model limits, SDK behaviour, authentication, deprecation guidance, tool-calling support, structured-output behaviour, or streaming format. Also recalculate after meaningful changes to your own application, including a prompt rewrite, longer conversation history, new retrieval sources, additional tools, higher traffic, or a change in retry policy.
A practical review cycle is to check assumptions before each model upgrade or provider switch and to perform a broader compatibility review on a regular engineering maintenance cycle. The exact interval should match your release frequency and the importance of the workflow.
At the end of each review, save a short post-migration form:
- What changed in the provider, model, SDK, or application?
- Which assumptions were confirmed or disproved?
- What were the measured usage, latency, error, and validation results?
- Which test cases regressed, and what mitigation was applied?
- Is rollback still tested and available?
- What should trigger the next review?
Finally, keep prompts, adapters, test cases, and migration notes under version control. A migration is easier to repeat when the team can compare the old and new contracts, reproduce the evaluation, and roll back a known configuration. For related operational controls, see Prompt Version Control, the Prompt Evaluation Framework, and the LLM deployment checklist.
