MarkdownChecklist

Model Upgrade/Migration Checklist

Current Model: _________________________ → New Model: _________________________
Service(s) Affected: ___________________
Migration Owner: _______________________
Target Date: ___________________________

1. Pre-Migration Assessment

Current model performance benchmarked
Accuracy: ____% | Latency P95: ms | Cost/request: $
Record baseline metrics before any changes.
New model benchmarked on same test set
Accuracy: ____% | Latency P95: ms | Cost/request: $
Must use identical test set for fair comparison.
Cost comparison calculated
Current monthly cost: $____ | Projected monthly cost: $____
Savings/increase: $____ (___%)
Ensure the cost impact is understood and approved.
Latency comparison measured under load
New model TTFT: ____ms | TPS: ____tok/s
Test under realistic production load, not just single requests.
Context window differences checked
Current: ____K tokens | New: ____K tokens
Smaller context windows may truncate existing prompts.
Quality threshold defined for approval
Metric: ________ | Minimum acceptable: ________
"The new model passes if accuracy ≥ X%."
Rate limits compared
Current RPM: ________ | New RPM: ________
Lower rate limits may require request queuing.
Pricing model differences noted
☐ Same pricing structure ☐ Different input/output rates ☐ Different billing unit
Some models charge differently (e.g., cached tokens, reasoning tokens).

2. Compatibility Check

Prompt format compatibility verified
☐ Same system/user/assistant format ☐ Changes needed: ________
Some models handle system prompts differently.
Response format compatibility verified
☐ JSON mode available ☐ Structured output supported ☐ Changes needed
Not all models support JSON mode or structured outputs.
Tool/function calling API compatibility checked
☐ Compatible ☐ Requires changes ☐ Not applicable
Function calling schemas may differ between providers.
Max output token limits checked
Current max: ______ | New max: ______
Some models have lower output limits.
Streaming API compatibility verified
☐ SSE streaming compatible ☐ Changes needed
Streaming chunk format may differ.
Special tokens / stop sequences checked
☐ Same behavior ☐ Adjustments needed
Stop sequences may behave differently on new models.

3. Testing Protocol

Shadow testing configured (compare without serving)
Duration: ____ days | Traffic: ____%
Run both models in parallel; compare outputs.
A/B test plan written
Test group: ____% | Control group: ____%
Clear split for measuring real-world impact.
Quality gates defined with pass/fail criteria
Gate 1: Accuracy ≥ ____% → proceed to canary
Gate 2: No new failure modes → proceed to 50%
Gate 3: User satisfaction stable → proceed to 100%
Explicit criteria prevent "it seems fine" deployment.
Rollback triggers documented
Auto-rollback if: quality drops > ____%, error rate > ____%, latency > ____ms
Automated triggers catch issues faster than humans.
Edge case regression test completed
Failures: ____ out of ____ edge cases
New models may fail on edge cases the old model handled.
Multi-language / locale testing completed (if applicable)
☐ Tested ☐ Not applicable
Model quality varies by language.
Long-context test completed (if applicable)
Tested at ____K tokens | Quality: ☐ Pass ☐ Fail
Performance often degrades at context window edges.
Concurrency/load test completed
Tested at ____× production load | Results: ☐ Pass
New model endpoints may have different scaling characteristics.

4. Deployment

Canary deployment configured
Canary traffic: ____% | Duration: ____ hours
Start small to catch issues before full rollout.
Feature flag configured for instant rollback
Flag name: ________ | Rollback method: ________
One-click rollback capability is non-negotiable.
Monitoring dashboards updated for new model
☐ Cost dashboard ☐ Quality dashboard ☐ Latency dashboard
Dashboards must show the new model as a distinct entity.
Alert thresholds recalibrated
New baselines: cost $____, latency ____ms, error rate ____%
Old thresholds may trigger false alarms or miss real issues.
Team notified of migration schedule
☐ Slack notification ☐ Email ☐ Standup announcement
On-call engineers must know a migration is in progress.
Rollout schedule defined
5% → 25% → 50% → 100% over ____ days
Gradual rollout with bake time at each stage.

5. Post-Migration Validation

Quality metrics compared to pre-migration baseline
Before: ____% | After: ____% | Delta: ____%
Must be within defined threshold.
Cost validated against projections
Projected: $____ | Actual: $____ | Variance: ____%
Investigate if actual differs from projected by >10%.
Latency validated within SLO
P50: ____ms | P95: ____ms | P99: ____ms | All within SLO: ☐ Yes
Latency regression is common with model changes.
Error rate compared to baseline
Before: ____% | After: ____% | Acceptable: ☐ Yes
New models may have different error patterns.
Customer feedback monitored (7 days post-migration)
Complaints: ____ | Positive: ____ | Neutral: ____
Some quality changes only surface through user feedback.
Old model configuration archived
Archived prompts, model config, and routing rules for 30 days
Enables rollback even after the migration is "complete."

Sign-Off

Role	Name	Date	☐ Approved
Migration Owner			☐
Service Owner			☐
Quality Lead			☐

Template from the TokenOps Atlas — tokenops-atlas