Content libraryChecklist

Pre-Production Launch Checklist

Tagging, budgets, alerts, fallbacks, and incident readiness before launching an LLM feature.

MarkdownChecklist

Pre-Production Launch Checklist — LLM-Powered Features

Feature Name: _________________________
Service/Team: _________________________
Target Launch Date: ___________________
Completed By: _________________________

1. Cost Estimation ✦ Finance

  • Projected daily request volume estimated
    Source: product analytics, load testing, or comparable feature data.
    Why: Without volume projections, you cannot estimate costs or set budgets.

  • Cost per request calculated
    Formula: (avg_input_tokens × input_rate + avg_output_tokens × output_rate) / 1M
    Why: This is your fundamental unit cost. It must be known before launch.

  • Monthly cost projection computed
    Formula: cost_per_request × daily_volume × 30
    Projected monthly cost: $________
    Why: Finance needs this for budget approval and forecasting.

  • Monthly budget approved by finance partner
    Budget: $________ | Approved by: ________
    Why: Unapproved AI spend creates surprise bills and organizational friction.

  • Cost ceiling (hard cap) configured
    Daily ceiling: $________ | Monthly ceiling: $________
    Why: Prevents runaway costs from bugs, retry loops, or traffic spikes.

  • Per-request cost guard configured
    Max cost per request: $________
    Why: A single malformed request shouldn't consume your daily budget.

  • Batch vs. real-time decision documented
    Mode: ☐ Real-time ☐ Batch ☐ Hybrid
    Why: Batch API is 50% cheaper. Only use real-time if latency is critical.

  • Model selected with cost-quality tradeoff documented
    Model: ________ | Reason: ________
    Why: Teams often default to the most expensive model. Justify the choice.

  • Scaling cost projection completed
    Cost at 2× volume: $________ | Cost at 10× volume: $________
    Why: Features that cost $5K/month at launch may cost $50K at scale.

  • Caching opportunity assessed
    Cacheable? ☐ Yes ☐ No | Expected hit rate: ___%
    Why: Even 20% cache hit rate reduces costs by 20% for free.

2. Quality Assurance ✦ Engineering

  • Test suite created with ≥200 test cases
    Test set location: ________
    Why: You cannot measure quality regression without a benchmark.

  • Accuracy/quality benchmarks met
    Metric: ________ | Score: ________ | Threshold: ________
    Why: Ensures the model meets minimum quality for the use case.

  • Edge cases tested (10%+ of test set)
    Examples: empty input, max-length input, adversarial input, non-English, special characters
    Why: Edge cases cause production failures and wasted tokens.

  • Output format validation implemented
    ☐ JSON schema validation ☐ Regex check ☐ Type checking
    Why: Malformed outputs waste downstream processing and may require retries.

  • Hallucination rate assessed
    Rate: ____% | Threshold: ____% | Mitigation: ________
    Why: Hallucinations create user trust issues and potential liability.

  • Human evaluation completed (sample of 50+ responses)
    Average score: ____/5.0 | Threshold: ____/5.0
    Why: Automated metrics miss nuances that humans catch.

  • Prompt version pinned and stored in version control
    Commit hash: ________ | Prompt ID: ________
    Why: Enables rollback and cost-quality tracking per prompt version.

  • Latency tested under expected load
    P50: ____ms | P95: ____ms | P99: ____ms | SLO: ____ms
    Why: Users won't wait. Latency SLOs must be met before launch.

3. Instrumentation ✦ Platform

  • Tagging schema applied
    Tags: team=____, service=____, feature=____, use_case=____
    Why: Cost attribution requires consistent tagging from day one.

  • Requests routed through centralized gateway
    Gateway: ☐ LiteLLM ☐ Custom ☐ Other: ________
    Why: Gateway enables tagging, routing, rate limiting, and cost logging.

  • Token usage logging verified
    Logged fields: input_tokens, output_tokens, model, latency, cost
    Why: Cannot optimize what you don't measure.

  • Cost dashboard created/updated
    Dashboard URL: ________
    Why: Real-time visibility enables fast detection and response.

  • Cost allocated to correct cost center
    Cost center: ________ | Budget line: ________
    Why: Chargeback requires accurate cost center mapping.

  • Request tracing enabled (correlation ID)
    Trace ID format: ________ | Propagated to LLM calls: ☐ Yes
    Why: Debugging production issues requires end-to-end tracing.

  • Metrics exported to monitoring system
    System: ☐ Prometheus ☐ Datadog ☐ CloudWatch ☐ Other
    Why: Centralized monitoring enables cross-service cost views.

  • Baseline metrics recorded
    Pre-launch baseline: cost/request $____, latency ____ms, quality ____%
    Why: Post-launch comparison requires a pre-launch baseline.

4. Security ✦ Security

  • PII filtering enabled on inputs
    Method: ☐ Regex ☐ NER model ☐ Provider-side ☐ N/A
    Why: Sending PII to LLM providers may violate data protection regulations.

  • API keys stored in secrets manager
    ☐ AWS Secrets Manager ☐ GCP Secret Manager ☐ Vault ☐ Other
    Why: Hardcoded API keys in code are a security vulnerability.

  • Tenant isolation verified (multi-tenant systems)
    Isolation method: ☐ Separate API keys ☐ Tagging ☐ N/A (single tenant)
    Why: Tenant A must not see tenant B's data in responses.

  • Data residency requirements checked
    Required region: ________ | Provider region: ________
    Why: Some regulations require data to stay within geographic boundaries.

  • Provider DPA (Data Processing Agreement) signed
    ☐ Yes ☐ In progress ☐ N/A
    Why: Required for GDPR compliance when processing personal data.

  • Training data opt-out confirmed
    Provider training opt-out: ☐ Confirmed ☐ N/A
    Why: Enterprise data should not be used to train public models.

5. Operations ✦ SRE/Platform

  • Budget alerts configured
    Alert at: 50% ($), 75% ($), 90% ($), 100% ($)
    Why: Graduated alerts enable proactive cost management.

  • Cost anomaly alert configured
    Threshold: >____% deviation from rolling average
    Why: Detects runaway costs, retry loops, and traffic spikes.

  • Latency alert configured
    Alert if P95 > ____ms for > ____ minutes
    Why: Latency spikes indicate provider issues or prompt regression.

  • Error rate alert configured
    Alert if error rate > ____% for > ____ minutes
    Why: Elevated errors waste tokens on retries and degrade user experience.

  • Runbook written for cost incidents
    Runbook location: ________
    Why: On-call engineers need step-by-step guidance during incidents.

  • Escalation contacts documented
    L1: ________ | L2: ________ | Finance: ________
    Why: Clear escalation prevents delayed response to cost incidents.

  • Rollback plan tested
    Rollback method: ☐ Feature flag ☐ Model revert ☐ Gateway config
    Why: If the launch causes issues, you need a tested rollback path.

  • Rate limiting configured
    RPM limit: ________ | TPM limit: ________
    Why: Prevents runaway consumption from bugs or abuse.

6. Sign-Off

Role Name Signature Date
Engineering Lead
Product Owner
Finance/FinOps Partner
Security Review

Launch Decision: ☐ Approved ☐ Approved with conditions ☐ Not approved

Conditions (if any): _______________________________________________

Template from the TokenOps Atlas — tokenops-atlas