Token Optimization Techniques

Start routine classification, extraction, tagging, and FAQ work on the smallest model that clears a measured quality threshold.

• Define task-specific quality and latency thresholds before comparing model tiers.
• Benchmark small models on representative production cases, including known failure modes.
• Escalate only low-confidence, failed-validation, or genuinely complex requests.

Watch out: A lower token price is not a saving when retries or quality failures rise. Compare cost per accepted outcome.

Reuse an unchanged prompt prefix instead of re-paying to process it. The single highest-leverage lever for repeated or multi-turn workloads.

• Put the most stable content FIRST: system prompt, role, policies, tool definitions, large reference docs.
• Put the volatile, per-request content (the user's actual question) LAST.
• Keep the prefix byte-identical across calls — even a date or whitespace change busts the cache.
• On Claude, mind the 5-minute TTL (early-2026 change): fire a lightweight keepalive every ~4 min for high-value caches, or pay for the 1-hour TTL when the workload warrants it.

Watch out: Cache writes cost MORE than base input (Claude: +25% for 5-min, +100% for 1-hour). Caching only pays off when the prefix is reused enough times within the TTL.

Strip grammatical scaffolding (articles, auxiliary verbs, pronouns, intensifiers) while preserving the load-bearing words (nouns, verbs, numbers, names, technical terms).

• Apply to system messages, few-shot examples, chat logs, and reference docs you re-feed each session.
• Keep: nouns, main verbs, meaningful adjectives, numbers, uncertainty qualifiers, critical prepositions, time/frequency markers, names, technical terms.
• Drop: articles (a/the), auxiliary verbs, redundant prepositions, pronouns, pure intensifiers.
• Pair with prompt caching — compress the doc once, cache the compressed version, reuse cheaply.

Watch out: Avoid on nuanced reasoning, legal/medical precision, or creative writing where grammar carries meaning. Best for directive tasks: code gen, data transforms, classification.

Match model power to task difficulty. Use budget models for classification, extraction and drafting; reserve flagships for hard reasoning.

• Classify each request's complexity (cheaply, with a small model or rules) before choosing a model.
• Default to the cheapest tier that passes your quality bar; escalate only on failure or low confidence.
• In agent stacks, route subagent/tool work to cheap models — agent teams burn ~7x more tokens than single sessions.
• Examples: Claude Haiku → Sonnet → Opus; Gemini Flash → Pro; GPT-5.x Nano/Mini → full.

Remove stale tool results and thinking blocks once they are no longer relevant, instead of carrying them forward every turn.

• Prune old tool outputs, logs, and intermediate reasoning that no longer affect the next step.
• On Claude, use built-in context editing thresholds; in your own apps, trim the message array before each call.
• Keep structural anchors (task, constraints, current state) — drop the exhaust.

Condense earlier conversation into a short summary so the thread can continue without re-sending the full transcript.

• Summarize at a threshold (e.g. 60–70% of window) into a compact state object.
• Claude Code: run /compact at natural breakpoints; /clear between unrelated tasks.
• Persist a 'handoff' summary (decisions, open questions, current files) and drop raw history.

Feed only the relevant chunks via RAG instead of pasting whole documents or codebases into the prompt.

• Index your corpus; retrieve top-k relevant chunks per query.
• Tune k and chunk size — more is not better; it adds tokens and dilutes attention.
• Let a small model reason over retrieved context instead of fine-tuning or prompting a giant model on everything.

Use a compressor model to drop low-information tokens from long prompts/retrieved chunks while preserving answerability.

• Run retrieved RAG chunks through a compressor before sending to the expensive model.
• Compress few-shot example banks aggressively; they are mostly redundant.
• Validate answer quality on a held-out set before raising compression ratio.

Match semantically similar queries to a cached answer and skip the model call entirely.

• Embed incoming queries; if cosine similarity to a prior query exceeds a threshold, return the cached answer.
• Great for FAQ-shaped traffic, repeated support questions, and common classifications.
• Set TTL and invalidation rules so answers don't go stale.

Return a previously validated answer for an identical request instead of paying for another model call.

• Build the key from the normalized request, prompt version, model configuration, and tenant boundary.
• Cache only validated responses and define a TTL from the underlying information's freshness requirement.
• Record hit, miss, stale, and bypass outcomes so the cache's economics and correctness stay visible.

Watch out: Never share private answers across users or serve a cached answer after its source data or prompt version changes.

Send non-urgent work through batch endpoints for a flat discount.

• Queue offline jobs (evals, enrichment, backfills, summaries) to the Batch API.
• OpenAI & Anthropic batch ≈ 50% off; stacks with prompt caching.
• Accept up-to-24h turnaround (most finish in 1–6h).

Output tokens are usually the priciest. Cap and shape them.

• Set max_tokens / max_output_tokens to a real ceiling.
• Ask for terse answers and a specific format; forbid preamble and restated questions.
• Use structured output / JSON schema and stop sequences to end generation early.
• Disable verbose chain-of-thought when the task doesn't need it; enable extended thinking only when it pays.

Request the smallest machine-readable artifact the workflow needs, reducing prose, parse failures, and repair calls.

• Use a concise JSON schema, enum, or typed field set rather than asking for an explanatory essay.
• Validate locally and repair deterministic formatting issues without another model call.
• Track schema-valid response rate and cost per accepted result, not only raw output tokens.

Watch out: Large schemas also consume input tokens. Keep them narrow and avoid returning fields the next step does not use.

Use deterministic code, search, calculators, databases, and APIs first; invoke a model only where language understanding or reasoning adds value.

• Map each request class to the cheapest reliable executor before adding an LLM route.
• Use direct tools for time, arithmetic, lookup, conversion, validation, and known business rules.
• Send only the tool result needed for synthesis instead of the full transaction or document history.

Watch out: Tool access must be allow-listed, authenticated, bounded, and observable; deterministic does not mean automatically safe.

Classify intent and complexity before execution so each request takes the shortest valid path: cache, search, RAG, tool, small model, or agent.

• Start with rules or a compact classifier using a small, stable label set.
• Route common intents directly and reserve agent workflows for requests that genuinely need planning or multiple tools.
• Measure route accuracy, fallback rate, accepted-outcome cost, and latency for each class.

Watch out: A confident misroute can cost more than no router. Use explicit fallbacks and sample routed traffic for quality review.

Encode conventions once in a project memory file instead of repeating them in every prompt.

• Claude Code: CLAUDE.md. Cursor: .cursor/rules (.cursorrules). Copilot: .github/copilot-instructions.md. Agents: AGENTS.md.
• Keep them lean — they ride in context on every call.
• Store stable architecture, naming, and 'do/don't' rules; not volatile task detail.

Use the fewest, shortest examples that hit your quality bar — and cache them.

• Trim example count; test where quality plateaus.
• Move the example bank into a cached system prefix so you pay for it once per TTL.
• Prefer one excellent example over five mediocre ones.

Reference or diff instead of re-pasting entire files/datasets each turn.

• Share only the changed lines or a pointer/path the agent can read on demand.
• Strip comments, minify JSON, and dedupe boilerplate where safe.
• Use tool-based file reads so the model pulls only what it needs.

Cap loops, steps, and tool calls so a runaway agent can't silently burn a fortune.

• Set max-steps, max-tool-calls, and a hard token budget per task.
• Add early-exit on success and confidence checks before escalation.
• Log per-task token spend and alert on outliers.

Train a small model on your repetitive task using flagship outputs, then serve the cheap model.

• Collect high-quality flagship outputs for a narrow, high-volume task.
• Fine-tune a lightweight model (or distill) and route that traffic to it.
• Keep the flagship as a fallback for low-confidence cases.

You can't optimize what you can't see. Tag every call by feature/team/use-case and watch unit economics.

• Attach metadata (feature, team, customer, env) to every request.
• Track cost per 1K calls, tagging coverage %, and budget utilization %.
• Set budgets and anomaly alerts; review in regular FinOps cadences.